From the architectural point of view, sysdig is closer to tcpdump/wireshark than to systemtap/ktap.
systemtap/ktap work similarly to dtrace:
- a script is loaded into a user level process
- the process compiles the script and dispatches it to a kernel module
- the kernel module hooks the script into specific places in the kernel
- the kernel module sends the results back to userspace where the user can see them
sysdig works this way:
- the kernel module hooks into specific places in the kernel (using tracepoints), captures everything, and puts it into a shared memory buffer
- the buffer is accessed from the user-level sysdig process that reconstructs state (so it knows that fd 23 means /etc/passwd)
- filtering is applied
- scripting in Lua is applied
- the whole thing is optionally saved to disk so you can analyze later
Both approaches have pros and cons. We think that the sysdig approach creates a more natural workflow, ideal for troubleshooting and system administration tasks. Plus, writing scripts in Lua, with access to its rich libraries, is quite fun. :)
I want to give more details in a future blog post, so stay tuned.
It's good to see this come out.
(I was thinking about to implement this kind of tool, lua or nodejs was my two choices, now sysdig use luajit, great.)
Interesting enough, ktap vm is also based on luajit, if you want to add kernel filter scripting functionality into sysdig in future(without GCC needed), then just integrate ktap into your solution. :)
"Reconstruction" step looks like a source of unjustified inefficiency: why reconstruct state by traversing proc and doing lots of system calls instead of capturing all data in the first place where it's much cheaper to do?
Of course, we create and update the state by inspecting the incoming stream of system calls.
We traverse proc only once, when you start a capture, and the reason to do that is collecting info for the PIDs/FDs that existed before we start the system call collection. That way, you can for example create a filter on the IP address of a socket even if that socket was created before sysdig started.
From the architectural point of view, sysdig is closer to tcpdump/wireshark than to systemtap/ktap.
systemtap/ktap work similarly to dtrace: - a script is loaded into a user level process - the process compiles the script and dispatches it to a kernel module - the kernel module hooks the script into specific places in the kernel - the kernel module sends the results back to userspace where the user can see them
sysdig works this way: - the kernel module hooks into specific places in the kernel (using tracepoints), captures everything, and puts it into a shared memory buffer - the buffer is accessed from the user-level sysdig process that reconstructs state (so it knows that fd 23 means /etc/passwd) - filtering is applied - scripting in Lua is applied - the whole thing is optionally saved to disk so you can analyze later
Both approaches have pros and cons. We think that the sysdig approach creates a more natural workflow, ideal for troubleshooting and system administration tasks. Plus, writing scripts in Lua, with access to its rich libraries, is quite fun. :)
I want to give more details in a future blog post, so stay tuned.