[lttng-dev] [RFC] Dynamic instrumentation support in UST
Simon Marchi
simon.marchi at polymtl.ca
Fri Jul 26 13:40:03 EDT 2013
Hi Zifei,
That looks great! Questions and comments below.
On 26 July 2013 11:40, Zifei Tong <soariez at gmail.com> wrote:
> Hi everyone,
>
> Here is the RFC for the dynamic instrumentation support in UST.
>
> It contains a command line interface proposal and some notes on
> implementation details.
>
> Please provide your comments. Any questions, or fixes are VERY welcome.
>
>
> Command Line Interface
> ----------------------
>
> The proposed command line interface is very similar to the current lttng kernel
> dynamic probe interface [1].
>
> To enable a dynamic probe in a running process at given address, you can use:
>
> lttng enable-event NAME -u --pid PID
> --probe (addr | symbol | symbol+offset)
> Dynamic UST probe.
> Addr and offset can be octal (0NNN...),
> decimal (NNN...) or hexadecimal (0xNNN...)
>
> This will place a bare probe at certain address.
>
> Examples:
>
> # lttng enable-event aname -u --pid 8964 --probe foo+0x6
> # lttng enable-event aname -u --pid 8964 --probe 0xdeadbeef
>
> You can also enable a tracepoint at the entry of a given function:
>
> lttng enable-event NAME -u --pid PID
> --function (addr | symbol | symbol+offset)
> Dynamic UST function entry probe.
> Addr and offset can be octal (0NNN...),
> decimal (NNN...) or hexadecimal (0xNNN...)
>
> This will place a bare probe at the entry point of certain function.
What is the difference if I do
--function foo
vs
--probe foo
?
> Examples:
>
> # lttng enable-event aname -u --pid 8964 --function foo
>
> Being only able to attach and instrument existing processes is sometimes
> restricted. If we can have #15 [2] implemented, then we can extend the command
> to support dynamic instrumentation.
>
> lttng trace -c COMMAND
> --probe (addr | symbol | symbol+offset)
> --function (addr | symbol | symbol+offset)
>
> This will execute given program and place probes at certain places.
>
> Examples:
>
> # lttng trace -c "ls -l" --probe main+0x10
> # lttng trace -c "ls -l" --function main
>
> Probes collecting context data are more useful than bare ones. We can extend
> the add-context command to support more context types.
>
> lttng add-context -u -e NAME --pid PID
> -t, --type TYPE Context type.
> TYPE can be one of the strings below:
> regs
> all registers
> reg:reg_name
> register with name reg_name
> var_type:var_name:addr
> variable at addr with field name var_name
> backtrace
> stack backtrace
>
> Examples:
>
> # lttng add-context -u --pid PID -e aname -t reg:rax
> # lttng add-context -u --pid PID -e aname -t int:varname:0xdeadbeef
> # lttng add-context -u --pid PID -e aname -t string:varname:0xdeadbeef
> # lttng add-context -u --pid PID -e aname -t backtrace
Does it make sense to put collected variable values in the context? I
would instinctively put them as event fields. Backtrace and registers
are probably more of a context though. Either way, it will require the
use of per-event context. I don't think that this feature has been
very much tested in the CTF parsers (looking at the ctf testsuite), so
it will be a good test :)
> These context types appear in other tracers [4],[5],[6], however
> implement all of
> them may be not possible within the GSoC schedule.
>
> The above proposed command line interface require defining tracepoint event at
> runtime thus free user form compiling tracepoint probes themselves. But I have
> a concern such runtime defined tracepoint event mechanism may eventually evolve
> to a tiny compiler/interpreter like Systemtap [7] or gdb [8].
>
>
> Implementation
> --------------
>
> After comparing dyninst with gdb's tracepoint code, I think using dyninst as
> current choice of implementing dynamic instrumentation is more appropriate.
> gdb's tracepoint function relies on a lot of support code they already have,
> such as disassembler, elf/DWARF parser. We'll need to a lot of work to port all
> the support code.
>
> The underlying technique of these two are quite similar. I will try if I can
> isolate interfaces to make changing back-end of dynamic instrumentation easier
> or even configurable.
>
> The technique behind scene is simple. We replace the instrumented instruction
> with a jmp to redirect the control flow to a trampoline. After saving registers
> in the trampoline, we call the probe callback function. Then restore the saved
> registers, execute the original instruction and jmp back to the original place.
>
> I will give a simple example here.
>
> Suppose function do_stuff has the following assembly code:
>
> 0x0000000000400a30 <+0>: push %rbx
> 0x0000000000400a31 <+1>: mov $0x400b34,%esi
> 0x0000000000400a36 <+6>: mov $0x400b36,%edi
> 0x0000000000400a3b <+11>: callq 0x4007b0 <fopen at plt>
> 0x0000000000400a62 <+16>: ret
>
> After executing the following command:
>
> # lttng enable-event aname -u --pid 8964 --probe do_stuff+0x6
>
> function do_stuff will become:
>
> 0x0000000000400a30 <+0>: push %rbx
> 0x0000000000400a31 <+1>: mov $0x400b34,%esi
> 0x0000000000400a36 <+6>: jmpq 0x1002a # jmp to trampoline
> 0x0000000000400a3b <+11>: callq 0x4007b0 <fopen at plt>
> 0x0000000000400a62 <+16>: ret
>
> (gdb) x/100i 0x1002a
> 0x1002a: push %rax # save registers
> ....................
> 0x1003e: push %r15
> 0x1004f: movabs $0x7f682f61eac7,%rbx
> 0x10059: callq *%rbx # call lttng probe
> 0x10060: pop %r15 # restore registers
> ....................
> 0x10075: pop %rax
> 0x1007a: mov $0x400b36,%edi # this is the original instruction
You have to think about instructions that act relatively to the
current ip. These have to be adjusted to give the correct result. See:
https://github.com/MentorEmbedded/Debug-Agent-Library/blob/master/x86-64-linux-backend.c#L239
> 0x1007f: jmpq 0x0000000000400a3b # jmp back
>
> This is an oversimplified example. In reality, we need to do more than that,
> like aligning stack. You can refer [3] for a detailed line-by-line analysis
> on dyninst's behavior.
>
>
> [1]: http://bugs.lttng.org/projects/lttng-tools/wiki
> [2]: http://bugs.lttng.org/issues/15
> [3]: https://github.com/5kg/lttng-gsoc/blob/master/notes/dyninst.md
> [4]: http://sourceware.org/systemtap/SystemTap_Beginners_Guide/utargetvariable.html
> [5]: http://sourceware.org/systemtap/SystemTap_Beginners_Guide/ustack.html
> [6]: http://sourceware.org/gdb/onlinedocs/gdb/Tracepoint-Actions.html#Tracepoint-Actions
> [7]: http://sourceware.org/systemtap/SystemTap_Beginners_Guide/understanding-how-systemtap-works.html#understanding-architecture-tools
> [8]: http://sourceware.org/gdb/onlinedocs/gdb/Agent-Expressions.html
>
> Thanks,
> 仝子飞 (Zifei Tong)
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
More information about the lttng-dev
mailing list