[lttng-dev] "Hands-free" tracepoints using LD_PRELOAD

[Francis Deslauriers]

Hi Brian,
If I understand correctly, you want developers to provide a whitelist
of functions that they want to instrument in an arbitrary shared
library, and then have a system that instruments those functions
without having to recompile the library with LTTng-UST tracepoints.
Can you give us an example of annotation that the shim developer would
provide?
Do you need to support the extraction of function arguments? If you
don't, here are two straightforward and tested approaches to solve
your problem:

First, overriding target symbols using LD_PRELOAD (like you suggested above):
1. Create boilerplate script #1 that generates tracepoints definitions
[1] based on function name,
2. Create boilerplate script #2 that generates a function calling the
right tracepoint and the original target function [2],
3. For each target symbol "bar", using script #1, create a tracepoint
definition "libfoo:bar_tp", and compile them into a tracepoints.o.
object file,
4. For each target symbol "bar", using script #2, create a function
"bar" that calls the bar_tp tracepoint (e.g. tracepoint(libfoo,
bar_tp) ) and then calls the original bar function in libfoo.so (using
dlsym(3)), and compile those callsites into a callsites.o object file,
5. Create a shared library libshim.so using tracepoints.o and
callsites.o object files and link it to lttng-ust lib and the dynamic
loader lib using :"-llttng-ust -ldl",
Those steps results in the creation of a libshim.so file that contains
the "bar" overriding function (containing the tracepoint callsite)
that will be called instead of "bar" function in the libfoo.so library
when LD_PRELOADing it (e.g. LD_PRELOAD=libshim.so ./fooapp). Here is
the Makefile I used to test this [3] to give you a sense of the
pipeline.

Second, using the newly added --userspace-probe instrumentation:
This option allows you to instrument any ELF symbol without
recompiling or restarting the target process.
For example, to instrument the "bar" function in the /tmp/libfoo.so
file you could run to following command:
    > lttng enable-event -k --userspace-probe="elf:/tmp/libfoo.so:bar"
$event_name
You could write a script that takes the list of target functions and
adds userspace-probes on each of them.
This approach has the advantage that you can also instrument functions
in the binary and not only functions in shared libraries.
This instrumentation type uses the kernel tracer and thus requires
using the lttng-modules.

If you do need to save the function arguments, I believe it's possible
using the first approach, by tweaking the two boilerplate scripts
depending on what are the annotations you want to provide.
Supporting argument extraction with userspace-probes is part of our
long term goal and would be a really powerful feature but will require
a significant engineering effort to accomplish.

[1] : https://lttng.org/docs/v2.10/#doc-tpp-header
[2]: https://lttng.org/docs/v2.10/#doc-probing-the-application-source-code
[3]: http://paste.ubuntu.com/p/5fBSgRqXsB/

I hope this make sense. Please don't hesitate if you want more details.

Thank you,
Francis

Le ven. 18 janv. 2019, à 16 h 04, Brian Rossa <br at f0cal.com> a écrit :
>
> Hello,
>
> I have a custom tracing solution that I'm interested in porting to LTTng. It works as follows:
>
> Input is a whitelist of mangled names taken from a target library, libfoo.so.
> For each name, generate a logging shim (example) for the corresponding call.
> Compile all the logging shims into a shared library, libshim.so.
> Trace any application using libfoo by running it as LD_PRELOAD=libshim.so ./fooapp
>
> There are two nice things about this approach that I would like to preserve:
>
> The shim developer only needs to provide the whitelist and a bit of extra annotation. From there, the shim can be be authored using a boilerplate generator.
> The app developer doesn't have to do anything other than pass the LD_PRELOAD flag.
>
> The downside is that the only tracepoints are those corresponding to the whitelist, but I'm fine with that.
>
> Can this kind of "hands-free" developer experience be supported by LTTng?
>
> Thanks!
> ~br
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev



--
Francis Deslauriers
Computer Engineer
EfficiOS inc.