[lttng-dev] "Hands-free" tracepoints using LD_PRELOAD

[Brian Rossa]

Francis,

Thanks for your detailed, helpful reply. I was especially excited to learn
about the progress on `--userspace-probe`, but unfortunately I *do* need to
extract the function arguments. It would seem, then, that interposition
through LD_PRELOAD is really the only option. The decision to port to LTTng
or not would just come down to how easy it was to generate all the required
boilerplate.

Here's a little more about my application:

   1. The whitelist is composed of mangled names that the user can extract
   from the target object with the standard `nm` workflow.
   2. Each mangled name in the whitelist gets demangled and parsed into
   `prefix`, `callable_name`, `argument_types`, etc.
   3. The list of `argument_types` is compared to the "extra annotation" I
   alluded to. This extra annotation is just a one-to-one map between argument
   type names and the names of structs that corresponding variables should be
   wrapped with before passing them to the logger. These structs just provide
   custom ostream operators for types that don't log nicely otherwise.
   4. Boilerplate that does the typical `log(...); auto return_val =
   dlsym(...); log(...); return return_val;` gets generated.
   5. `log(...)` is a thin interface to spdlog
   <https://github.com/gabime/spdlog> that handles `__attribute__`-based
   setup and teardown of a logger.

So at the end of the day, the shim developer provides:

   - The whitelist of mangled names
   - Implementations of struct "wrappers" that provide custom ostream
   operators
   - A map between type names and wrapper names

The machinery here seems fairly general-purpose, but I don't presume to be
an expert. My implementation is somewhat error-prone, and my main hope in
reaching out to the mailing list was that LTTng already had some of these
steps better-implemented. Step #2 is particularly problematic due to
ambiguities in the mangling grammar, and will need support going forward to
generalize well.

I would be happy to contribute some or all of my implementation if it's
something that the LTTng community would be interested in supporting and
extending.

Cheers!
~br

On Tue, Jan 22, 2019 at 12:16 PM Francis Deslauriers <
francis.deslauriers at efficios.com> wrote:

> Hi Brian,
> If I understand correctly, you want developers to provide a whitelist
> of functions that they want to instrument in an arbitrary shared
> library, and then have a system that instruments those functions
> without having to recompile the library with LTTng-UST tracepoints.
> Can you give us an example of annotation that the shim developer would
> provide?
> Do you need to support the extraction of function arguments? If you
> don't, here are two straightforward and tested approaches to solve
> your problem:
>
> First, overriding target symbols using LD_PRELOAD (like you suggested
> above):
> 1. Create boilerplate script #1 that generates tracepoints definitions
> [1] based on function name,
> 2. Create boilerplate script #2 that generates a function calling the
> right tracepoint and the original target function [2],
> 3. For each target symbol "bar", using script #1, create a tracepoint
> definition "libfoo:bar_tp", and compile them into a tracepoints.o.
> object file,
> 4. For each target symbol "bar", using script #2, create a function
> "bar" that calls the bar_tp tracepoint (e.g. tracepoint(libfoo,
> bar_tp) ) and then calls the original bar function in libfoo.so (using
> dlsym(3)), and compile those callsites into a callsites.o object file,
> 5. Create a shared library libshim.so using tracepoints.o and
> callsites.o object files and link it to lttng-ust lib and the dynamic
> loader lib using :"-llttng-ust -ldl",
> Those steps results in the creation of a libshim.so file that contains
> the "bar" overriding function (containing the tracepoint callsite)
> that will be called instead of "bar" function in the libfoo.so library
> when LD_PRELOADing it (e.g. LD_PRELOAD=libshim.so ./fooapp). Here is
> the Makefile I used to test this [3] to give you a sense of the
> pipeline.
>
> Second, using the newly added --userspace-probe instrumentation:
> This option allows you to instrument any ELF symbol without
> recompiling or restarting the target process.
> For example, to instrument the "bar" function in the /tmp/libfoo.so
> file you could run to following command:
>     > lttng enable-event -k --userspace-probe="elf:/tmp/libfoo.so:bar"
> $event_name
> You could write a script that takes the list of target functions and
> adds userspace-probes on each of them.
> This approach has the advantage that you can also instrument functions
> in the binary and not only functions in shared libraries.
> This instrumentation type uses the kernel tracer and thus requires
> using the lttng-modules.
>
> If you do need to save the function arguments, I believe it's possible
> using the first approach, by tweaking the two boilerplate scripts
> depending on what are the annotations you want to provide.
> Supporting argument extraction with userspace-probes is part of our
> long term goal and would be a really powerful feature but will require
> a significant engineering effort to accomplish.
>
> [1] : https://lttng.org/docs/v2.10/#doc-tpp-header
> [2]: https://lttng.org/docs/v2.10/#doc-probing-the-application-source-code
> [3]: http://paste.ubuntu.com/p/5fBSgRqXsB/
>
> I hope this make sense. Please don't hesitate if you want more details.
>
> Thank you,
> Francis
>
> Le ven. 18 janv. 2019, à 16 h 04, Brian Rossa <br at f0cal.com> a écrit :
> >
> > Hello,
> >
> > I have a custom tracing solution that I'm interested in porting to
> LTTng. It works as follows:
> >
> > Input is a whitelist of mangled names taken from a target library,
> libfoo.so.
> > For each name, generate a logging shim (example) for the corresponding
> call.
> > Compile all the logging shims into a shared library, libshim.so.
> > Trace any application using libfoo by running it as
> LD_PRELOAD=libshim.so ./fooapp
> >
> > There are two nice things about this approach that I would like to
> preserve:
> >
> > The shim developer only needs to provide the whitelist and a bit of
> extra annotation. From there, the shim can be be authored using a
> boilerplate generator.
> > The app developer doesn't have to do anything other than pass the
> LD_PRELOAD flag.
> >
> > The downside is that the only tracepoints are those corresponding to the
> whitelist, but I'm fine with that.
> >
> > Can this kind of "hands-free" developer experience be supported by LTTng?
> >
> > Thanks!
> > ~br
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev at lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
>
> --
> Francis Deslauriers
> Computer Engineer
> EfficiOS inc.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190122/2dfd772f/attachment.html>