[lttng-dev] "Hands-free" tracepoints using LD_PRELOAD

[Brian Rossa]

Ok, after further review and some performance analysis on spdlog, I'm
considering skipping the tracef(...) step and jumping to a tracepoint-based
approach. I believe I can use code generation to create both (1) the
tracepoint definitions and (2) function interposition (LD_PRELOAD) shims
calling to #1. This may take some time, but I'll ping this thread if I
encounter any issues. Thanks! ~br

On Fri, Jan 25, 2019 at 9:35 AM Philippe Proulx <eeppeliteloop at gmail.com>
wrote:

> On Fri, Jan 25, 2019 at 4:05 AM Brian Rossa <br at f0cal.com> wrote:
> >
> > Francis,
> >
> > These are great suggestions, thanks!
> >
> >> #Third idea:
> >> Do you know of tracef() [3] ? Using it, you can save any string to a
> >> UST trace. As a first step, you could directly replace your calls to
> >> spdlog by calls to tracef. It's an highly inefficient way of using
> >> LTTng, but it works (and probably lower overhead than writing to a
> >> file).
> >
> >
> > Replacing spdlog::debug(...) with tracef(...) may be an easy way for me
> to get familiar with LTTNg workflows without having to go through a
> complete port.
>
> However please note that spdlog uses {fmt} while tracef() uses a
> vsnprintf()-family function, so you would need to adapt the format
> strings too.
>
> Phil
>
> >
> > This brings up an interesting question, though, and the answer may
> motivate me to close the gap further: What kind of latency reduction can I
> expect from moving from spdlog to LLTng? I know this is ill-posed without
> knowing more about how many pseudo-tracepoints I'm implementing, the log
> message sizes that I'm pushing to disk, etc, etc., but something notional
> would help my motivation.
> >
> >>
> >> You mentioned that you wrap structures with ostreams to output them in
> >> text format. Can you explain this a bit more?
> >
> >
> > I'm just implementing what spdlog suggests:
> https://github.com/gabime/spdlog#user-defined-types
> >
> > Cheers!
> > ~br
> >
> > On Thu, Jan 24, 2019 at 11:06 AM Francis Deslauriers <
> francis.deslauriers at efficios.com> wrote:
> >>
> >> "
> >>
> >> Le mer. 23 janv. 2019, à 19 h 20, Brian Rossa <br at f0cal.com> a écrit :
> >> >
> >> > Jonathan,
> >> >
> >> > Responses below:
> >> >
> >> >> AFAIK, lttng does not have an equivalent.
> >> >
> >> >
> >> > I believe my code could significantly reduce the need for
> hand-writing the tracepoints. But I won't likely take on a port to LTTng
> immediately, as a "vanilla" interposition approach seems to be meeting my
> requirements.
> >> >
> >> >>
> >> >> > Step #2 is particularly problematic due to
> >> >> > ambiguities in the mangling grammar, and will need support going
> forward to
> >> >> > generalize well.
> >> >>
> >> >> What is the status of this step in your project?
> >> >
> >> >
> >> > I demangle the symbol name using c++filt and then use regex to
> extract the list of argument types. Using something like
> ItaniumPartialDemangler would be better.
> >>
> >> Hi Brian,
> >>
> >> If you have a list of the argument types, you can programmatically
> >> generate the tracepoint descriptions and callsites accordingly. The
> >> main blocker I see here is tracing arguments that are pointers to
> >> classes or containers. We need to be able to map each argument with a
> >> CTF type [1]. It's easy enough for int, float and char * but it's
> >> harder for complex structs and data structures.
> >>
> >> You mentioned that you wrap structures with ostreams to output them in
> >> text format. Can you explain this a bit more?
> >>
> >> Here are a few ideas:
> >>
> >> #First idea
> >> If you are already defining the printing format and order of each of
> >> the fields of each structures in your libfoo.so maybe you could do the
> >> same but in LTTng-UST format. See "my-custom-structure.h" example [2].
> >>
> >> #Second idea:
> >> If you prefer not convert those ostreams wrappers to ctf wrapper, you
> >> could reuse them to generate CTF_STRINGs.
> >> 1. Simple data types (int, float, char*) are mapped directly to CTF
> types,
> >> 2. Complex data types are wrapped with ostream function,
> >> 3. Complex data types are saved in the trace as CTF_STRING using the
> ostream.
> >> All this could be done by the boilerplate scripts I mentioned earlier.
> >> By using string format for some argument, you don't get the full power
> >> of LTTng but it will still be faster that saving everything in text.
> >>
> >> #Third idea:
> >> Do you know of tracef() [3] ? Using it, you can save any string to a
> >> UST trace. As a first step, you could directly replace your calls to
> >> spdlog by calls to tracef. It's an highly inefficient way of using
> >> LTTng, but it works (and probably lower overhead than writing to a
> >> file).
> >>
> >> [1]: https://lttng.org/man/3/lttng-ust/v2.10/
> >> [2]: https://lttng.org/docs/v2.10/#doc-defining-tracepoints
> >> [3]: https://lttng.org/docs/v2.10/#doc-tracef
> >>
> >> Thank you,
> >> Francis
> >>
> >> >
> >> >>
> >> >> What are the problems that make your implementation "error-pone"?
> >> >
> >> >
> >> > Use of regex as above.
> >> >
> >> >>
> >> >> Would you mind linking us to said project so we can have a look?
> >> >
> >> >
> >> > It's a project for my employer, and I would need to keep the source
> closed until I can get an OK. A signal of interest from LTTng would be
> helpful there. In the meantime, I think it would be fine to share as
> read-only with the LTTng maintainers if someone wants to send me a Github
> username.
> >> >
> >> >>
> >> >> I would be interested in seeing at first lttng tracepoint used as
> Francis
> >> >> demonstrated and see from there were this project can go.
> >> >
> >> > ...
> >> >>
> >> >> > I would be happy to contribute some or all of my implementation if
> it's
> >> >> > something that the LTTng community would be interested in
> supporting and
> >> >> > extending.
> >> >>
> >> >> We are clearly open for discussion and helping you improve the
> project. I am not
> >> >> so sure on supporting and extending it. Others might have a
> different opinion.
> >> >
> >> >
> >> > Sounds good. Looking forward to connecting via personal email.
> >> >
> >> > Cheers!
> >> > ~br
> >> >
> >> >
> >> > On Tue, Jan 22, 2019 at 2:17 PM Jonathan Rajotte-Julien <
> jonathan.rajotte-julien at efficios.com> wrote:
> >> >>
> >> >> Hi Brian,
> >> >>
> >> >> On Tue, Jan 22, 2019 at 01:30:23PM -0500, Brian Rossa wrote:
> >> >> >    4. Boilerplate that does the typical `log(...); auto return_val
> =
> >> >> >    dlsym(...); log(...); return return_val;` gets generated.
> >> >>
> >> >> As proposed by Francis, this is when you need to "generate" a
> corresponding
> >> >> tracepoint definition and call the tracepoint() call with the
> appropriate
> >> >> arguments.
> >> >>
> >> >> As Francis demonstrated we do not see any reason for lttng-ust not
> to work here
> >> >> given that you compile the libshim object correctly.
> >> >>
> >> >> >    5. `log(...)` is a thin interface to spdlog
> >> >> >    <https://github.com/gabime/spdlog> that handles
> `__attribute__`-based
> >> >> >    setup and teardown of a logger.
> >> >> >
> >> >> > So at the end of the day, the shim developer provides:
> >> >> >
> >> >> >    - The whitelist of mangled names
> >> >> >    - Implementations of struct "wrappers" that provide custom
> ostream
> >> >> >    operators
> >> >> >    - A map between type names and wrapper names
> >> >> >
> >> >> > The machinery here seems fairly general-purpose, but I don't
> presume to be
> >> >> > an expert. My implementation is somewhat error-prone, and my main
> hope in
> >> >> > reaching out to the mailing list was that LTTng already had some
> of these
> >> >> > steps better-implemented.
> >> >>
> >> >> AFAIK, lttng does not have an equivalent.
> >> >>
> >> >> > Step #2 is particularly problematic due to
> >> >> > ambiguities in the mangling grammar, and will need support going
> forward to
> >> >> > generalize well.
> >> >>
> >> >> What is the status of this step in your project?
> >> >>
> >> >> What are the problems that make your implementation "error-pone"?
> >> >>
> >> >> Would you mind linking us to said project so we can have a look?
> >> >>
> >> >> I would be interested in seeing at first lttng tracepoint used as
> Francis
> >> >> demonstrated and see from there were this project can go.
> >> >>
> >> >> >
> >> >> > I would be happy to contribute some or all of my implementation if
> it's
> >> >> > something that the LTTng community would be interested in
> supporting and
> >> >> > extending.
> >> >>
> >> >> We are clearly open for discussion and helping you improve the
> project. I am not
> >> >> so sure on supporting and extending it. Others might have a
> different opinion.
> >> >>
> >> >>
> >> >> Cheers
> >> >>
> >> >> --
> >> >> Jonathan Rajotte-Julien
> >> >> EfficiOS
> >>
> >>
> >>
> >> --
> >> Francis Deslauriers
> >> Computer Engineer
> >> EfficiOS inc.
> >
> > _______________________________________________
> > lttng-dev mailing list
> > lttng-dev at lists.lttng.org
> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190129/3cb132ee/attachment.html>