[lttng-dev] "Hands-free" tracepoints using LD_PRELOAD

[Philippe Proulx]

On Fri, Jan 25, 2019 at 4:05 AM Brian Rossa <br at f0cal.com> wrote:
>
> Francis,
>
> These are great suggestions, thanks!
>
>> #Third idea:
>> Do you know of tracef() [3] ? Using it, you can save any string to a
>> UST trace. As a first step, you could directly replace your calls to
>> spdlog by calls to tracef. It's an highly inefficient way of using
>> LTTng, but it works (and probably lower overhead than writing to a
>> file).
>
>
> Replacing spdlog::debug(...) with tracef(...) may be an easy way for me to get familiar with LTTNg workflows without having to go through a complete port.

However please note that spdlog uses {fmt} while tracef() uses a
vsnprintf()-family function, so you would need to adapt the format
strings too.

Phil

>
> This brings up an interesting question, though, and the answer may motivate me to close the gap further: What kind of latency reduction can I expect from moving from spdlog to LLTng? I know this is ill-posed without knowing more about how many pseudo-tracepoints I'm implementing, the log message sizes that I'm pushing to disk, etc, etc., but something notional would help my motivation.
>
>>
>> You mentioned that you wrap structures with ostreams to output them in
>> text format. Can you explain this a bit more?
>
>
> I'm just implementing what spdlog suggests: https://github.com/gabime/spdlog#user-defined-types
>
> Cheers!
> ~br
>
> On Thu, Jan 24, 2019 at 11:06 AM Francis Deslauriers <francis.deslauriers at efficios.com> wrote:
>>
>> "
>>
>> Le mer. 23 janv. 2019, à 19 h 20, Brian Rossa <br at f0cal.com> a écrit :
>> >
>> > Jonathan,
>> >
>> > Responses below:
>> >
>> >> AFAIK, lttng does not have an equivalent.
>> >
>> >
>> > I believe my code could significantly reduce the need for hand-writing the tracepoints. But I won't likely take on a port to LTTng immediately, as a "vanilla" interposition approach seems to be meeting my requirements.
>> >
>> >>
>> >> > Step #2 is particularly problematic due to
>> >> > ambiguities in the mangling grammar, and will need support going forward to
>> >> > generalize well.
>> >>
>> >> What is the status of this step in your project?
>> >
>> >
>> > I demangle the symbol name using c++filt and then use regex to extract the list of argument types. Using something like ItaniumPartialDemangler would be better.
>>
>> Hi Brian,
>>
>> If you have a list of the argument types, you can programmatically
>> generate the tracepoint descriptions and callsites accordingly. The
>> main blocker I see here is tracing arguments that are pointers to
>> classes or containers. We need to be able to map each argument with a
>> CTF type [1]. It's easy enough for int, float and char * but it's
>> harder for complex structs and data structures.
>>
>> You mentioned that you wrap structures with ostreams to output them in
>> text format. Can you explain this a bit more?
>>
>> Here are a few ideas:
>>
>> #First idea
>> If you are already defining the printing format and order of each of
>> the fields of each structures in your libfoo.so maybe you could do the
>> same but in LTTng-UST format. See "my-custom-structure.h" example [2].
>>
>> #Second idea:
>> If you prefer not convert those ostreams wrappers to ctf wrapper, you
>> could reuse them to generate CTF_STRINGs.
>> 1. Simple data types (int, float, char*) are mapped directly to CTF types,
>> 2. Complex data types are wrapped with ostream function,
>> 3. Complex data types are saved in the trace as CTF_STRING using the ostream.
>> All this could be done by the boilerplate scripts I mentioned earlier.
>> By using string format for some argument, you don't get the full power
>> of LTTng but it will still be faster that saving everything in text.
>>
>> #Third idea:
>> Do you know of tracef() [3] ? Using it, you can save any string to a
>> UST trace. As a first step, you could directly replace your calls to
>> spdlog by calls to tracef. It's an highly inefficient way of using
>> LTTng, but it works (and probably lower overhead than writing to a
>> file).
>>
>> [1]: https://lttng.org/man/3/lttng-ust/v2.10/
>> [2]: https://lttng.org/docs/v2.10/#doc-defining-tracepoints
>> [3]: https://lttng.org/docs/v2.10/#doc-tracef
>>
>> Thank you,
>> Francis
>>
>> >
>> >>
>> >> What are the problems that make your implementation "error-pone"?
>> >
>> >
>> > Use of regex as above.
>> >
>> >>
>> >> Would you mind linking us to said project so we can have a look?
>> >
>> >
>> > It's a project for my employer, and I would need to keep the source closed until I can get an OK. A signal of interest from LTTng would be helpful there. In the meantime, I think it would be fine to share as read-only with the LTTng maintainers if someone wants to send me a Github username.
>> >
>> >>
>> >> I would be interested in seeing at first lttng tracepoint used as Francis
>> >> demonstrated and see from there were this project can go.
>> >
>> > ...
>> >>
>> >> > I would be happy to contribute some or all of my implementation if it's
>> >> > something that the LTTng community would be interested in supporting and
>> >> > extending.
>> >>
>> >> We are clearly open for discussion and helping you improve the project. I am not
>> >> so sure on supporting and extending it. Others might have a different opinion.
>> >
>> >
>> > Sounds good. Looking forward to connecting via personal email.
>> >
>> > Cheers!
>> > ~br
>> >
>> >
>> > On Tue, Jan 22, 2019 at 2:17 PM Jonathan Rajotte-Julien <jonathan.rajotte-julien at efficios.com> wrote:
>> >>
>> >> Hi Brian,
>> >>
>> >> On Tue, Jan 22, 2019 at 01:30:23PM -0500, Brian Rossa wrote:
>> >> >    4. Boilerplate that does the typical `log(...); auto return_val =
>> >> >    dlsym(...); log(...); return return_val;` gets generated.
>> >>
>> >> As proposed by Francis, this is when you need to "generate" a corresponding
>> >> tracepoint definition and call the tracepoint() call with the appropriate
>> >> arguments.
>> >>
>> >> As Francis demonstrated we do not see any reason for lttng-ust not to work here
>> >> given that you compile the libshim object correctly.
>> >>
>> >> >    5. `log(...)` is a thin interface to spdlog
>> >> >    <https://github.com/gabime/spdlog> that handles `__attribute__`-based
>> >> >    setup and teardown of a logger.
>> >> >
>> >> > So at the end of the day, the shim developer provides:
>> >> >
>> >> >    - The whitelist of mangled names
>> >> >    - Implementations of struct "wrappers" that provide custom ostream
>> >> >    operators
>> >> >    - A map between type names and wrapper names
>> >> >
>> >> > The machinery here seems fairly general-purpose, but I don't presume to be
>> >> > an expert. My implementation is somewhat error-prone, and my main hope in
>> >> > reaching out to the mailing list was that LTTng already had some of these
>> >> > steps better-implemented.
>> >>
>> >> AFAIK, lttng does not have an equivalent.
>> >>
>> >> > Step #2 is particularly problematic due to
>> >> > ambiguities in the mangling grammar, and will need support going forward to
>> >> > generalize well.
>> >>
>> >> What is the status of this step in your project?
>> >>
>> >> What are the problems that make your implementation "error-pone"?
>> >>
>> >> Would you mind linking us to said project so we can have a look?
>> >>
>> >> I would be interested in seeing at first lttng tracepoint used as Francis
>> >> demonstrated and see from there were this project can go.
>> >>
>> >> >
>> >> > I would be happy to contribute some or all of my implementation if it's
>> >> > something that the LTTng community would be interested in supporting and
>> >> > extending.
>> >>
>> >> We are clearly open for discussion and helping you improve the project. I am not
>> >> so sure on supporting and extending it. Others might have a different opinion.
>> >>
>> >>
>> >> Cheers
>> >>
>> >> --
>> >> Jonathan Rajotte-Julien
>> >> EfficiOS
>>
>>
>>
>> --
>> Francis Deslauriers
>> Computer Engineer
>> EfficiOS inc.
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev