[lttng-dev] sdt.h tracepoints with unicode data and/or structs

[Milian Wolff]

On Montag, 12. September 2016 16:24:04 CEST Mathieu Desnoyers wrote:
> ----- On Sep 12, 2016, at 11:40 AM, Milian Wolff milian.wolff at kdab.com 
wrote:
> > On Monday, September 12, 2016 3:03:04 PM CEST Mathieu Desnoyers wrote:
> >> ----- On Sep 6, 2016, at 3:00 PM, Milian Wolff milian.wolff at kdab.com 
wrote:
> >> > Hey all,
> >> > 
> >> > where can I find more documentation on how to use sdt.h to add static
> >> > tracepoints to user-land applications? If this is not the right place
> >> > to
> >> > ask, please refer to to elsewhere.
> >> 
> >> Hi Milian,
> > 
> > Hey Mathieu,
> > 
> >> LTTng-UST offers a "tracepoint" instrumentation facility, which can
> >> optionally emit "sdt.h" probe points too for compatibility with
> >> SystemTAP.
> >> 
> >> > I plan to upstream a collection of tracepoints to Qt, and possibly
> >> > elsewhere. One problem I'm having right now is figuring out how to
> >> > "design" the tracepoints such that they have minimal overhead.
> >> 
> >> The main question here would be: do you want your instrumentation to be
> >> usable with LTTng-UST ? If yes, then you want the tracepoint
> >> instrumentation facility. Else, if you only plan on using SystemTAP, you
> >> can use sdt.h instrumentation.
> >> 
> >> > So my questions:
> >> My answers will be about lttng-ust tracepoints.
> > 
> > Thanks for the clarifications! Are the LTTNG-UST tracepoints also "zero-
> > overhead" like the SystemTap once, i.e. put into a separate section and
> > "only" one nop is added when tracing is disabled? I could not find such
> > an information anywhere yet.
> 
> The lttng-ust tracepoint mechanism comes from the design of the
> Linux kernel tracepoints, which have proven to be unnoticeable
> when disabled. There is one slight technicality though:
> 
> The linux kernel tracepoints use the asm goto mechanism and code patching
> to flip between a no-op and a jump to dynamically enable each tracepoint.
> This is all very fine for the kernel.
> 
> Now for user-space, the lttng-ust tracepoints rather use a conditional
> branch to skip over the entire stack setup and the function call. There
> are a few reasons for using a old-fashioned conditional branch: first,
> there are no widely adopted multi-core, live code patching library
> available in user-space that am I aware of. Second, even if we would
> have one, we may not want to send SIGSTOP/SIGCONT to the traced process,
> due to debugger interactions. Thirdly, doing code patching may not
> interact will security features such as read-only code sections and
> code checksumming. Finally, doing code patching of library and
> executable code will trigger copy-on-write of the touched pages, which
> will prevent sharing those pages between processes running the same
> apps/libraries. All in all, the conditional branch does not seem like
> a too bad alternative after all.
> 
> If we look at the code generated by SystemTAP sdt.h probes (as well
> as DTrace), they are actually more heavyweight than what has been
> claimed by Sun's marketing department back in the days: it does
> cost a stack setup of all the variables passed to the the probe,
> and indeed the function call is no-op'd.
> 
> The downside here is that all side-effects, and layout of the arguments
> for the no-op'd call, are taken by the instrumented application, even
> though tracing is not dynamically enabled.
> 
> Therefore, the SDT mechanism is not a lightweight as is generally
> claimed. It is not "just a single no-op per site", but rather a
> function call stack setup and a no-op.

Thanks a lot for the in-depth explanation Mathieu, much appreciated! From what 
I gather, sdt.h does allow to check whether a given trace point is enabled or 
not, no? I.e. via the semaphores. Is there any performance difference between 
checking the semaphore of a sdt.h tracepoint vs. doing the analogous check for 
lttng-ust tracepoints? 

> >> > What data types can I use for trace point arguments?
> >> 
> >> See http://lttng.org/docs/#doc-c-application
> >> 
> >> > Can I pass UTF16 strings? Do they need to be null-terminated?
> >> 
> >> You should convert them to UTF8.
> >> 
> >> A ctf_string() needs to be null-terminated. A ctf_sequence_text() is
> >> not required to be null-terminated. See
> >> http://lttng.org/docs/#doc-liblttng-ust-tp-fields
> > 
> > This sounds like tracing would then incur a huge runtime overhead, when I
> > need to convert all my UTF-16 strings to UTF-8. How does one deal with
> > that?
>
> Are you concerned about the runtime overhead when tracing is disabled, or
> enabled ? It would be good to gather some metrics on the overhead of this
> conversion.

I'm mostly concerned about the overhead when tracing is disabled. Ideally, I 
would like to see these tracepoints unconditionally compiled into Qt. But if 
the overhead is noticeable when they are not in use, I may have a hard time 
achieving this goal. Instead, one will then probably need to recompile Qt with 
the tracepoints enabled.

When in use, the overhead of the tracepoints should also be minimal. 
Allocating memory for a temporary UTF-16 to UTF-8 conversion alone has a large 
overhead, and then converting the data to UTF-8 also adds on top of that. 
Thus, if at all possible, I would like to prevent that.

Similarly, I am looking for a way to put URLs into tracepoints, and converting 
a QUrl to string data is far from cheap.

At this point, I don't see a way around only enabling tracepoints optionally. 
Independent of the mechanism in use for the actual tracepoints (i.e. lttng-ust 
or sdt). The conditional to check whether tracing is enabled may not be too 
bad. But then in the conditional it looks like $some code will be required to 
massage the data into a form that the tracepoints accept them. This increase 
in code size negatively influences code caches and I don't see any way around 
that.
 
> > Looking at the API documentation, using ctf_array instead with ushort as C
> > type sounds like a better approach. In a custom babeltrace consumer
> > script, I could then convert it to UTF8 offline, if needed.
> > 
> > The advantage here is that I have "zero" overhead while tracing, which is
> > crucial for me to get the tracepoints accepted upstream in Qt, without
> > hiding them behind a compiler switch.
> 
> As a short term solution this appears to be fine.
> 
> In the long run, we're currently designing CTF 2.0, and we're been wondering
> whether we keep it limited to UTF8, or extend it somehow. I'm CCing
> Philippe Proulx who is driving this effort.

Thanks again! 

Bye

-- 
Milian Wolff | milian.wolff at kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5903 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160920/6a638493/attachment.bin>