[lttng-dev] CTF semantics

[Milian Wolff]

On Tuesday, June 14, 2016 4:50:58 PM CEST Mathieu Desnoyers wrote:
> ----- On Jun 14, 2016, at 12:31 PM, Milian Wolff milian.wolff at kdab.com 
wrote:
> > On Tuesday, June 14, 2016 4:10:46 PM CEST Mathieu Desnoyers wrote:
> >> ----- On Jun 14, 2016, at 7:09 AM, Milian Wolff milian.wolff at kdab.com 
wrote:
> >> > Hey all,
> >> > 
> >> > I have looked through the CTF specification and ponder using it to
> >> > replace
> >> > my custom text-based output format of heaptrack.
> >> 
> >> Very cool!
> >> 
> >> > At this stage, I have a fundamental question: How do the existing
> >> > viewers
> >> > like Trace Compass understand the semantics of the data? Or are the
> >> > viewers not generic but instead rely on the existing generators like
> >> > lttng? How does one know e.g. what the backtrace of a given event is?
> >> 
> >> CTF only specifies the data layout and associates events/fields to names
> >> (namespacing). The analyses associate meaning to the information gathered
> >> by using the namespace associated to the tracer that collected the trace,
> >> event and field names.
> >> 
> >> For a lttng backtrace (we have a ongoing work prototype branch here which
> >> requires frame pointers for user-space applications:
> >> https://github.com/compudj/lttng-modules-dev/commits/callstack), we can
> >> associate the callstack_user context name to this callstack concept, and
> >> if the trace viewer wants to really show this as a callstack (linked with
> >> debug information to find the function names for instance), it needs to
> >> know the semantic of this new context field for LTTng.
> >> 
> >> With the upcoming CTF 2.0, we plan on adding much more flexibility to the
> >> spec, so we could declare user attributes that would "flag" a specific
> >> aspect of the semantic, across various tracers. But we intend to leave
> >> the
> >> tracers express their own semantic as much as possible, and then
> >> eventually
> >> agree on common sets of constructs that are found in many
> >> implementations,
> >> perhaps to create a side-spec of "standard user attributes" in the
> >> future.
> > 
> > Great, thanks for the in-depth explanation.
> > 
> > One off-topic question: You say call stacks require frame pointers, why?
> > libunwind can unwind based on DWARF debug information. Sure, on embedded
> > you don't want that, but on a desktop that is just fine. Or did you
> > reinvent the unwinding and don't use libunwind?
> 
> For lttng-modules (kernel tracer): I would ideally like to implement
> libunwind within the kernel to unwind user-space stack. I know systemtap has
> something that does this, but its performance seems rather slow, and having
> to pass each ELF file to consider explicitly before tracing is cumbersome.
> We work with a student at Ecole Polytechnique currently prototyping on this
> topic.
> 
> For lttng-ust: libunwind appears to be too slow, and we would like to
> make this reentrant wrt signal handlers, and not have to share state
> (locks) between cores, for scalability reasons.

As I'm heavily using libunwind in heaptrack, I'd be interested in this work. 
It does the job for me and is quite fast for what it does - i.e. I can unwind 
the stack millions of times per second using DWARF information which is quite 
a feat, imo. So if you ever have an unwinder that is even faster than 
libunwind, I'm very much interested in learning about that.

> >> CTF 2.0 will keep the data streams as-is, and change the metadata format
> >> from TSDL (custom grammar) to JSON.
> > 
> > Good choice, but JSON does not allow comments. Did you think about that? I
> > haven't used CTF at all yet, but I could think of cases where one wants to
> > add a comment to a complicated grammar. Maybe YAML is a better choice for
> > that reason.
> 
> Philippe could tell us more on this topic.
> 
> >> > In heaptrack's current format heavily interns data to greatly reduce
> >> > the
> >> > file size of the output data. This is crucial, and can be done with
> >> > minimal overhead. So I'd like to do the same if and when I convert to
> >> > using CTF. But how would e.g. know how to interpret that an integer
> >> > member of a struct actually is an index into a list of backtraces?
> >> 
> >> Just trying to understand here. So you store a backtrace once in the
> >> trace,
> >> associate it with a unique number, and later on, if you need to save the
> >> same backtrace, you just use this number instead ?
> > 
> > Basically, yes. I actually go even further, and intern also the parts of
> > the trees, e.g.:
> > 
> > A
> > 
> >|- B
> >|
> >   |- C
> >   |- D
> >|
> >|- E
> >|
> >   |- F
> > 
> > If we assume the leafs of this tree would trigger events with call stacks,
> > then I'd intern the data such that I only output the debug data for every
> > item in the tree once. I.e. I don't do
> > 
> > A | B | C
> > A | B | D
> > A | E | F
> > 
> > Instead, I essentially store it as
> > 
> > 1: A 0
> > 2: B 1
> > 3: C 2
> > 4: D 2
> > 5: E 1
> > 6: F 5
> > 
> > This is enough to rebuild the call stacks, and reduces the report size
> > dramatically. E.g. an allocation of 9 bytes from D is represented as
> > 
> > a 9 4
> > + <id of above line>
> 
> OK. One main question I have is whether you would like to deal with
> lost CTF packets or not ? There are a few possible approaches there:

For my use case of heaptrack, I don't ever plan to lose any packets. I'm fine  
with having a noticeable (but low) slow-down during tracing, as I'm doing this 
on one application at a time (and not the full system). If events would be 
lost, the data cannot be reliably interpreted afterwards. I.e. if you have

malloc(1)
malloc(100);
malloc(1);

and the event for the second allocation is lost, the result would be 
completely wrong.

> 1) you keep track of packet sequence number and discarded event counts,
>    and tell the user when there is missing information,
> 
> 2) you dump the entire table at the beginning of each CTF packet (e.g.
>    as a packet context field). Could be achievable if not too large.
> 
> 3) you "reset" the information about which state has been dumped at the
>    beginning of each packet, so each packet is self-contained state-wise.
> 
> Dealing with lost packets is useful if you plan to use flight-recorder
> (snapshot) type of tracing, where the beginning of the trace is often
> overwritten when capturing the trace snapshot.

I don't think creating snapshots are record time are desired for my use-case. 
I much rather want to record everything and have the GUI then to diff between 
time points (i.e. snapshots).

> With respect to the event field type to use, I would probably use a variant.
> 
> For instance:
> 
> enum : uint32_t {
>   "id" = 0 .. 4294967294,
>   "map" = 4294967295,
> } f1;
> 
> variant <f1> {
>   struct {} id;  /* empty field */
>   struct {
>     uint32_t id;
>     string func;
>   } map;
> } f2;
> 
> The above would typically store a 32-bit "id" (the common case), and
> reserve the value 4294967295 to indicate that a non-empty variant
> follows, which has a mapping between id and function string.
> 
> Would that fit your use-case ?

Potentially, I don't think I have enough experience to answer this without 
trying it out first.

Thanks
-- 
Milian Wolff | milian.wolff at kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5903 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160614/e55af8ba/attachment-0001.bin>