[lttng-dev] CTF semantics

[Mathieu Desnoyers]

----- On Jun 14, 2016, at 12:31 PM, Milian Wolff milian.wolff at kdab.com wrote:

> On Tuesday, June 14, 2016 4:10:46 PM CEST Mathieu Desnoyers wrote:
>> ----- On Jun 14, 2016, at 7:09 AM, Milian Wolff milian.wolff at kdab.com wrote:
>> > Hey all,
>> > 
>> > I have looked through the CTF specification and ponder using it to replace
>> > my custom text-based output format of heaptrack.
>> 
>> Very cool!
>> 
>> > At this stage, I have a fundamental question: How do the existing viewers
>> > like Trace Compass understand the semantics of the data? Or are the
>> > viewers not generic but instead rely on the existing generators like
>> > lttng? How does one know e.g. what the backtrace of a given event is?
>> 
>> CTF only specifies the data layout and associates events/fields to names
>> (namespacing). The analyses associate meaning to the information gathered
>> by using the namespace associated to the tracer that collected the trace,
>> event and field names.
>> 
>> For a lttng backtrace (we have a ongoing work prototype branch here which
>> requires frame pointers for user-space applications:
>> https://github.com/compudj/lttng-modules-dev/commits/callstack), we can
>> associate the callstack_user context name to this callstack concept, and
>> if the trace viewer wants to really show this as a callstack (linked with
>> debug information to find the function names for instance), it needs to
>> know the semantic of this new context field for LTTng.
>> 
>> With the upcoming CTF 2.0, we plan on adding much more flexibility to the
>> spec, so we could declare user attributes that would "flag" a specific
>> aspect of the semantic, across various tracers. But we intend to leave the
>> tracers express their own semantic as much as possible, and then eventually
>> agree on common sets of constructs that are found in many implementations,
>> perhaps to create a side-spec of "standard user attributes" in the future.
> 
> Great, thanks for the in-depth explanation.
> 
> One off-topic question: You say call stacks require frame pointers, why?
> libunwind can unwind based on DWARF debug information. Sure, on embedded you
> don't want that, but on a desktop that is just fine. Or did you reinvent the
> unwinding and don't use libunwind?

For lttng-modules (kernel tracer): I would ideally like to implement
libunwind within the kernel to unwind user-space stack. I know systemtap has
something that does this, but its performance seems rather slow, and having
to pass each ELF file to consider explicitly before tracing is cumbersome.
We work with a student at Ecole Polytechnique currently prototyping on this
topic.

For lttng-ust: libunwind appears to be too slow, and we would like to
make this reentrant wrt signal handlers, and not have to share state
(locks) between cores, for scalability reasons.

> 
>> CTF 2.0 will keep the data streams as-is, and change the metadata format
>> from TSDL (custom grammar) to JSON.
> 
> Good choice, but JSON does not allow comments. Did you think about that? I
> haven't used CTF at all yet, but I could think of cases where one wants to add
> a comment to a complicated grammar. Maybe YAML is a better choice for that
> reason.

Philippe could tell us more on this topic.

> 
>> > In heaptrack's current format heavily interns data to greatly reduce the
>> > file size of the output data. This is crucial, and can be done with
>> > minimal overhead. So I'd like to do the same if and when I convert to
>> > using CTF. But how would e.g. know how to interpret that an integer
>> > member of a struct actually is an index into a list of backtraces?
>> 
>> Just trying to understand here. So you store a backtrace once in the trace,
>> associate it with a unique number, and later on, if you need to save the
>> same backtrace, you just use this number instead ?
> 
> Basically, yes. I actually go even further, and intern also the parts of the
> trees, e.g.:
> 
> A
>|- B
>   |- C
>   |- D
>|- E
>   |- F
> 
> If we assume the leafs of this tree would trigger events with call stacks,
> then I'd intern the data such that I only output the debug data for every item
> in the tree once. I.e. I don't do
> 
> A | B | C
> A | B | D
> A | E | F
> 
> Instead, I essentially store it as
> 
> 1: A 0
> 2: B 1
> 3: C 2
> 4: D 2
> 5: E 1
> 6: F 5
> 
> This is enough to rebuild the call stacks, and reduces the report size
> dramatically. E.g. an allocation of 9 bytes from D is represented as
> 
> a 9 4
> + <id of above line>

OK. One main question I have is whether you would like to deal with
lost CTF packets or not ? There are a few possible approaches there:

1) you keep track of packet sequence number and discarded event counts,
   and tell the user when there is missing information,

2) you dump the entire table at the beginning of each CTF packet (e.g.
   as a packet context field). Could be achievable if not too large.

3) you "reset" the information about which state has been dumped at the
   beginning of each packet, so each packet is self-contained state-wise.

Dealing with lost packets is useful if you plan to use flight-recorder
(snapshot) type of tracing, where the beginning of the trace is often
overwritten when capturing the trace snapshot.

With respect to the event field type to use, I would probably use a variant.

For instance:

enum : uint32_t {
  "id" = 0 .. 4294967294,
  "map" = 4294967295,
} f1;

variant <f1> {
  struct {} id;  /* empty field */
  struct {
    uint32_t id;
    string func;
  } map;
} f2;

The above would typically store a 32-bit "id" (the common case), and
reserve the value 4294967295 to indicate that a non-empty variant
follows, which has a mapping between id and function string.

Would that fit your use-case ?

Thanks,

Mathieu


> 
> Bye
> --
> Milian Wolff | milian.wolff at kdab.com | Software Engineer
> KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
> Tel: +49-30-521325470
> KDAB - The Qt Experts

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com