[lttng-dev] CTF semantics

[Milian Wolff]

On Tuesday, June 14, 2016 4:10:46 PM CEST Mathieu Desnoyers wrote:
> ----- On Jun 14, 2016, at 7:09 AM, Milian Wolff milian.wolff at kdab.com wrote:
> > Hey all,
> > 
> > I have looked through the CTF specification and ponder using it to replace
> > my custom text-based output format of heaptrack.
> 
> Very cool!
> 
> > At this stage, I have a fundamental question: How do the existing viewers
> > like Trace Compass understand the semantics of the data? Or are the
> > viewers not generic but instead rely on the existing generators like
> > lttng? How does one know e.g. what the backtrace of a given event is?
> 
> CTF only specifies the data layout and associates events/fields to names
> (namespacing). The analyses associate meaning to the information gathered
> by using the namespace associated to the tracer that collected the trace,
> event and field names.
> 
> For a lttng backtrace (we have a ongoing work prototype branch here which
> requires frame pointers for user-space applications:
> https://github.com/compudj/lttng-modules-dev/commits/callstack), we can
> associate the callstack_user context name to this callstack concept, and
> if the trace viewer wants to really show this as a callstack (linked with
> debug information to find the function names for instance), it needs to
> know the semantic of this new context field for LTTng.
> 
> With the upcoming CTF 2.0, we plan on adding much more flexibility to the
> spec, so we could declare user attributes that would "flag" a specific
> aspect of the semantic, across various tracers. But we intend to leave the
> tracers express their own semantic as much as possible, and then eventually
> agree on common sets of constructs that are found in many implementations,
> perhaps to create a side-spec of "standard user attributes" in the future.

Great, thanks for the in-depth explanation.

One off-topic question: You say call stacks require frame pointers, why? 
libunwind can unwind based on DWARF debug information. Sure, on embedded you 
don't want that, but on a desktop that is just fine. Or did you reinvent the 
unwinding and don't use libunwind?

> CTF 2.0 will keep the data streams as-is, and change the metadata format
> from TSDL (custom grammar) to JSON.

Good choice, but JSON does not allow comments. Did you think about that? I 
haven't used CTF at all yet, but I could think of cases where one wants to add 
a comment to a complicated grammar. Maybe YAML is a better choice for that 
reason.

> > In heaptrack's current format heavily interns data to greatly reduce the
> > file size of the output data. This is crucial, and can be done with
> > minimal overhead. So I'd like to do the same if and when I convert to
> > using CTF. But how would e.g. know how to interpret that an integer
> > member of a struct actually is an index into a list of backtraces?
> 
> Just trying to understand here. So you store a backtrace once in the trace,
> associate it with a unique number, and later on, if you need to save the
> same backtrace, you just use this number instead ?

Basically, yes. I actually go even further, and intern also the parts of the 
trees, e.g.:

A
|- B
   |- C
   |- D
|- E
   |- F

If we assume the leafs of this tree would trigger events with call stacks, 
then I'd intern the data such that I only output the debug data for every item 
in the tree once. I.e. I don't do

A | B | C
A | B | D
A | E | F

Instead, I essentially store it as

1: A 0
2: B 1
3: C 2
4: D 2
5: E 1
6: F 5

This is enough to rebuild the call stacks, and reduces the report size 
dramatically. E.g. an allocation of 9 bytes from D is represented as

a 9 4
+ <id of above line>

Bye
-- 
Milian Wolff | milian.wolff at kdab.com | Software Engineer
KDAB (Deutschland) GmbH&Co KG, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5903 bytes
Desc: not available
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160614/5e5df3e4/attachment.bin>