[lttng-dev] Human read-writeable format for CTF traces

[Mathieu Desnoyers]

----- Original Message -----
> From: "Geneviève Bastien" <gbastien+lttng at versatic.net>
> To: lttng-dev at lists.lttng.org
> Sent: Monday, February 3, 2014 9:59:34 AM
> Subject: [lttng-dev] Human read-writeable format for CTF traces
> 
> Hello all,
> 
> Short question:
> 
> As we develop analyses in TMF for LTTng traces, we'd like to unit test
> them with very specific sequences of events, something no real-life
> trace can provide. We'd like to build our own traces manually.
> 
> Does a human read-writeable format already exist to work with CTF traces
> or convert from/to CTF traces? I heard of a few CTF trace generators,
> some which haven't been rebased in a while. What's their status?
> Anything that answers our need there?
> 
> Thanks,
> 
> ==================
> Detailed specification of what I'm looking for and why:
> 
> * In TMF, analyses use event requests and they handle events. It does
> not matter at all whether the event came from a CTF binary stream or
> from a text file, as long as they have the same name and fields, it will
> be handled the same.
> 
> * Also, while we may want to hand make a full test trace, we may also
> wish to take (part of) a real life CTF trace, convert it to a human
> read-writeable format, tweak it to our needs and... as far as TMF is
> concerned, we'll be perfectly happy with this format, but it may
> eventually be desirable to convert it back to a CTF trace.
> 
> * XML-defined analyses are about to make it into TMF master. That means
> users will be able to develop in XML their own state providers, but
> eventually also their own views, filters, analyses, etc. Before trying
> them on real traces, it must be possible to test them on controlled
> hand-made traces. So that trace format is not just a feature for unit
> tests, it should be made available in the main TMF for users to use.
> 
> * We must keep in mind that users of this format may not have access to
> the complete LTTng toolchain, or even to a linux command line as they
> may be working... on Windows! That's even more true now that work is
> being done to convert Windows traces to CTF format. So the solution must
> be completely standalone in TMF.
> 
> * In a quick prototype last week, I came up with this format for the
> human read-writeable test traces:
> 
> <trace>
> <event>
> <timestamp value="1" />
> <eventName value="A" />
> <source value="0" /> /** the cpu */
> <field name="b" value="1" type="int" />
> <field name="c" value="abcdef" type="string" />
> </event>
> </trace>
> 
> Why XML? Because it's easy to validate with a XSD and we can use
> existing tool to automatically generate from this XSD a user interface
> to edit the events. But that's just a suggestion.
> 
> * Lastly and not least, whatever we end up implementing in TMF must use
> or at least play nice with other tools around CTF that do more or less
> the same thing. I wouldn't want to reinvent the wheel here.
> 
> Any thoughts?  Any thing I should look at?

Hi Geneviève,

Jérémie will probably have more to add when he gets back from FOSDEM, but
I would expect that the ctf writer API recently added to babeltrace
(currently in master branch), along with the Python bindings that cover
trace read and write APIs, should allow you to implement things like:

- A plugin to read a CTF trace, and output it in an intermediate format
  to facilitate edits (e.g. XML as you propose),
- A plugin to read this XML format and output a CTF trace.

You could also generate the XML trace completely by hand if you like, and
then convert it to CTF with the second plugin I'm relating to above.

Another possibility is that the XML description also allows describing
what the trace contains at a slightly higher level. For instance, if you
have a periodic event happening for a certain amount of time, it would
be described in XML, and then "generated" by the XML-to-CTF converter.

Indeed, having this kind of infrastructure for unit-testing is going to
be *extremely* useful to validate the state system.

Thanks!

Mathieu


> 
> Thanks,
> Geneviève
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> 

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com