[ltt-dev] [RFC] Common Trace Format Requirements (v1.3)

[Spear, Aaron]

> > > > If the metadata is not given in a standard form, then how do
> > > > envision general trace analysis tools (those not hard-coded for
> > > > some particular trace source) working?
> >
> > > We can have a metadata format selector at the beginning of the
> > > metadata section, with reserved IDs for metadata formats. We can
> > > think of a format generated natively by TRACE_EVENT(), a format
> > > generated in some sort of XML. The trace analyzer would need the
> > > metadata format parser in order to be able to read the trace.
> >
> > If one hopes that such tools should be able to consume data with
> > drifting versions of TRACE_EVENT() or XML or whatnot, they
themselves
> > had better be fixed/standardized.  Otherwise, the data will not be
> > self-describing, and all the tools will have to chase kernel/etc.
> > versions.
> 
> The tool can bail out if it detects a type it does not know. So a tool
> upgrade would be required.
> 
> Also the standard plans to have a major/minor trace version in there,
> so we can explicitly break compatibility if it is required. It might
> end up being much better than trying to support backward compatibility
> forever. Then it would be up to the tool implementors to choose if
they
> want to provide backward compatibility for older trace versions or
not.

What we have done in our own internal format (which I distributed for
reference some time ago) is have a fixed set of fundamental types that
can be inserted into the trace log (integers of various bit sizes,
strings, Booleans, etc).  An event in the log is a collection of these
fundamental types then,  (e.g. a semaphore post event might have a 16
bit unsigned event id, a 32 bit unsigned semaphore handle, 32 bit
unsigned current thread id, 64 bit timestamp, ...).  So at a base layer
there can be a trace analyzer that simply knows how to display
attributes of events but doesn't really know what they MEAN.  On top of
that you can layer plugin functionality in a trace analyzer that
understands what a given type of event means and as such what sort of
thing makes the most sense to do in a presentation.  Scheduling is a
good example.  If there are scheduler events that indicate context
switches, then if the trace analyzer knows this, it can use that
information to help it draw a nice Gantt chart as opposed to a linear
graph of simple event data.

I have been hoping that as a part of this standard that we will end up
with common schemas for meta-data that are shared by various use cases,
so that things that are common concepts to many OS'es for example could
in fact be shared and thus you could have a trace analyzer that works
for OS'es that have similar models and generate data in a compatible
schema.  In practice this may be difficult, but it seems like a noble
goal to aim for.  If we can get a fixed schema for the fundamental types
that will be a step in the right direction. 

Best regards,
Aaron Spear
Tools Architect, Mentor Graphics