[lttng-dev] [LTTNG-TOOLS PATCH] On-disk multiple tracefiles circular buffer

[Thibault, Daniel]

-----Message d'origine-----
From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>

> I having similar questions as Daniel about tracefile-size and tracefile-count.
>
> For each stream, do we have max size used on disk = size * count or size used = size ?

   My understanding is that the maximum size on disk of a stream (channel) would be size*count*CPUs*processes, where processes is the number of user-space processes emitting events on that channel (this number can theoretically be quite large); for the kernel this number is 1.

> If there is a doubt about the semantic, maybe using clearer terms that don't need to be explained might help. I guess the confusion here is:
> what is a tracefile exactly ? The notion of "stream" is much clearer than "tracefile" in the CTF spec, so we might want to consider:
>
>--stream-chunks-num NUM: each stream, on disk, is divided in at most
>   NUM chunks (one file per chunk). When the maximum number of chunks is
>   reached, oldest chunks are deleted for this stream, thus creating
>   a flight recorder on disk.
>
>--stream-chunks-size SIZE: chunk size. Must be a multiple of
>   subbuf-size. Default value: subbuffer size.

   According to the CTF spec ("Event Stream: An ordered sequence of events, containing a subset of the trace event types."), an event stream is an LTTng channel.  This is clearly not the same thing as an on-disk file (and its corresponding file stream).  This would mean:

--stream-chunks-num NUM: each file stream (a channel, CPU ID, originating process ID triplet),
   on disk, is divided in at most NUM chunks (one file per chunk).  When the maximum
   number of chunks is reached, the oldest chunks are overwritten by this stream, thus
   creating a circular flight recorder on disk.

> Actually, you might want to consider that the stream-chunk-size need to be a multiple of the subbuffer size. The check needs to be more strict.

   Good idea.  Not necessarily a power of two, though.

##########

   On a somewhat related topic, I find the CTF (1.8.2) spec confusing in sections 5 and 6:

> 5. Event Packet Header
>
> The event packet header consists of two parts: the "event packet header"
> is the same for all streams of a trace. The second part, the "event
> packet context", is described on a per-stream basis. Both are described
> in the TSDL meta-data. The packets are aligned on architecture-page-sized
> addresses.
[...]
> 5.1 Event Packet Header Description
[...]
> 5.2 Event Packet Context Description
[...]
> 6. Event Structure
>
> The overall structure of an event is:
>
> 1 - Stream Packet Context (as specified by the stream meta-data)
>  2 - Event Header (as specified by the stream meta-data)
>   3 - Stream Event Context (as specified by the stream meta-data)
>    4 - Event Context (as specified by the event meta-data)
>     5 - Event Payload (as specified by the event meta-data)
[...]
> 6.1 Event Header
[...]
> 6.1.1 Type 1 - Few event IDs
[...]
> 6.1.2 Type 2 - Many event IDs
[...]
> 6.2 Event Context
>
> The event context contains information relative to the current event.
> The choice and meaning of this information is specified by the TSDL
> stream and event meta-data descriptions. The stream context is applied
> to all events within the stream. The stream context structure follows
> the event header. The event context is applied to specific events. Its
> structure follows the stream context structure.
[...]
> 6.3 Event Payload
>
> An event payload contains fields specific to a given event type. The fields
> belonging to an event type are described in the event-specific meta-data
> within a structure type.

   The babeltrace output labels the following structured fields in each event record:

stream.packet.context
stream.event.header
stream.event.context
event.context
event.fields

   They match section 6:

> 1 - Stream Packet Context (as specified by the stream meta-data)
>  2 - Event Header (as specified by the stream meta-data)
>   3 - Stream Event Context (as specified by the stream meta-data)
>    4 - Event Context (as specified by the event meta-data)
>     5 - Event Payload (as specified by the event meta-data)

   But where does section 5 fit in?  There is no "Event Packet Header" or "Event Packet Context" in the above.  Section 3 mentions that "The event stream header will therefore be referred to as the 'event packet header' throughout the rest of this document."  Judging by the C structs described, the lack of a Stream Packet Context struct in section 6, and the babeltrace output, I'm pretty sure that the "stream packet context" is the "event packet context" and that it also subsumes the "event packet header" (a.k.a. "event stream header").  But it would be nice if this were actually stated in the CTF spec.

   I would also like to know if LTTng can populate the "Event Context" (6.4) of a trace, or if this is something only other CTF producers may do.  I haven't yet been able to produce any traces that have that field in them, and don't see how to do so.

Daniel U. Thibault
R & D pour la défense Canada - Valcartier (RDDC Valcartier) / Defence R&D Canada - Valcartier (DRDC Valcartier)
Cyber sécurité pour les missions essentielles (CME) / Mission Critical Cyber Security (MCCS)
Protection des systèmes et contremesures (PSC) / Systems Protection & Countermeasures (SPC)
2459 route de la Bravoure
Québec, QC  G3J 1X5
CANADA
Vox : (418) 844-4000 x4245
Fax : (418) 844-4538
NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
Gouvernement du Canada / Government of Canada
<http://www.valcartier.drdc-rddc.gc.ca/>