[lttng-dev] Running trace-rotate and session-rotate together

[Jérémie Galarneau]

----- Message original -----
> De: "lttng-dev" <lttng-dev at lists.lttng.org>
> À: "lttng-dev" <lttng-dev at lists.lttng.org>
> Envoyé: Mardi 21 Décembre 2021 01:47:12
> Objet: [lttng-dev] Running trace-rotate and session-rotate together

> Hi,
> 1. In the lttng-enable-rotation manpage it is mentioned that we should not use
> trace rotate when automatic session rotate is enabled. Can you please tell me
> why?
> ```
> Important
> You may only use the enable-rotation command when:
> * The selected recording session was created in normal mode or in network
> streaming mode
> (see lttng-create(1)).
> * No channel was created with a configured trace file count or size limit (see
> the
> --tracefile-size and --tracefile-count options of the lttng-enable-channel(1)
> command).
> ```

Hi, 

The reason mostly boils down to the fact that we didn't see a use case for this. Hence, it's not implemented nor tested. 

Let me know if you have a use case for combining both features, though.

> 2. When automatic session rotate is running, is there any race condition in
> writing the current ring buffers and archiving the current trace chunk?

I am not sure I understand your question, but I think you're asking if session rotations are "atomic" with respect to the switchover of all per-CPU streams from one trace archive to the next?

The switchover is not instantaneous. The file output of each per-CPU stream is switched to the new trace archive sequentially. That leaves you with a window of time during which information, for a given stream, may be found in either the old trace archive or the new one. To be clear, no information is ever duplicated or lost during this operation. 

I briefly touched on this topic in a presentation at the Open Source Summit 2018, see slide 32+[1]

The various colours represent successive trace archives produced by session rotations.

This transition window makes writing analyses a bit harder since the beginning and end of trace archives may not have all of the information for all per-CPU streams during a short window of time. That window is contained between issuing the rotation and receiving a confirmation that it has been completed (using the CLI or the liblttng-ctl API). 

You can see that in that context, I chose to process trace archives in pairs to constitute an "effective trace archive" (slide 34) during which I can be sure I always have access to all events from all per-CPU streams.

Thanks,
Jérémie

[1] https://events19.linuxfoundation.org/wp-content/uploads/2017/12/Fine-grained-Distributed-Application-Monitoring-Using-LTTng-J%C3%A9r%C3%A9mie-Galarneau-EfficiOS.pdf

> --------------------------------------------------------------
> Regards,
> Subrata Paul

> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev