[lttng-dev] Workflow to view old and current trace data

[Benjamin Poirier]

Hi, long time no see!

Consider a system that's recording trace data continuously from a
userspace application. A problem is noticed and I'd like to investigate
it. What workflow would allow me to view past trace data to analyze the
cause of the problem and view current (live) trace data while working on
fixing the problem, all the while keeping a continuous record of events
(ie. not stopping the trace)?

I thought of the following approaches but both seem to have
disadvantages:
1)
Have one continuous tracing session in normal mode. When a problem is
noticed, use lttng-rotate to be able to read old trace data. Start a
second tracing session in live mode with the same event rules to view
live trace data. IIUC, there's no option to prevent lttng-relayd from
writing the traces to disk, so we end up writing two sets of identical
traces to disk during the time we want to look at live trace data.
2)
Have one tracing session running in live mode all the time. When a
problem is noticed, use the viewer on whatever lttng-relayd has flushed
to disk to access old trace data (session rotation is not available).
Furthermore, this workflow has the disadvantage that the trace data is
going through lttng-relayd all the time (which I guess is less efficient
than writing it from lttng-consumerd) merely to support the rare case
where data needs to be seen live while analyzing a problem.

Is there some better approach that I didn't think about? What's your
recommendation to support this workflow?

I'm comparing lttng to the situation where the userspace application is
writing its logs directly to a file. In that case, it's simple to read
the file to access old and "live" data. Of course I realize that lttng
shines in other areas where simple file writing does not ;)

Thank you,
-Benjamin