[lttng-dev] User-space tracing by the root lttng-sessiond

[Mathieu Desnoyers]

* Thibault, Daniel (Daniel.Thibault at drdc-rddc.gc.ca) wrote:
>    I ran a little experiment whose results surprised me.  Using a non-tracing group user account, I first made sure I had a root lttng-sessiond running and a user-space lttng-sessiond.  In the first I set up a trace for some kernel events plus all user-space events.  In the second I set up a trace for all user-space events only.  Both traces were started, then the lttng-ust/doc/examples/easy-ust/sample demo was run to generate the user-space events.  Then both traces were stopped and destroyed.
> 
>    I noted there were three lttng-consumerd daemons: one in user-space, the other two owned by root.  Presumably one is servicing the kernel events while the other is servicing the user-space events.

yes

>  (Question: If there were several users active at once, would there be several root user-space lttng-consumerd daemons or still just one?)

no

> 
>    As expected, both traces show the sample user-space events (with no losses, it turns out).  What surprised me, however, is that the pairs of events were not in synch. Since they're all coming out of the same trace provider, I would expect the time-stamp to be issued once (by the tracepoint provider), and then two copies of the event to go into the two trace buffers for eventual storage.  Could it be that the tracepoint provider instead issues two separate events, one for each of its clients?

yes

>  (For the record, the pairs of events were out of synch by at least 4187 ns (670 times out of 100 000), and as much as 3 326 405 ns (once); in 93% of the time root event lags after the user-space event)

can be caused by interruption (for 4k ns) and preemption (for the 3s
hit).

> 
>    This seems problematic to me, since multiple traces capturing the same event won't be able to reconcile their copies with each other.

since timestamps have to match the event physical order in the streams,
we need to assign one timestamp for each buffer. no way around this.

and yes, tracing is not stopping the world, so you can expect to have
slight timing inaccuracy like this. It's a trade-off between perfect
timing accuracy and tracer overhead.

Thanks,

Mathieu


> 
> Daniel U. Thibault
> Protection des systèmes et contremesures (PSC) | Systems Protection & Countermeasures (SPC)
> Cyber sécurité pour les missions essentielles (CME) | Mission Critical Cyber Security (MCCS)
> R & D pour la défense Canada - Valcartier (RDDC Valcartier) | Defence R&D Canada - Valcartier (DRDC Valcartier)
> 2459 route de la Bravoure
> Québec QC  G3J 1X5
> CANADA
> Vox : (418) 844-4000 x4245
> Fax : (418) 844-4538
> NAC : 918V QSDJ <http://www.travelgis.com/map.asp?addr=918V%20QSDJ>
> Gouvernement du Canada | Government of Canada
> <http://www.valcartier.drdc-rddc.gc.ca/>
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com