[lttng-dev] Discarded events

[Erica Bugden]

On 2023-08-13 10:58, Bala Gundeboina via lttng-dev wrote:
> Hi,
> 
> I am Using LTTng in my project , I am recording a kernel session for 
> long duration (eg more than 15minutes) I am enabling all the events for 
> recording but some events discarding after stopping the session, if i 
> run more than 5 minutes only some events are discarding. I have tried to 
> change in the .ltttngrc file also to increase buffer-size.
> 
> lttng destroy
> Destroying session `my-kernel-session`...
> Session `my-kernel-session` destroyed
> Warning: 94424667 events were discarded, please refer to the 
> documentation on channel configuration
> 

Hello Bala,

Based on my understanding of the scenario, discarded events are to be 
expected. Rather than trying to avoid discarded events, I would 
recommend adjusting how much data is being collected. In most tracing 
use cases, it is not relevant to enable all kernel tracepoints for a 
long period of time (e.g. several minutes) as it can quickly generate an 
enormous amount of noisy data that is nearly impossible to sort through. 
As a reference, a busy 8 core machine could generate 100 MB/s of data.

Typically when trying to understand a problem in detail with tracing, 
you would start with a very small number of tracepoints enabled for a 
longer period of time (with the goal of developing a high-level 
understanding of when/where the problem is happening). Then as you 
iteratively narrow down when/where the problem happens you can gradually 
increase the number of relevant tracepoints without being overwhelmed 
with data.

A general rule of thumb is that if you're storing trace data for a long 
period of time (e.g. more than a minute) then very few tracepoints 
should be enabled (e.g. 2-5). Maximum tracing detail would typically 
only be used to trace for a couple seconds (or ideally less if you can 
automate starting and stopping tracing).

Here are some general references about tracing strategy:
  - Whether tracing is the appropriate approach: 
https://github.com/tuxology/tracevizlab/tree/master/labs/001-what-is-tracing#when-to-trace
  - Iterative investigation, selecting tracepoints: 
https://wiki.linuxfoundation.org/realtime/documentation/howto/debugging/debug-steps 
(Recommended sections: Isolate the source, Trace detail)

Hope this helps!

Best,
Erica

> actually when i am running below command i am getting below warning i am 
> thinking this is the issue because sometimes this warning is not coming 
> that i didn't seen discarded events.  how to overcome this problem can 
> you provide some detailed
> lttng-sessiond --daemonize
> Warning: Failed to produce a random seed using getrandom(), falling back 
> to pseudo-random device seed generation which will block until its pool 
> is initialized: Failed to get true random data using getre
> 

At first glance, this error seems unrelated to whether or not events are 
discarded.

> Thanks & Regards
> Bala Gundeboina
> This message contains information that may be privileged or confidential 
> and is the property of the KPIT Technologies Ltd. It is intended only 
> for the person to whom it is addressed. If you are not the intended 
> recipient, you are not authorized to read, print, retain copy, 
> disseminate, distribute, or use this message or any part thereof. If you 
> receive this message in error, please notify the sender immediately and 
> delete all copies of this message. KPIT Technologies Ltd. does not 
> accept any liability for virus infected mails.
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev