[lttng-dev] userspace logs cpu analysis and LTTng UST Callstack

[Erica Bugden]

On 2023-08-23 13:14, Matthew Khouzam via lttng-dev wrote:
> Here is a youtube video showing how userspace and kernel traces can play 
> nicely together.
> 
> https://www.youtube.com/watch?v=K1JNQ-HkC6w 

> ------------------------------------------------------------------------
> *From:* lttng-dev <lttng-dev-bounces at lists.lttng.org> on behalf of Bala 
> Gundeboina via lttng-dev <lttng-dev at lists.lttng.org>
> *Sent:* Wednesday, August 23, 2023 2:17 AM
> *To:* lttng-dev at lists.lttng.org <lttng-dev at lists.lttng.org>
> *Subject:* [lttng-dev] userspace logs cpu analysis and LTTng UST Callstack
> Hi all,
>              I have capturing kernel and userspace trace logs with LTTng 
> but kernel logs will give more useful information but userspace logs is 
> not giving more useful information compared to kernel trace logs, how to 
> get the more data from userspace trace logs. PFA of userspace and kernel 
> screenshots if you observe the kernel screenshot we will get per core 
> cpu usage, thread analysis and more .

Hello Bala,

I'm not sure exactly what you mean by, "userspace logs is not giving 
more useful information compared to kernel trace logs", but I would 
guess you mean: when reading the _userspace trace by itself_ in Trace 
Compass, there are way fewer visualisations available compared to when 
you view the _kernel trace by itself_ in Trace Compass.

It can be normal for a kernal trace to provide much more data than a 
userspace trace. Each visualisation in Trace Compass requires certain 
kinds of data (event types). The amount of events in a trace is a 
function of the number of tracepoints (the number of instrumented 
locations) in the code that are available and enabled. The kernel has a 
very large number of tracepoints that already come with the code. On the 
other hand, userspace applications often need to have the tracepoints 
added manually by the folks interested in tracing it. This means that 
userspace traces could have fewer events (if there are fewer available 
tracepoints).

One purpose of a userspace trace is to provide context for a kernel 
trace. To get the most information, you would capture both a kernel and 
userspace trace simultaneously and then view the traces together as a 
combined trace (rather that viewing them separately). When viewing the 
userspace and kernel information together you get the volume and 
richness of kernel information combined with the more "human-readable" 
abstract information from userspace that makes the kernel information 
easier to navigate.

As Matthew mentioned, the tool Trace Compass allows you to open the 
kernel and userspace traces together as a combined trace. Trace Compass 
uses the term "Experiment" to refer to a set of traces viewed together 
on a single timeline, but in practice this is a combined trace. To get 
the full analytical power of the kernel and userspace traces I would 
recommend creating a Trace Compass Experiment with the kernel and 
userspace traces. The video Matthew shared gives you information about 
how to combine and view the traces using an Experiment (see resources 
section below).

In the resources section below, I also included a link to a set of 
tracing analysis examples. I haven't checked, but there may be an 
example there that shows how to create a combined trace (Trace Compass 
Experiment). The Trace Compass user interface has likely changed since 
the screenshots were taken so you may not be able to follow the guide 
word-for-word, but I think it may still serve as a useful reference.

Best regards,

Erica

----

RESOURCES

- Creating a combined kernel and userspace trace using a Trace Compass 
Experiment (via Matthew Khouzam):
  https://www.youtube.com/watch?v=K1JNQ-HkC6w
  (A method of trace Experiment creation is shown starting at 0:58)

- Trace viz labs: Analyze a kernel trace in Trace Compass
https://github.com/dorsal-lab/Tracevizlab/tree/master/labs/101-analyze-system-trace-in-tracecompass 


- All Trace viz labs:
https://github.com/dorsal-lab/Tracevizlab/tree/master/labs

> 
> i am not able to load userspace trace logs in tracealyzer ,is 
> tracealyzer is not supported the userspace logs ? , is there anyway we 
> can make enable it ? kindly please me to sort out of this problem.

I'm not familiar with Tracealyzer so I'm not sure about its 
compatibility with these traces. I would suggest you reach out to the 
Tracealyzer folks to ask about compatibility with their tools.

Tracealyzer is a proprietary tool and my understanding is that on this 
mailing list we prefer to focus on free and open-source tools when 
possible (since information about them can be more widely used).

> 
> Thanks
> Bala Gundeboina
> This message contains information that may be privileged or confidential 
> and is the property of the KPIT Technologies Ltd. It is intended only 
> for the person to whom it is addressed. If you are not the intended 
> recipient, you are not authorized to read, print, retain copy, 
> disseminate, distribute, or use this message or any part thereof. If you 
> receive this message in error, please notify the sender immediately and 
> delete all copies of this message. KPIT Technologies Ltd. does not 
> accept any liability for virus infected mails.
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev