[lttng-dev] Latency tracing using LTTng

[Jonathan Rajotte-Julien]

Hi,

On Wed, Dec 27, 2017 at 05:21:30PM -0800, Anup Pemmaiah wrote:
> Hi,
> 
> I am trying to find the reason of a intermittent latency that I am noticing
> on my Linux system running on a RT_PREEMPT kernel. I have patched it on a
> Ubuntu 16.04 LTS. Have stopped the desktop lightdm service and this is on a

Could we get more details on the version of the RT kernel? Version/patchset?

> Arm aarch64 platform.
> 
> At a very high level, there are two processes interacting with each other.
> Process-1 does some work and sends UDP messages to Process-2 (Using the
> FASTRTPS real time Publish-Subscribe protocol).

Are you using the eProsima implementation? If so, could you give us more
information on the tracing strategy?

> Data is sent from Process-1 to Process-2 every 100ms. I see that often
> there is a delay in Process-2 receiving the message.
> 
> 
> Trying to use LTTng and along with other supported profiling tools like
> latency-tracker, lttng-profile.
> Something like the description in this link
> http://lttng.org/blog/2015/03/18/full-stack-latencies is what I am looking
> for, but I am facing issues. The latency_tracker kernel modules and also
> lttng-profile is a bit flaky as it sometimes work and sometimes panics the
> kernel.  Simple test cases as provided in
> https://github.com/fdoray/latency_tracker_demo works sometimes. And also

Not sure if those aged properly.

If you did not already, I would recommend you take a look at the latency-tracker
github page [1] that this project references and uses. It changed quite a lot
in the last 3 years.

[1] https://github.com/efficios/latency-tracker


> profiling Process-1 works by providing some useful traces(intermittently
> kernel panics), but profiling Process-2 the process just hangs.

What version of lttng-modules are you using? The latency_tracker_demo install
script seems to use lttng-modules 2.6. 2.6 is not supported anymore, it would be
interesting to perform you experimentation with a newer version.

Still, could you provide the panics call stack?

> 
> Not sure how updated the above pasted links are, because I notice there is
> no more usec_threshold parameter and I use the threshold
> in /sys/kernel/debug/latency/syscalls/default/threshold
> 
> I understand each situation is different, but was curious if  any of you
> have comments/suggestions that would help in debugging latency related
> issues using LTTng. I have looked many places to get required info but of
> not much help.

Most of the maintainers are on vacation during this part of the year so it might
take some time to get answers. I'm sure that Julien Desfossez will be
interested in your use case.

Cheers

> 
> Your comments/suggestions will be greatly appreciated.
> 
> Thanks
> Anup

> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


-- 
Jonathan Rajotte-Julien
EfficiOS