[lttng-dev] Regarding LTTng Support for ARM NO-HZ

[Mathieu Desnoyers]

----- Original Message -----

> From: "Hongbo Zhang" <hongbo.zhang at linaro.org>
> To: "mathieu desnoyers" <mathieu.desnoyers at efficios.com>,
> lttng-dev at lists.lttng.org
> Sent: Tuesday, March 18, 2014 4:07:19 AM
> Subject: Regarding LTTng Support for ARM NO-HZ

> Hi Mathieu Desnoyers and all LTTng developers,

Hi, 

> This is Hongbo Zhang, working for Linaro. We have to enable the NO-HZ feature
> for some special case, and this leads to the question whether the LTTng can
> work without kernel ticks on ARM or not.

> I found out some clue that LTTng only supports x86 NO-HZ in an old link:
> http://git.lttng.org/?p=lttv.git;a=blob_plain;f=doc/developer/lttng-lttv-roadmap.html

This is an old LTTng 0.x link, and does not apply to LTTng 2.x. 

> I think the the LTTng doesn't support ARM NO-HZ, right?

While lib ring buffer within lttng-modules has configuration modes to allow 
minimizing the impact on NO_HZ systems, the current lttng-modules 2.x 
is not setup to use those configuration options, so lttng-modules will wakeup 
cpus periodically to flush buffers (when kernel tracing is active). 

> And I am going to add the ARM NO-HZ support to it now, so could you please
> give me some instructions for quick start?

You will probably want to start by looking at the "read timer" option of 
channels, and see how it is handled within lttng-modules, and how it impacts 
the lib ring buffer client configuration with lttng-modules. 

what you probably want is lttng-modules: 

lib/ringbuffer/config.h 

RING_BUFFER_WAKEUP_BY_TIMER 
RING_BUFFER_WAKEUP_BY_WRITER 
(and compare the various wakeup modes) 

However, if you use wakeup by writer, this has impacts on the instrumentation 
coverage (e.g. you then can trigger issues when instrumenting some core scheduler 
code paths, NMIs, etc). So wakeup by timer is really the mode that allows being 
called from the entire kernel code base without risking having side-effects due to 
interaction of the wakeup mechanism with the rest of the kernel. 

> 1. Which part of LTTng relies on kernel ticks? currently I only find the
> ring-bugger by "grep NO*HZ -i -r" command.

ring_buffer_tick_nohz_callback() was a callback meant to be called by the 
in-kernel code that manages no hz. This was implemented when lib ring buffer 
was an in-kernel library (a patchset to the Linux kernel) rather than sitting within 
lttng-modules. 

When it was a in-kernel patchset, we could modify the nohz notifier code from 
the Linux kernel to notify the ring buffer that it enters or exits nohz mode. Unfortunately, 
there is no such notifier in the kernel today (and Thomas Gleixner refused to add one). 

So we need to add this notifier to the Linux kernel if we want to make the "timer-based" 
mode of the lib ring buffer work well on nohz kernels (meaning: don't wake up the CPUs 
that entered idle state needlessly). 

A possible work-around to try is to change the per-cpu timers within lib ring buffer into 
deferrable timers. This should be attempted too. 

Another possible solution would be to implement a global system wide timer rather than 
a per-cpu timer. This would lessen the impact of the timers on idle CPUs, at the expense 
of scalability. The loss of scalability is the reason why I did not implement this solution. 

> 2. How does this part(ring-buffer and maybe others) relies on kernel ticks?
> can you give some explanations or do you have any document of it/them?

> I will look through the codes, but I think some clear explanations can make
> me go to the right way and speed up before reading so many codes without any
> dedicated target.

lttng-modules lib ring buffer is the only part of lttng-modules that depends on timers, 
and therefore is the only part that should interact with NO HZ. 

Hopefully the above explanations shed some light on the topic. 

Please let me know if you have more questions, 

Thanks, 

Mathieu 

> Thanks.

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20140320/d1c4b0eb/attachment-0001.html>