[lttng-dev] Some confusion about cpu usage of the lttng-consumerd process

[Mathieu Desnoyers]

----- On Nov 28, 2020, at 1:49 AM, 熊毓华 xiongyuhua at zju.edu.cn wrote:

> Hi,

> I put my test scripts in the attachment.

> You can just run the script directly, create the trace session with the "--live"
> option on the 8core server,

> then you will find the cpu usage of the lttng-consumerd process reached 10% or
> more.

> About the streaming mode of lttng,I did the test before, it worked well.

> When I create the trace session with "lttng create my-session
> --output=/tmp/my-kernel-trace", or with "lttng create my-session
> --set-url=net://ip",

> the number of CPU seems not affect the cpu usage with lttng-consumerd.

> It seems that only live-mode will be affected.

The overhead of consumer daemon in live mode will increase with the number
of cpus on the system. This is expected.

There is one knob you can try to configure to adapt the amount of overhead
caused by the live timer in the consumer daemon: lttng create --live=NNN
where NNN is the live timer period in microseconds. The default is 1000000us
(1 second). Try changing this value to something larger and you will
probably notice that it lessen the consumer daemon overhead.

Shipping trace data with relatively low latency so it can be immediately
read adds overhead, and it increases with the number of cpus in the system
because there are then more per-cpu buffers to flush.

Note that your instrumentation also targets unrelated kernel and user-space
execution, so whatever is executed in the background also gets traced. And
this generate trace data traffic on each cpu. Having even just a bit of
trace data to send out in live mode can very much explain the overhead you
observe with the lttng consumer daemon.

If you care that much about overhead of the consumer daemon, don't use live
mode, and use the alternatives we discussed earlier instead, such as the
session rotation mode, if you need to consume the trace data while it is
produced.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com