[lttng-dev] Some confusion about cpu usage of the lttng-consumerd process

[Jonathan Rajotte-Julien]

Hi,

On Fri, Nov 27, 2020 at 02:39:28PM +0800, 熊毓华 via lttng-dev wrote:
> Hi，dear.
> 
> I have been using lttng to monitor my server these days，but I found something interesting.
> 
> The cpu usage of lttng varies with the number of cpu cores of the server.

Which is a bit expected since more CPU means more "data" source from the point
of view of lttng hence more "work" overall.

> 
> On the server, I create a tracing session in live mode, using "lttng create my-session --live". 
> 
> Then,I Start the babeltrace2 and configure it to connect to the relay daemon,using "--input-format=lttng-live" mode.
> 
> I used 5 cloud servers，1core4G 2core8G 4core16G 8core16G 8core16G.
> 
> And,the same test script was executed above to provide the same workload.

We would need the test script to have some context here of the workload.

> 
> As we all know,lttng has 5 processes,
> 
> 1.lttng-runas    --daemonize
> 
> 2.lttng-runas      -k --consumerd-cmd-sock /var/run/lttng/kconsumerd/command --consumerd-err-sock /var/run/lttng/kconsumerd/error --group tracing

Based on this you are performing kernel tracing.

> 
> 3.lttng-sessiond --daemonize
> 
> 4.lttng-relayd -L tcp://localhost:5344
> 
> 5.lttng-consumerd  -k --consumerd-cmd-sock /var/run/lttng/kconsumerd/command --consumerd-err-sock /var/run/lttng/kconsumerd/error --group tracing
> 
> 
> The CPU usage of the first four processes is below 2% on the 5 servers,but the lttng-consumerd process is different.
> 
> On 1-core、2-core、4-core servers，the CPU usage of the lttng-consumerd process is below 2%.

How is the cpu usage measured here?

> 
> But on two 8-core machines, the cpu usage of the lttng-consumerd process reached 10% or more.

Consumerd is responsible of "fetching" data from the ring buffers and "saving"
it either locally (trace on disk) or remotely (streaming/live session). CPU usage
should be a bit correlated with the event production rate. Did you have a look at the
number of events generated for a similar interval?

> And,the cpu usage of the babeltrace process is not much different,just the cpu usage of the lttng-consumerd process varies with the number of cpu cores of the server.
> 
> Why it is like this？How should this phenomenon be analyzed?
> 
> Looking forward to your reply.
> 
> thanks,
> yuhua
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev


-- 
Jonathan Rajotte-Julien
EfficiOS