[lttng-dev] tracing multithread user program and API support for enabling/disabling events and for adding/removing context fields

[Mathieu Desnoyers]

Hi, 

Can you define what you mean by "per-user-thread tracepoint" and "whole-user-process" ? AFAIK 
those concepts don't appear anywhere in the LTTng documentations. 

Thanks, 

Mathieu 

----- On Dec 19, 2018, at 6:06 PM, Yonghong Yan <yanyh15 at gmail.com> wrote: 

> Got another question about lttng_enable_event(): Using this API will impact
> per-user-thread tracepoint or the whole-user-process? I am thinking of the
> whole process, but want to confirm.

> Yonghong

> On Wed, Dec 19, 2018 at 4:20 PM Mathieu Desnoyers < [
> mailto:mathieu.desnoyers at efficios.com | mathieu.desnoyers at efficios.com ] >
> wrote:

>> Hi Yonghong,

>> ----- On Dec 19, 2018, at 1:19 PM, Yonghong Yan < [ mailto:yanyh15 at gmail.com |
>> yanyh15 at gmail.com ] > wrote:

>>> We are experimenting LTTng for tracing multi-threaded program, it works very
>>> well for us. Thank you for having this great tool. But we have some concerns
>>> about the overhead and scalability of the tracing. Could you share some insight
>>> of the following questions?
>>> 1. The session domain communicates with the user application via Unix domain
>>> socket, from LTTng document. is the communication frequent, such as each event
>>> requires communication, or the communication just happens at the beginning to
>>> configure user space tracing?

>> This Unix socket is only for "control" of tracing (infrequent communication).
>> The high-throughput tracing data goes through a shared memory map (per-cpu
>> buffers).

>>> 2. For the consumer domain, is the consumer domain has a thread per CPU/channel
>>> to write to disk or relay the traces, or it is a single threaded-process
>>> handling all the channels and ring buffers, which could become a bottleneck if
>>> we have large number of user threads all feeding traces?

>> Each consumer daemon is a single thread at the moment. It could be improved by
>> implementing a multithreaded design in the future. It should help especially in
>> NUMA setups, where having the consumer daemon on the same NUMA node as the ring
>> buffer it reads from would minimize the amount of remote NUMA accesses.

>> Another point is cases where I/O is performed to various target locations
>> (different network interfaces or disks). When all I/O goes through the same
>> interface, the bottleneck becomes the block device or the network interface.
>> However, for scenarios involving many network interfaces or block devices, then
>> multithreading the consumer daemon could become useful.

>> This has not been a priority for anyone so far though.

>>> 3. In the one channel/ring buffer per CPU setting, if a user thread migrates
>>> from one CPU to another, are the traces generated by that user thread fed to
>>> the two channels/buffers for the two CPUs?

>> The event generated will belong to the per-cpu buffer on which the
>> "sched_getcpu()" invocation occurs for the event. It is only saved into a
>> single per-cpu buffer, even if the thread is migrated to a different CPU before
>> it completes writing the event. This effectively creates infrequent situations
>> where threads write into other cpu's per-cpu buffers. Note that the "reserve"
>> and "commit" operations are smp-safe in lttng-ust for that reason.

>>> 4. So far, the events for tracing can be enabled and disabled from command line,
>>> are you considering to have runtime options (APIs) to enable or disable certain
>>> events? Or this is the feature that already in or can be implemented in
>>> different way?

>> We already expose a public LGPLv2.1 API for this. See lttng-tools:

>> include/lttng/event.h: lttng_enable_event()

>> It is implemented by liblttng-ctl.so

>>> 5. For context field, from the document, context fields cannot be removed from a
>>> channel once you add it. I would like to request a feature to allow removing
>>> context fields in the user program.

>> That's unfortunately not that simple. The channel context field belongs to the
>> channel, which maps to the "stream" description in the resulting CTF metadata
>> in the output trace. That stream description is invariant once it has been
>> created.

>> So currently the only way to remove a context would be to destroy your tracing
>> session and create a new one.

>> Thanks for your interest in LTTng!

>> Mathieu

>>> Thank you very much.
>>> Yonghong

>>> _______________________________________________
>>> lttng-dev mailing list
>>> [ mailto:lttng-dev at lists.lttng.org | lttng-dev at lists.lttng.org ]
>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> [ http://www.efficios.com/ | http://www.efficios.com ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20181220/bc74a91b/attachment.html>