[lttng-dev] tracing multithread user program and API support for enabling/disabling events and for adding/removing context fields
yanyh15 at gmail.com
Wed Dec 19 17:07:02 EST 2018
Thank you for your response. see inline ...
On Wed, Dec 19, 2018 at 4:20 PM Mathieu Desnoyers <
mathieu.desnoyers at efficios.com> wrote:
> Hi Yonghong,
> ----- On Dec 19, 2018, at 1:19 PM, Yonghong Yan <yanyh15 at gmail.com> wrote:
> We are experimenting LTTng for tracing multi-threaded program, it works
> very well for us. Thank you for having this great tool. But we have some
> concerns about the overhead and scalability of the tracing. Could you share
> some insight of the following questions?
> 1. The session domain communicates with the user application via Unix
> domain socket, from LTTng document. is the communication frequent, such as
> each event requires communication, or the communication just happens at the
> beginning to configure user space tracing?
> This Unix socket is only for "control" of tracing (infrequent
> communication). The high-throughput tracing data goes through a shared
> memory map (per-cpu buffers).
> 2. For the consumer domain, is the consumer domain has a thread per
> CPU/channel to write to disk or relay the traces, or it is a single
> threaded-process handling all the channels and ring buffers, which could
> become a bottleneck if we have large number of user threads all feeding
> Each consumer daemon is a single thread at the moment. It could be
> improved by implementing a multithreaded design in the future. It should
> help especially in NUMA setups, where having the consumer daemon on the
> same NUMA node as the ring buffer it reads from would minimize the amount
> of remote NUMA accesses.
> Another point is cases where I/O is performed to various target locations
> (different network interfaces or disks). When all I/O goes through the same
> interface, the bottleneck becomes the block device or the network
> interface. However, for scenarios involving many network interfaces or
> block devices, then multithreading the consumer daemon could become useful.
> This has not been a priority for anyone so far though.
That makes sense.
> 3. In the one channel/ring buffer per CPU setting, if a user thread
> migrates from one CPU to another, are the traces generated by that user
> thread fed to the two channels/buffers for the two CPUs?
> The event generated will belong to the per-cpu buffer on which the
> "sched_getcpu()" invocation occurs for the event. It is only saved into a
> single per-cpu buffer, even if the thread is migrated to a different CPU
> before it completes writing the event. This effectively creates infrequent
> situations where threads write into other cpu's per-cpu buffers. Note that
> the "reserve" and "commit" operations are smp-safe in lttng-ust for that
so after the migration, new events will still be written to another CPU's
> 4. So far, the events for tracing can be enabled and disabled from command
> line, are you considering to have runtime options (APIs) to enable or
> disable certain events? Or this is the feature that already in or can be
> implemented in different way?
> We already expose a public LGPLv2.1 API for this. See lttng-tools:
> include/lttng/event.h: lttng_enable_event()
> It is implemented by liblttng-ctl.so
Great and will try that.
> 5. For context field, from the document, context fields cannot be removed
> from a channel once you add it. I would like to request a feature to allow
> removing context fields in the user program.
> That's unfortunately not that simple. The channel context field belongs to
> the channel, which maps to the "stream" description in the resulting CTF
> metadata in the output trace. That stream description is invariant once it
> has been created.
That makes senses.
We use babeltrace to process trace records. So far, it presents a
single-ordered event sequence and we have to use a dedicated event record
filed to separate the traces what are from different channels/threads. Is
there library or API (C or python-based) that allows us to process traces
channel by channel? I saw the traces are saved in files of channels.
> So currently the only way to remove a context would be to destroy your
> tracing session and create a new one.
> Thanks for your interest in LTTng!
> Thank you very much.
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> Mathieu Desnoyers
> EfficiOS Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the lttng-dev