[lttng-dev] tracing multithread user program and API support for enabling/disabling events and for adding/removing context fields

[Yonghong Yan]

Mathieu,

Thank you for your response. see inline ...

On Wed, Dec 19, 2018 at 4:20 PM Mathieu Desnoyers <
mathieu.desnoyers at efficios.com> wrote:

> Hi Yonghong,
>
> ----- On Dec 19, 2018, at 1:19 PM, Yonghong Yan <yanyh15 at gmail.com> wrote:
>
> We are experimenting LTTng for tracing multi-threaded program, it works
> very well for us. Thank you for having this great tool. But we have some
> concerns about the overhead and scalability of the tracing. Could you share
> some insight of the following questions?
> 1. The session domain communicates with the user application via Unix
> domain socket, from LTTng document. is the communication frequent, such as
> each event requires communication, or the communication just happens at the
> beginning to configure user space tracing?
>
> This Unix socket is only for "control" of tracing (infrequent
> communication). The high-throughput tracing data goes through a shared
> memory map (per-cpu buffers).
>
> 2. For the consumer domain, is the consumer domain has a thread per
> CPU/channel to write to disk or relay the traces, or it is a single
> threaded-process handling all the channels and ring buffers, which could
> become a bottleneck if we have large number of user threads all feeding
> traces?
>
> Each consumer daemon is a single thread at the moment. It could be
> improved by implementing a multithreaded design in the future. It should
> help especially in NUMA setups, where having the consumer daemon on the
> same NUMA node as the ring buffer it reads from would minimize the amount
> of remote NUMA accesses.
>
> Another point is cases where I/O is performed to various target locations
> (different network interfaces or disks). When all I/O goes through the same
> interface, the bottleneck becomes the block device or the network
> interface. However, for scenarios involving many network interfaces or
> block devices, then multithreading the consumer daemon could become useful.
>
> This has not been a priority for anyone so far though.
>
That makes sense.

> 3. In the one channel/ring buffer per CPU setting, if a user thread
> migrates from one CPU to another, are the traces generated by that user
> thread fed to the two channels/buffers for the two CPUs?
>
> The event generated will belong to the per-cpu buffer on which the
> "sched_getcpu()" invocation occurs for the event. It is only saved into a
> single per-cpu buffer, even if the thread is migrated to a different CPU
> before it completes writing the event. This effectively creates infrequent
> situations where threads write into other cpu's per-cpu buffers. Note that
> the "reserve" and "commit" operations are smp-safe in lttng-ust for that
> reason.
>

so after the migration, new events will still be written to another CPU's
buffer?

> 4. So far, the events for tracing can be enabled and disabled from command
> line, are you considering to have runtime options (APIs) to enable or
> disable certain events? Or this is the feature that already in or can be
> implemented in different way?
>
> We already expose a public LGPLv2.1 API for this. See lttng-tools:
>
> include/lttng/event.h: lttng_enable_event()
>
> It is implemented by liblttng-ctl.so
>
Great and will try that.

> 5. For context field, from the document, context fields cannot be removed
> from a channel once you add it. I would like to request a feature to allow
> removing context fields in the user program.
>
> That's unfortunately not that simple. The channel context field belongs to
> the channel, which maps to the "stream" description in the resulting CTF
> metadata in the output trace. That stream description is invariant once it
> has been created.
>
That makes senses.

We use babeltrace to process trace records. So far, it presents a
single-ordered event sequence and we have to use a dedicated event record
filed to separate the traces what are from different channels/threads. Is
there library or API (C or python-based) that allows us to process traces
channel by channel? I saw the traces are saved in files of channels.

Thank you.
Yonghong


>
> So currently the only way to remove a context would be to destroy your
> tracing session and create a new one.
>
> Thanks for your interest in LTTng!
>
> Mathieu
>
>
>
> Thank you very much.
> Yonghong
>
>
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20181219/4f3721f1/attachment-0001.html>