[lttng-dev] lttng

[Mathieu Desnoyers]

Hi Kubitz,

Sorry for the late reply, CCing lttng-dev, since this explanation about
robustness against crash can be useful to others,

* Kubitz Jörg (joerg.kubitz at hspg.de) wrote:
> You use per-cpu-datastructures only to prevent cache-line-bouncing and
> not for exclusive access?

For kernel tracing, we can disable preemption (and thus migration)
around accesses to per-cpu data structures. Therefore, it prevents both
cache line bouncing and ensures that only a single CPU accesses the
per-cpu data (although an IRQ/softirq could access the same per-cpu data
concurrently).

For user-space tracing, given that we cannot cheaply disable preemption,
we only use per-cpu data to prevent cache-line bouncing.

> Then you would need to atomically write the
> whole message.

This is correct. In both cases. We need to protect against interrupts in
the kernel, and against signals and other threads in user-space.

> That cannot be done if the message is bigger than the
> bus.

It's the space reservation that needs to be performed atomically, not
writing the message. Please see:

https://lttng.org/publications
M. Desnoyers, Low-Impact Operating System Tracing Ph.D. dissertation,
École Polytechnique de Montréal, 2009.

See the chapter describing the ring buffer.


> Or if you only reserve space atomically and the thread dies
> before it uses the reserved space the reader that waits for the insert
> is in deadlock.

This issue is not so relevant for kernel tracing, because we disable
preemption around use of the buffer, so only a kernel crash would cause
a thread to die between reserve and commit. But at that point, we have
bigger problems in the system than tracing.

For user-space tracing, this question is relevant, because we have to
deal with the fact that a user-space application can be killed at any
point.

There are now two tracing modes implemented in lttng-ust (user-space
tracer), each with their own advantages (+)/disadvantages(-):

1) per-pid buffer tracing,
  - each process in the system has its own per-cpu buffers,
    increased memory consumption, increased overhead for buffer
    allocation in workloads consisting of short-lived processes,
  + when a process dies, its associated wakeup pipe write-side is
    closed by the kernel, so the consumer knows it needs to read the
    last buffer. It can read it up to the last contiguous point in the
    last subbuffer where data was written (commit seq counter).

2) per-uid (shared) buffer tracing, (buffers shared across applications
   with same user ID)
  + less memory overhead, less runtime overhead for short-lived
    processes,
  - if a process dies between reserve and commit, it creates a whole in
    the subbuffer. This whole will be detected only when the producer
    will fill up the buffer and find the unbalanced reserved/commit
    counter the next time it reaches this sub-buffer. Until the buffer
    the unbalanced reserve/commit count is observed by a producer,
    the consumer is unable to read the buffer futher than the packet
    containing the "whole". When the producer rebalances this packet's
    reserve/commit count, it will be counted as a "discarded" packet,
    and we lose all data within the packet that contains the whole.

> So lttng is not robust?

It's actually a question of trade-offs: robustness (1) vs speed (2). We
provide configurations for both.

Thanks,

Mathieu

> 
> Thanks for your explanaition.
> 
> Jörg Kubitz

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com