[lttng-dev] RFC Streaming and reading traces over the network

[Julien Desfossez]

Hello,

Here is the proposal for the implementation of streaming and reading
live traces over the network.

Comments are welcome !

Thanks,

Julien


Objective : displaying the trace of a remote machine while it is being
recorded

Prerequisites/assumptions :
- work over TCP and UDP
- play nicely with NAT
- packets may arrive in any order
- trace data and control data are exchanged on different connections
  and possibly on different protocols
- control data are mandatory and must use a reliable connection (TCP)

Name convention :
- A : the traced machine streaming trace data over the network to B
- B : the remote consumer, receiving data from A
- C : the viewer on B displaying the streamed trace to the user
- trace data : the multiplexed stream of trace streams
- trace packet header contains :
        - a trace file ID (previously defined over the data control link)
        - a packet sequence number (sequential ID of packet relative to each
          stream)

Sending traces over the network (A to B) :
- for each packet, the consumer generates an index entry
- packets are sent over the trace data connection
- indexes are sent over the control data connection
- the packets and indexes are sent as soon as they are ready
- the buffers are flushed at a defined frequency
- at every buffer flush, if no trace data has been produced at all : do
  nothing; if at least one stream has produced trace data : generate an
empty
  index for all the inactive streams

Receiving traces on the network (B) :
- the trace packets within a stream must be written on disk in order
relative
  to other packets of the same stream
- when B receives a trace packet, it must check if the sequence number
  immediately follows the previous packet seen for this stream
- if the sequence numbers are consecutive, it writes the packet to the trace
  file
- if the sequence numbers are non-consecutive, it keeps the packet in memory
  (in a priority queue) for a predefined number of packets (the queue should
  not be infinite)
- when the number of received packets while one is missing is reached, the
  packet missing is considered lost, and the process continues
- if an index corresponding to a missing packet is received, the index is
  discarded
- an index is written on disk only after its corresponding data is fully
  written into its tracefile (flock(2))
- an empty index is the heartbeat that informs the remote consumer that
it can
  make available all the data received for the other streams because it will
  not receive older data
- there is one index file per trace file (chan_0.idx)

Consumer live reading (B to C) :
- the index file written by B and read by C is the synchronisation mechanism
  between B and C (using flock(2)).
- once the index is flushed on disk, the trace data corresponding to the
period
  covered by the index must already be flushed on disk
- as soon as we have an index entry for each file stream, the viewer can
read
  the tracefiles up to the min(index end timestamp).
- we use the prio_heap as usual to read the trace