[lttng-dev] RFC (v2) Streaming and reading traces over the network
Jim Dumont
jim.dumont at ericsson.com
Mon May 7 14:05:42 EDT 2012
Couple of comments:
If I understand correctly:
Overload - nothing new will be implemented on the sender node (A), lttng-ust and lttng kerner traces will continue their existing buffering as usual.
However, you will be introducing a receiving node (B) consumer which will have some overload support - to be added to RFC.
Question - Do the A's and B's have one-to-one relationship? For every A there is a corresponding B, which means there are multiple B's on a single controller node - OR - does a single B handle multiple A's?
Bandwidth - A's sessionD will have some bandwidth management support - e.g., max-bandwidth config param, then drop packets...
Question - Per session bandwidth setting is required? This would not be the configured global setting, but something set at session creation?
Thanks,
/Jim
-----Original Message-----
From: lttng-dev-request at lists.lttng.org [mailto:lttng-dev-request at lists.lttng.org]
Sent: April-20-12 13:24
To: lttng-dev at lists.lttng.org
Subject: lttng-dev Digest, Vol 48, Issue 26
Send lttng-dev mailing list submissions to
lttng-dev at lists.lttng.org
To subscribe or unsubscribe via the World Wide Web, visit
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
or, via email, send a message with subject or body 'help' to
lttng-dev-request at lists.lttng.org
You can reach the person managing the list at
lttng-dev-owner at lists.lttng.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of lttng-dev digest..."
Today's Topics:
1. Re: RFC (v2) Streaming and reading traces over the network
(Mathieu Desnoyers)
----------------------------------------------------------------------
Message: 1
Date: Fri, 20 Apr 2012 13:23:36 -0400
From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
To: Jim Dumont <jim.dumont at ericsson.com>
Cc: "lttng-dev at lists.lttng.org" <lttng-dev at lists.lttng.org>
Subject: Re: [lttng-dev] RFC (v2) Streaming and reading traces over
the network
Message-ID: <20120420172336.GA11277 at Krystal>
Content-Type: text/plain; charset=us-ascii
* Jim Dumont (jim.dumont at ericsson.com) wrote:
[...]
> -----Original Message-----
> From: Jim Dumont
> Sent: April-19-12 17:16
> To: 'lttng-dev at lists.lttng.org'
> Subject: RE: lttng-dev Digest, Vol 48, Issue 22
>
> Hey there,
>
> Was wondering if you are considering any specific
> support/implementation with respect to overload handling, buffering,
> bandwidth management...?
>
> I'm thinking this is of particular interest when streaming traces over
> a network.
>
For buffering on the sender node (A), the tracer buffers (lttng-ust,
lttng kernel tracer) are those doing the buffering.
Given that in this initial phase, we don't plan on implementing proxy
support, thinking about buffer size on the proxy consumer seems to be a
bit too far ahead.
For node B (the receiver consumer), it would be important to put maximum
limits on how much data we keep in local buffers before we start
dropping incoming data, so this should be added to the RFC. This would
provide overload handling on the receiver node.
The other item is bandwidth management. I think it should be failrly
straightforward to have a configuration option for the sender node's
sessiond that adds a "--max-bandwidth" limit or something like that,
both globally and per-session. The sender would be responsible for
keeping track of how much bandwidth it's using, and would drop trace
data packets and their associated sync data if it notices that it's
about to go beyond its cap.
Thoughts ?
Thanks,
Mathieu
> BR,
>
> /Jim.
>
> -----Original Message-----
> From: lttng-dev-request at lists.lttng.org [mailto:lttng-dev-request at lists.lttng.org]
> Sent: April-18-12 16:31
> To: lttng-dev at lists.lttng.org
> Subject: lttng-dev Digest, Vol 48, Issue 22
>
> Send lttng-dev mailing list submissions to
> lttng-dev at lists.lttng.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
> or, via email, send a message with subject or body 'help' to
> lttng-dev-request at lists.lttng.org
>
> You can reach the person managing the list at
> lttng-dev-owner at lists.lttng.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of lttng-dev digest..."
>
>
> Today's Topics:
>
> 1. RFC (v2) Streaming and reading traces over the network
> (Julien Desfossez)
> 2. [RFC] Session Daemon Network Session (David Goulet)
> 3. Re: Babeltrace Buffer Warning (Mathieu Desnoyers)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 18 Apr 2012 15:00:18 -0400
> From: Julien Desfossez <julien.desfossez at efficios.com>
> To: "lttng-dev at lists.lttng.org" <lttng-dev at lists.lttng.org>
> Cc: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>,
> dgoulet at efficios.com
> Subject: [lttng-dev] RFC (v2) Streaming and reading traces over the
> network
> Message-ID: <4F8F0F42.9030605 at efficios.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> RFC - Streaming And Reading Traces Over The Network
>
> Author: Julien Desfossez <julien.desfossez at efficios.com>
>
> Contributors:
> * Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> * David Goulet <david.goulet at efficios.com>
>
> Version:
> - v0.1: 26/03/2012
> * Initial proposal
> - v0.2: 17/04/2012
> * First revision
>
>
> Introduction
> ------------
>
> This RFC proposes a way for the LTTng kernel and user-space tracer to stream
> traces over the network, and for a viewer to read traces while they are
> being
> generated.
>
>
> Changelog
> ---------
>
> The first version of this RFC was posted on march 26th 2012 on lttng-dev
> mailing list, this version is a rewrite and adds more details about the
> inner
> working of the streaming protocol and clarifies the synchronization
> operations.
>
>
> Prerequisites/assumptions
> -------------------------
>
> - work over TCP and UDP
> - play nicely with NAT
> - trace data and control data are exchanged on different connections
> and possibly on different protocols
> - control data is mandatory and must use a reliable connection (TCP)
> - trace packets (as defined in CTF) may arrive in any order when using
> connection-less protocol.
>
>
> Name convention
> ---------------
>
> - A : the traced machine streaming trace data over the network to B
> - B : the remote consumer, receiving data from A
> - C : the viewer on B displaying the streamed trace to the user
> - trace data : the multiplexed stream of trace streams
>
>
> Creating a network session
> --------------------------
>
> The first step when creating a trace to stream over the network is to
> create a
> tracing session on A that contains all the information to reach B. The inner
> details of this part are covered in a separate RFC from David Goulet,
> lets just
> say that it includes the IP/name of the receiving machine (B) as well as the
> control and data ports/protocols.
>
> Once the session is started, A sends data to B over the control and data
> paths.
> The data path contains only trace data, the control path is streamed over a
> reliable network protocol and contains the session information, indexes and
> synchronization informations.
>
> When a network trace starts, A sends each of its streams to B over the
> control path. The information associated with each stream is A's
> hostname, the session name, each channel name, and each stream id. B
> responds with a stream handle identifier, unique across B, for each
> stream.
>
> B creates the folder hierarchy and acknowledges when it is ready, the trace
> starts on A and the streaming begins.
>
> Upon CPU hotplug, which dynamically adds streams to A, a control message
> is exchanged with B to send the new stream, along with hostname,
> session name, stream id. B responds with a unique stream handle
> identifier.
>
>
> Trace packets encapsulation
> ---------------------------
>
> The data network stream received by B contains all the trace streams
> multiplexed. In order to write the data into the appropriate trace files and
> ensure the data is written in order, a network header must be added by the
> tracer. This header located in the CTF packet header contains the following
> unsigned 64 bits identifiers :
>
> - #stream_handle : the stream handle unique identifier
> - #seq : a sequence number relative to each stream
> - #prev_seq : the previous sequence number to determine if a packet was
> lost by
> the network or the tracer
> - #circuit_id : unique routing ID across all proxy, A, and B (unused for
> now, set to 0).
>
> Apart from #seq which is generated by the tracer, the other identifiers are
> inserted by the consumer on A before sending the packet. In order to do so
> without having to copy the data, when the tracer generates a trace
> packet, it
> leaves an empty space at the position where those fields are located and
> provides API/ABI calls to allow the consumer to find the offsets and
> fill them
> with the appropriate value. Then the packet is directly spliced over the
> network.
>
> Note that the API/ABI calls to get the positions where to write the
> stream id and prev_seq value are provided by the tracers and do _not_
> involve any interaction with a CTF reader.
>
> Note that circuit_id is present for future use in the case of routing
> across proxy consumers.
>
>
> Synchronization
> ---------------
>
> In order to allow the trace viewer to display the traces without the risk of
> receiving information belonging to a timestamp prior to the current
> information (could be caused by low traffic streams), we have to define
> a buffer flush frequency and a synchronization algorithm.
>
> In order to avoid sending empty trace data for inactive streams, the
> consumer
> on A is in charge of the synchronization information. At a predefined
> frequency, the consumer must trigger a buffer flush, and for each stream it
> must save the current sequence number. The trace packets are then sent
> over the
> network. During that time, the consumer generates a synchronization
> packet to
> send on the control path. This packet contains the sequence number for each
> stream sampled before their last buffer flush. When B receives this packet,
> it knows that it is safe to read the trace up to, and including, each
> sequence number. If a stream is inactive and did not generate any trace
> data since the last buffer flush, the last known sequence number is sent.
>
>
> Receiving the trace data
> ------------------------
>
> On B, when trace packets arrive, they must be saved on disk. But since the
> medium may not be considered reliable (UDP for example), packets may be
> lost or
> arrive in a different order. The trace data must be written in order on
> disk.
> When B receives a packet, it reads the stream handle (at fixed position
> in the
> packet) to determine in which trace file to write. Then, looking at the
> sequence and previous sequence number, it can determine whether it must
> queue
> the packet (to wait for a previous packet to arrive) or if it can write
> it to
> disk. The previous packet ID (#prev_seq) is a way to detect if data has been
> discarded by the tracer or by the network transport. When packets are not
> received in consecutive order, B must assume that the missing packet(s) may
> arrive later and stop writing trace data for this trace file until the
> gap is
> closed. UDP packets can be lost forever, so a threshold (number of packets
> queued or timeout) must be defined to avoid waiting forever. When this
> condition is reached, B can either discard the missing packet or ask for a
> retransmission (if supported on A).
>
>
>
> ------------------------------
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>
>
> End of lttng-dev Digest, Vol 48, Issue 22
> *****************************************
>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
------------------------------
_______________________________________________
lttng-dev mailing list
lttng-dev at lists.lttng.org
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
End of lttng-dev Digest, Vol 48, Issue 26
*****************************************
More information about the lttng-dev
mailing list