[ltt-dev] Extended network tracepoints

[Mathieu Desnoyers]

* Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote:
> * Benjamin Poirier (benjamin.poirier at polymtl.ca) wrote:
> (append)
> 
> Modify tracepoints and probes to record many header fields from TCP and UDP
> messages. Here are the consequences of this:
> 1) it allows analyzing network traffic to provide some pcap-like
>    functionality within LTTng
> 2) it allows offline synchronization of a group of concurrent traces
>    recorded on different nodes
> 3) it increases tracing overhead
> 
> > When tracing, you can leave out these extended probes or not activate the
> > *_extended markers if you are not especially interested in the details of
> > network traffic and do not wish to synchronize distributed traces.
> > 
> > For those eager to try out time synchronization in lttv, a development tree is
> > available these days at:
> > git clone git://hal.drap.polymtl.ca/lttv-sync.git
> > 
> > See lttv/lttv/sync/README in there for instructions and info. You need traces
> > with the *_extended events, so you need an lttng with the patches in this
> > series too.
> > 
> > Sample traces showcasing a web client and server are also available at:
> > http://hal.drap.polymtl.ca/results55.tar.bz2
> > 
> > Using lttv built from the git tree above, first have a look at the traces
> > without synchronization. From the base of the build dir and adjusting the
> > paths to the traces:
> > $ ./runlttv /tmp/results55/client /tmp/results55/server
> > 
> > The web client is wget (PID 1709). Apache spawns a truckload of processes, the
> > relevant ones are PID 2171 & 2196. Display the relevant processes in lttv
> > using the following filter in the control flow view:
> > (event.target_pid=1709|event.target_pid=2171|event.target_pid=2196)
> > Notice that wget (around time 323) is done before the trace with apache is
> > even started. Apache processes the request around time 363. Something is not
> > right, and that's because the traces are not synchronized.
> > 
> > Now have a look at the traces with synchronization:
> > ./runlttv -a"--sync" /tmp/results55/client /tmp/results55/server
> > Zoom in around time 363 and redraw the window or use the same filter as above
> > to see the wget and apache processes. We can now follow the request from the
> > client to the server. We see that wget blocks a little bit waiting for the
> > connection to be established and then a lot longer while apache is processing.
> > We can see that apache is waiting on a lot of page faults, probably because it
> > was the first request it served since being started, ...
> > 
> > Have fun, feedback is welcome,
> 
> This is a very neat patchset, thanks a lot !
> 
> I'm merging it in the forthcoming LTTng release, which will be 0.163.

I'm also modifying ltt-armall and ltt-disarmall so the _extended markers
are not enabled by default. It's important for configurations where
probes are built into the kernel. I'm adding the net
ltt-armnetsync/ltt-disarmnetsync to control these extended information
markers. It will be in ltt-control 0.72.

Mathieu


> 
> Mathieu
> 
> > -Ben
> > 
> > --
> > 
> > [PATCH 1/4] Move network tracepoints
> > [PATCH 2/4] Trace incoming TCP traffic
> > [PATCH 3/4] Trace incomming UDP traffic
> > [PATCH 4/4] Add probes for detailed tracing of network traffic
> > 
> >  include/linux/ltt-type-serializer.h |   49 ++++++++++++
> >  include/trace/net.h                 |    6 ++
> >  ltt/probes/Makefile                 |    3 +-
> >  ltt/probes/net-extended-trace.c     |  143 +++++++++++++++++++++++++++++++++++
> >  ltt/probes/net-trace.c              |   28 +++++++
> >  net/core/dev.c                      |   10 ++-
> >  net/ipv4/tcp_ipv4.c                 |    7 ++
> >  net/ipv4/udp.c                      |    5 +
> >  8 files changed, 248 insertions(+), 3 deletions(-)
> > 
> 
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68