[lttng-dev] What is the size overhead of UST tracepoints? (hint: very large indeed)

[Mathieu Desnoyers]

----- Original Message -----

> From: "Amit Margalit" <AMITM at il.ibm.com>
> To: "lttng-dev" <lttng-dev at lists.lttng.org>
> Sent: Tuesday, October 29, 2013 4:42:20 AM
> Subject: [lttng-dev] What is the size overhead of UST tracepoints? (hint:
> very large indeed)

> Hello all,

> Let me start with my bottom line conclusions, and then explain them:

> LTTng's size overhead and start-up time overhead is huge! An application
> cannot use (directly or through linking in shared libraries with TPs) more
> than about 1000-2000 tracepoints.
Hi Amit, 

We've pushed this to 16k in J9 instrumentation experiments. I had to make changes to UST to make sure we scale well to those large number of tracepoints. Which UST versions are you working on ? 

> I am working on incorporating LTTng as an alternative logging method for an
> existing project, which already has many thousands of places in the code
> that do logging of some data.

> By many thousands, I mean > 15K.

> I have basically written an automated instrumentor that generates custom
> tracepoints, for every log location plus 1 entry-tracepoints and 1
> exit-tracepoint for every function.
You might want to re-think the way you instrument function entry/exit. It sounds like this could be achieved either with a single trace point for entry, single trace point for exit. We're also working on dynamic instrumentation with dyninst. This could be another approach to look into. 

How many tracepoints to you get for every log locations ? (not including function entry/exit) 

> This totals about 100K tracepoints, with approximately 600K different fields,
> and has increased the size of the project's resulting executables from
> ~100MB to ~900MB (!!) - although this is with debug symbols.
Comparing binary size with debug symbols is meaningless. What are the numbers without debug symbols ? 

> Analysis of the reasons for this has revealed some interesting (but
> discouraging) facts:

> 1. Each tracepoint name is stored in a static buffer sized 256 bytes,
> regardless of how long the name actually is (contributing some 25MB to the
> size)
> 2. Each trace event field name is store also in a 256-byte buffer
> (contirbuting another 150MB)
> 3. Each event / tracepoint holds an event structure which holds the fields,
> etc. and is padded generously.

Be aware that whatever we do, we need to keep backward ABI compatibility. This has influenced some decisions (e.g. padding in event structures) so we can add features in the future without breaking compatibility between applications and UST. An old application binary should always be able to link with a newer liblttng-ust. As long as we guarantee that, and keep room for introduction of new features in the future without having to break the ABI, size optimizations are welcome. 

> Additionally, there is a ton of code to calculate the event size, its
> alignment, and to verify that the names are not longer than the 256 byte
> buffer, and more.
When you say "there is a ton of code", are you talking about actual instruction bytes, of about "lines of C code" ? The former the the proper way to analyze the size footprint. 

> Some of this is either used just once, or even never used, merely compiled to
> let the compiler complain if the sizes are wrong, etc.
Let me know which ones translate into actual instructions. Make sure you compile with "-O2". 

> Now this code won't be included if you compile a statically linked
> executable, but my project compiles almost everything into shared libs.
I don't understand how static exec vs shared lib changes the space overhead, can you clarify ? 

> Another big big problem is the time it takes to register all of these
> tracepoints. For one of my executables which includes ~10K tracepoints, it
> took over 5 seconds just to get to "main()" due to tracepoint constructor
> code registering the tracepoints...
We have seen this kind of issue when trying to instrument J9. I have pushed fixes for this. On which UST version are you seeing these figures ? 

Thanks for looking into this. Making sure that UST instrumentation scales well to large applications is indeed very important to us. 

Mathieu 

> I've started working on an alternative method of doing the same...

> Is anyone already aware of this? Is anyone already working on improving this?
> I'd be very happy to work together on this.

> Amit Margalit
> IBM XIV - Storage Reinvented
> XIV-NAS Development Team
> Tel. 03 -689-7774
> Fax. 03-689-7230
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20131031/07047f42/attachment-0001.html>