<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Amit Margalit" <AMITM@il.ibm.com><br><b>To: </b>"lttng-dev" <lttng-dev@lists.lttng.org><br><b>Sent: </b>Tuesday, October 29, 2013 4:42:20 AM<br><b>Subject: </b>[lttng-dev] What is the size overhead of UST tracepoints? (hint: very large indeed)<br><div><br></div><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Hello all,</span><br><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Let me start with my bottom line conclusions,
and then explain them:</span><br><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">LTTng's size overhead and start-up time
overhead is huge! An application cannot use (directly or through linking
in shared libraries with TPs) more than about 1000-2000 tracepoints.</span></blockquote><div>Hi Amit,<br></div><div><br></div><div>We've pushed this to 16k in J9 instrumentation experiments. I had to make changes to UST to make sure we scale well to those large number of tracepoints. Which UST versions are you working on ?<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">I am working on incorporating LTTng
as an alternative logging method for an existing project, which already
has many thousands of places in the code that do logging of some data.</span><br><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">By many thousands, I mean > 15K.</span><br><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">I have basically written an automated
instrumentor that generates custom tracepoints, for every log location
plus 1 entry-tracepoints and 1 exit-tracepoint for every function.</span></blockquote><div><br></div><div>You might want to re-think the way you instrument function entry/exit. It sounds like this could be achieved either with a single trace point for entry, single trace point for exit. We're also working on dynamic instrumentation with dyninst. This could be another approach to look into.<br></div><div><br></div><div>How many tracepoints to you get for every log locations ? (not including function entry/exit)<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">This totals about 100K tracepoints,
with approximately 600K different fields, and has increased the size of
the project's resulting executables from ~100MB to ~900MB (!!) - although
this is with debug symbols.</span></blockquote><div><br></div><div>Comparing binary size with debug symbols is meaningless. What are the numbers without debug symbols ?<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Analysis of the reasons for this has
revealed some interesting (but discouraging) facts:</span><ol><li value="1"><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Each tracepoint name is stored
in a static buffer sized 256 bytes, regardless of how long the name actually
is (contributing some 25MB to the size)</span></li><li value="2"><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Each trace event field name
is store also in a 256-byte buffer (contirbuting another 150MB)</span></li><li value="3"><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Each event / tracepoint holds
an event structure which holds the fields, etc. and is padded generously.</span></li></ol></blockquote><div><br></div><div>Be aware that whatever we do, we need to keep backward ABI compatibility. This has influenced some decisions (e.g. padding in event structures) so we can add features in the future without breaking compatibility between applications and UST. An old application binary should always be able to link with a newer liblttng-ust. As long as we guarantee that, and keep room for introduction of new features in the future without having to break the ABI, size optimizations are welcome.<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Additionally, there is a ton of code
to calculate the event size, its alignment, and to verify that the names
are not longer than the 256 byte buffer, and more.</span></blockquote><div><br></div><div>When you say "there is a ton of code", are you talking about actual instruction bytes, of about "lines of C code" ? The former the the proper way to analyze the size footprint.<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Some of this is either used just once,
or even never used, merely compiled to let the compiler complain if the
sizes are wrong, etc.</span></blockquote><div><br></div><div>Let me know which ones translate into actual instructions. Make sure you compile with "-O2".<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Now this code won't be included if you
compile a statically linked executable, but my project compiles almost
everything into shared libs.</span></blockquote><div><br></div><div>I don't understand how static exec vs shared lib changes the space overhead, can you clarify ?<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Another big big problem is the time
it takes to register all of these tracepoints. For one of my executables
which includes ~10K tracepoints, it took over 5 seconds just to get to
"main()" due to tracepoint constructor code registering the tracepoints...</span></blockquote><div><br></div><div>We have seen this kind of issue when trying to instrument J9. I have pushed fixes for this. On which UST version are you seeing these figures ?<br></div><div><br></div><div>Thanks for looking into this. Making sure that UST instrumentation scales well to large applications is indeed very important to us.<br></div><div><br></div><div>Mathieu<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">I've started working on an alternative
method of doing the same...<br>
</span><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Is anyone already aware of this? Is
anyone already working on improving this? I'd be very happy to work together
on this.</span><br><br><br><span style="color: #000080; font-family: sans-serif; font-size: small;" data-mce-style="color: #000080; font-family: sans-serif; font-size: small;" color="#000080" face="sans-serif" size="2">Amit Margalit</span><br><span style="color: #808000; font-family: sans-serif; font-size: small;" data-mce-style="color: #808000; font-family: sans-serif; font-size: small;" color="#808000" face="sans-serif" size="2">IBM XIV </span><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">-
<i>Storage Reinvented</i></span><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">XIV-NAS Development Team</span><br><span style="font-family: sans-serif; font-size: small;" data-mce-style="font-family: sans-serif; font-size: small;" face="sans-serif" size="2">Tel. 03</span><span style="font-family: Arial; font-size: small;" data-mce-style="font-family: Arial; font-size: small;" face="Arial" size="2">-689-7774</span><br><span style="font-family: Arial; font-size: small;" data-mce-style="font-family: Arial; font-size: small;" face="Arial" size="2">Fax. 03-689-7230</span><br>_______________________________________________<br>lttng-dev mailing list<br>lttng-dev@lists.lttng.org<br>http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev<br></blockquote><div><br><br></div><div><br></div><div>-- <br></div><div><span name="x"></span>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com<span name="x"></span><br></div></div></body></html>