<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><hr id="zwchr"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Brian Robbins" <brianrob@microsoft.com><br><b>To: </b>"Francis Giraldeau" <francis.giraldeau@gmail.com><br><b>Cc: </b>lttng-dev@lists.lttng.org<br><b>Sent: </b>Monday, March 9, 2015 7:17:51 PM<br><b>Subject: </b>Re: [lttng-dev] Userspace Tracing and Backtraces<br><div><br></div><style><!--
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks Francis.</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">This is what I expected to have to do. I do agree though that adding this to lttng-ust would be a good way to go.</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Should we end up on this path, it certainly seems like it might we worth our time to investigate what it would take to add it to lttng-ust. Do you know who is
the right person to talk to about this? I’d want to make sure that this would not be a non-starter.</span></p></div></blockquote><div>Hi!<br></div><div><br></div><div>I'm the maintainer of LTTng-UST. I agree that adding a backtrace context<br></div><div>to lttng-ust would be a very useful feature.<br></div><div><br></div><div>Some comments inline below,<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div class="WordSection1"><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks.</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Brian
</span></p><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></p><p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Francis Giraldeau [mailto:francis.giraldeau@gmail.com]
<br>
<b>Sent:</b> Monday, March 9, 2015 12:39 PM<br>
<b>To:</b> Brian Robbins<br>
<b>Cc:</b> lttng-dev@lists.lttng.org<br>
<b>Subject:</b> Re: [lttng-dev] Userspace Tracing and Backtraces</span></p><div><div><div><p class="MsoNormal">2015-03-06 13:51 GMT-05:00 Brian Robbins <<a href="mailto:brianrob@microsoft.com" target="_blank">brianrob@microsoft.com</a>>:</p><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"><div><div><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks Francis.</span></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></p><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Is it accurate to say then that the array of addresses would need to be captured by app code by writing
a stack walker by hand</span></p></div></div></blockquote><div><p class="MsoNormal">Yes, the callstack can be recorded in userspace. You would need a tracepoint with a varying length field:</p></div><div><p class="MsoNormal">TRACEPOINT_EVENT(myprovider, callstack,</p></div><div><div><p class="MsoNormal"> TP_ARGS(unsigned long *, buf, size_t, depth),</p></div><div><p class="MsoNormal"> TP_FIELDS(</p></div><div><p class="MsoNormal"> ctf_sequence(unsigned long, addr, buf, size_t, depth)</p></div><div><p class="MsoNormal"> )</p></div><div><p class="MsoNormal">)</p></div></div><div><p class="MsoNormal">In the app code, use libunwind [1] to get the addresses, then call the tracepoint: </p></div><div><p class="MsoNormal"> do_unwind() // use libunwind here</p></div><div><p class="MsoNormal"> tracepoint(myprovider, buf, depth);</p></div><div><p class="MsoNormal">However, the unwind will be done whether or not the tracepoint is active (~10us-100us in steady state, so it's may become expansive if called often). I know there was discussion about tp_code() for such use case (some code to call before
the tracepoint only if it is enabled). Or you can cheat: </p></div></div></div></div></div></blockquote><div>Francis: Did you define UNW_LOCAL_ONLY before including</div><div>the libunwind header in your benchmarks ? (see</div><div><a href="http://www.nongnu.org/libunwind/man/libunwind%283%29.html">http://www.nongnu.org/libunwind/man/libunwind%283%29.html</a>)<br></div><div><br></div><div>The seems to change performance dramatically according to the documentation.<br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div class="WordSection1"><div><div><div><div><p class="MsoNormal">if (__builtin_expect(!!(__tracepoint_myprovider___callstack.state), 0)) {</p></div><div><p class="MsoNormal"> do_unwind(...)</p></div><div><p class="MsoNormal"> tracepoint(myprovider, buf, depth);</p></div><div><p class="MsoNormal">}</p></div><div><p class="MsoNormal">That said, instead of having a callstack tracepoint, IMHO the best solution would be instead extending lttng-ust to add callstack event context (itself linked to libunwind). Then, recording the callstack would be simple like that:</p></div><div><p class="MsoNormal"> $ lttng add-context -u -t callstack</p></div></div></div></div></div></blockquote><div>Agreed on having the backtrace as a context. The main question left is<br></div><div>to figure out if we want to call libunwind from within the traced application<br></div><div>execution context.<br></div><div><br></div><div>Unfortunately, libunwind is not reentrant wrt signals. This is already<br></div><div>a good argument for not calling it from within a tracepoint. I wonder<br></div><div>if the authors of libunwind would be open to make it signal-reentrant<br></div><div>in the future (not by disabling signals, but rather by keeping a TLS<br></div><div>nesting counter, and returning an error if nested, for performance</div><div>considerations).<br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div class="WordSection1"><div><div><div><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in"><div><div><p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">or using the perf capture mechanism that you describe below?</span></p></div></div></blockquote><div><p class="MsoNormal">Perf is peeking at the userspace from kernel space, it's another story. I guess that libunwind was not ported to the kernel because it is a large chunk of complicated code that performs a lot of I/O and computation, while copying a portion
of the stack is really about KISS and low runtime overhead.</p></div></div></div></div></div></blockquote><div>If using libunwind does not work out, another alternative I would consider<br></div><div>would be to copy the stack like perf is doing from the kernel. However,<br></div><div>in the spirit of compacting trace data, I would be tempted to do the following<br></div><div>if we go down that route: check each pointer-aligned address for its content.<br></div><div>If it looks like a pointer to an executable memory area (library, executable, or<br></div><div>JIT'd code), we keep it. Else, we zero this information (not needed). We can<br></div><div>then do a RLE-alike compression on the zeroes, so we can keep the layout<br></div><div>of the stack after uncompression.<br></div><div><br></div><div>Thoughts ?<br></div><div><br></div><div>Thanks,<br></div><div><br></div><div>Mathieu<br></div><div><br></div><div><br></div><div><br></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div class="WordSection1"><div><div><div><div><p class="MsoNormal">Cheers,</p></div><div><p class="MsoNormal">Francis</p></div></div></div></div></div><br>_______________________________________________<br>lttng-dev mailing list<br>lttng-dev@lists.lttng.org<br>http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev<br></blockquote><div><br><br></div><div><br></div><div>-- <br></div><div><span name="x"></span>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com<span name="x"></span><br></div></div></body></html>