<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">2015-03-10 21:47 GMT-04:00 Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span>:<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="font-family:'times new roman','new york',times,serif;font-size:12pt;color:rgb(0,0,0)"><div>Francis: Did you define UNW_LOCAL_ONLY before including</div><div>the libunwind header in your benchmarks ? (see</div><div><a href="http://www.nongnu.org/libunwind/man/libunwind%283%29.html" target="_blank">http://www.nongnu.org/libunwind/man/libunwind%283%29.html</a>)<span style="font-family:arial,sans-serif;font-size:small;color:rgb(34,34,34)"> </span></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="font-family:'times new roman','new york',times,serif;font-size:12pt;color:rgb(0,0,0)"><div></div><div><br></div><div>The seems to change performance dramatically according to the documentation.<br></div></div></blockquote><div><br></div><div><br></div><div>Yes, this is the case. Time to unwind is higher at the beginning (probably related to internal cache build), and also vary according to call-stack depth.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="font-family:'times new roman','new york',times,serif;font-size:12pt;color:rgb(0,0,0)"><div>Agreed on having the backtrace as a context. The main question left is<br></div><div>to figure out if we want to call libunwind from within the traced application<br></div><div>execution context.<br></div><div><br></div><div>Unfortunately, libunwind is not reentrant wrt signals. This is already<br></div><div>a good argument for not calling it from within a tracepoint. I wonder<br></div><div>if the authors of libunwind would be open to make it signal-reentrant<br></div><div>in the future (not by disabling signals, but rather by keeping a TLS<br></div><div>nesting counter, and returning an error if nested, for performance</div><div>considerations).<br></div></div></blockquote><div><br></div><div>The functions unw_init_local() and unw_step() are signal safe [1]. The critical sections are protected using lock_acquire() that blocks all signals before taking the mutex, which prevent the recursion.</div><div><br></div><div><div>#define lock_acquire(l,m) \</div><div>do { \</div><div> SIGPROCMASK (SIG_SETMASK, &unwi_full_mask, &(m)); \</div><div> mutex_lock (l); \</div><div>} while (0)</div><div>#define lock_release(l,m) \</div><div>do { \</div><div> mutex_unlock (l); \</div><div> SIGPROCMASK (SIG_SETMASK, &(m), NULL); \</div><div>} while (0)</div></div><div><br></div><div>To understand the implications, I did a small program to study nested signals [2], where a signal is sent from within a signal, or when segmentation fault occurs in a signal handler. Blocking a signal differs it when it is unblocked, while ignored signals are discarded. Blocked signals that can't be ignored have their default behaviour. It prevents a possible deadlock, let's say if lock_acquire() was nesting with a custom SIGSEGV handler trying to get the same lock.<br></div><div><br></div><div>So, let's say that instead of blocking signals, we have a per-thread mutex, that returns if try_lock() fails. It would be faster, but from the user's point of view, the backtrace will be dropped randomly. I would prefer it a bit slower, but reliable.</div><div><br></div><div>In addition, could it be possible that TLS is not signal safe [3]? </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div style="font-family:'times new roman','new york',times,serif;font-size:12pt;color:rgb(0,0,0)"><span class=""><blockquote style="border-left-width:2px;border-left-style:solid;border-left-color:rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div><div><div><div><blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in"><div><div><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">or using the perf capture mechanism that you describe below?</span></p></div></div></blockquote><div><p class="MsoNormal">Perf is peeking at the userspace from kernel space, it's another story. I guess that libunwind was not ported to the kernel because it is a large chunk of complicated code that performs a lot of I/O and computation, while copying a portion
of the stack is really about KISS and low runtime overhead.</p></div></div></div></div></div></blockquote></span><div>If using libunwind does not work out, another alternative I would consider<br></div><div>would be to copy the stack like perf is doing from the kernel. However,<br></div><div>in the spirit of compacting trace data, I would be tempted to do the following<br></div><div>if we go down that route: check each pointer-aligned address for its content.<br></div><div>If it looks like a pointer to an executable memory area (library, executable, or<br></div><div>JIT'd code), we keep it. Else, we zero this information (not needed). We can<br></div><div>then do a RLE-alike compression on the zeroes, so we can keep the layout<br></div><div>of the stack after uncompression.<br></div><div><br></div></div></blockquote><div><br></div><div>Interesting! For comparison, here is a perf event [4] that shows there is a lot of room for reducing the event size. We should check if discarding other saved register values on the stack impacts restoring the instruction pointer register. Doing the unwind offline also solves signal safety, should be fast and scalable.</div><div><br></div><div>Francis</div><div><br></div><div>[1] <a href="http://www.nongnu.org/libunwind/man/unw_init_local(3).html">http://www.nongnu.org/libunwind/man/unw_init_local(3).html</a></div><div>[2] <a href="https://gist.github.com/giraldeau/98f08161e83a7ab800ea">https://gist.github.com/giraldeau/98f08161e83a7ab800ea</a></div><div>[3] <a href="https://sourceware.org/glibc/wiki/TLSandSignals">https://sourceware.org/glibc/wiki/TLSandSignals</a></div><div>[4] <a href="http://pastebin.com/sByfXXAQ">http://pastebin.com/sByfXXAQ</a></div></div></div></div>