<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Yes, <span>tracepoint_unregister_lib</span> is the function I was referring. If the function is okay to call with background thread still writing trace, I am definitely fine. I just want to double check it, if there could be a potential bug. Thanks.<br>
</p>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Mathieu Desnoyers <mathieu.desnoyers@efficios.com><br>
<b>Sent:</b> Wednesday, September 21, 2016 2:05 PM<br>
<b>To:</b> Jeffrey Chen<br>
<b>Cc:</b> Jeffrey Chen; lttng-dev<br>
<b>Subject:</b> Re: segfault when exiting process with background thread</font>
<div> </div>
</div>
<div>
<div style="font-family:arial,helvetica,sans-serif; font-size:12pt; color:#000000">
<div></div>
<div>I don't see anything relevant that changed since commit:<br>
</div>
<div><br>
</div>
<div>commit 664ccf245fcb343e1ea10e145bab3749423e9d0f<br>
Author: Mathieu Desnoyers <mathieu.desnoyers@efficios.com><br>
Date: Thu Mar 10 09:32:59 2016 -0500<br>
<br>
Add tracepoint_disable_destructors()<br>
<br>
Calling this function from an instrumented program allows disabling<br>
tracepoint destructors. This allows threads to continue calling<br>
tracepoint code even after the tracepoint destructors have run. This is<br>
needed for applications that exit without joining all their threads.<br>
<br>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com><br>
CC: Jeffrey Chen <cpthk@live.com><br>
<br>
</div>
<div>Can you point it out ? And let me know what is segfaulting, with a backtrace.<br>
</div>
<div><br>
</div>
<div>I see that tracepoint_dlopen_ptr->tracepoint_unregister_lib(__start___tracepoints_ptrs)<br>
</div>
<div>is still invoked from __tracepoints__ptrs_destroy() even if the disable destructors is set,<br>
</div>
<div>but this only unregisters the library from the list of available events. I don't see how it would<br>
</div>
<div>trigger a crash of threads still actively performing tracing.<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div><br>
</div>
<div>Mathieu</div>
<div><br>
</div>
<div><span id="zwchr">----- On Sep 21, 2016, at 4:00 PM, Jeffrey Chen <cpthk@live.com> wrote:<br>
</span></div>
<div>
<blockquote style="border-left:2px solid #1010FF; margin-left:5px; padding-left:5px; color:#000; font-weight:normal; font-style:normal; text-decoration:none; font-family:Helvetica,Arial,sans-serif; font-size:12pt">
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; font-family:Calibri,Arial,Helvetica,sans-serif">
<p><span>It seems that some changes went in recently that caused this workaround to not work again. Originally, the __tracepoints__disable_destructors variable should be checked around the beginning of the destructor functions (<span class="pl-c1">__tracepoints__destroy</span>
and <span class="pl-c1">__tracepoints__ptrs_destroy</span>), and return the function when the variable is true. It is not in the beginning any more. So, the segfault problem could happen. Is it possible to fix it again at your convenience? Thanks a lot.</span><br>
</p>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><span color="#000000" face="Calibri, sans-serif" style="font-size:11pt; color:#000000; font-family:Calibri,sans-serif"><b>From:</b> lttng-dev <lttng-dev-bounces@lists.lttng.org> on behalf of Mathieu Desnoyers <mathieu.desnoyers@efficios.com><br>
<b>Sent:</b> Tuesday, March 15, 2016 12:00 PM<br>
<b>To:</b> Jeffrey Chen<br>
<b>Cc:</b> lttng-dev<br>
<b>Subject:</b> Re: [lttng-dev] segfault when exiting process with background thread</span>
<div> </div>
</div>
<div>
<div style="font-family:arial,helvetica,sans-serif; font-size:12pt; color:#000000">
<div>Commit merged into master, thanks for your feedback!<br>
</div>
<div><br>
</div>
<div>Mathieu<br>
</div>
<div><br>
</div>
<span id="zwchr">----- On Mar 15, 2016, at 2:48 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:<br>
</span>
<div>
<blockquote style="border-left:2px solid #1010FF; margin-left:5px; padding-left:5px; color:#000; font-weight:normal; font-style:normal; text-decoration:none; font-family:Helvetica,Arial,sans-serif; font-size:12pt">
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi Mathieu:</p>
<p><br>
</p>
<p>Will this fix be checked-in? Thanks.<br>
</p>
<p><br>
</p>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-size:11pt; color:#000000; font-family:Calibri,sans-serif"><b>From:</b> Mathieu Desnoyers <mathieu.desnoyers@efficios.com><br>
<b>Sent:</b> Thursday, March 10, 2016 3:27 PM<br>
<b>To:</b> Jeffrey Chen<br>
<b>Cc:</b> Jonathan Rajotte Julien; lttng-dev<br>
<b>Subject:</b> Re: segfault when exiting process with background thread</span>
<div> </div>
</div>
<div>
<div style="font-family:arial,helvetica,sans-serif; font-size:12pt; color:#000000">
<div><br>
</div>
<div><br>
</div>
<span id="zwchr">----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:<br>
</span>
<div>
<blockquote style="border-left:2px solid #1010FF; margin-left:5px; padding-left:5px; color:#000; font-weight:normal; font-style:normal; text-decoration:none; font-family:Helvetica,Arial,sans-serif; font-size:12pt">
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Thanks, Mathieu.</p>
<p><br>
</p>
<p>Yes, the fix works as a workaround. This is also the same workaround we have now.</p>
<p>It seems the problem is due to a race condition, that it checked whether "<span style="font-size:12pt; font-family:"Calibri",sans-serif; color:black; background:rgb(255,255,255) none repeat scroll 0% 0%">tracepoint_dlopen.rcu_read_lock_sym_bp</span>" is
not NULL. If not NULL, it calls to that. But, at this moment, memset cleared the memory.</p>
<p>This fix is okay with us, but I am not certain if it would break other component, since I do not understand LTTng enough to tell. Thanks.</p>
</div>
</blockquote>
<div><br>
</div>
<div>In addition to memset clearing that memory, dlclose() of the lttng-ust-tracepoint<br>
</div>
<div>shared object will clear the memory containing the called functions. Therefore,<br>
</div>
<div>we need to skip both steps of the destructor if we want threads to survive after<br>
</div>
<div>this destructor execution.<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div><br>
</div>
<div>Mathieu<br>
</div>
<div><br>
</div>
<blockquote style="border-left:2px solid #1010FF; margin-left:5px; padding-left:5px; color:#000; font-weight:normal; font-style:normal; text-decoration:none; font-family:Helvetica,Arial,sans-serif; font-size:12pt">
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<p><br>
</p>
<p><br>
</p>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-size:11pt; color:#000000; font-family:Calibri,sans-serif"><b>From:</b> Mathieu Desnoyers <mathieu.desnoyers@efficios.com><br>
<b>Sent:</b> Thursday, March 10, 2016 10:52 AM<br>
<b>To:</b> Jeffrey Chen<br>
<b>Cc:</b> Jonathan Rajotte Julien; lttng-dev<br>
<b>Subject:</b> Re: segfault when exiting process with background thread</span></div>
<div>
<div style="font-family:arial,helvetica,sans-serif; font-size:12pt; color:#000000">
<div>Hi Jeffrey,<br>
</div>
<div><br>
</div>
<div>I CC'd your @live.com address on the RFC patch. Please let me know if the approach<br>
</div>
<div>is OK with you (and try it out).<br>
</div>
<div><br>
</div>
<div>See:<br>
</div>
<div>https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div><br>
</div>
<div>Mathieu<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><span id="zwchr">----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk@hotmail.com> wrote:<br>
</span></div>
<div>
<blockquote style="border-left:2px solid #1010FF; margin-left:5px; padding-left:5px; color:#000; font-weight:normal; font-style:normal; text-decoration:none; font-family:Helvetica,Arial,sans-serif; font-size:12pt">
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Thanks Mathieu.</p>
<p>Is there a plan at LTTng side to fix this issue? If so, we could wait for the fix. If not, we will have to workaround the problem for now (probably by applying your fix ourselves). Thanks.<br>
</p>
<p><br>
</p>
<br>
<br>
<div style="color:rgb(0,0,0)">
<hr style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><span style="font-size:11pt; color:#000000; font-family:Calibri,sans-serif"><b>From:</b> Mathieu Desnoyers <mathieu.desnoyers@efficios.com><br>
<b>Sent:</b> Tuesday, March 8, 2016 6:09 PM<br>
<b>To:</b> Jonathan Rajotte Julien<br>
<b>Cc:</b> lttng-dev<br>
<b>Subject:</b> Re: [lttng-dev] segfault when exiting process with background thread</span></div>
<div>
<div style="font-family:arial,helvetica,sans-serif; font-size:12pt; color:#000000">
<div>The attached patch works around the issue. The problem is<br>
</div>
<div>the tracepoint destructors that run in parallel with the thread<br>
</div>
<div>that still exists.<br>
</div>
<div><br>
</div>
<div>If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared<br>
</div>
<div>object on dlclose of the instrumented code. Not sure if we should<br>
</div>
<div>care though...<br>
</div>
<div><br>
</div>
<div>Thoughts ?<br>
</div>
<div><br>
</div>
<div>Thanks,<br>
</div>
<div><br>
</div>
<div>Mathieu<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><span id="zwchr">----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers <mathieu.desnoyers@efficios.com> wrote:<br>
</span></div>
<div>
<blockquote style="border-left:2px solid #1010FF; margin-left:5px; padding-left:5px; color:#000; font-weight:normal; font-style:normal; text-decoration:none; font-family:Helvetica,Arial,sans-serif; font-size:12pt">
<div style="font-family:arial,helvetica,sans-serif; font-size:12pt; color:#000000">
<div>Reproduced it on master (ust and tools).<br>
</div>
<br>
<div>Repro with:<br>
</div>
<br>
<div>lttng create<br>
</div>
<div>lttng enable-event -u -a<br>
</div>
<div>lttng start<br>
</div>
<br>
<div>then run hello a few times, it eventually segfaults in the spawned thread.<br>
</div>
<br>
<div>Looking into it.<br>
</div>
<br>
<div>Thanks,<br>
</div>
<br>
<div>Mathieu<br>
</div>
<br>
<span id="zwchr">----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien <Jonathan.rajotte-julien@efficios.com> wrote:<br>
</span>
<div>
<blockquote style="border-left:2px solid #1010FF; margin-left:5px; padding-left:5px; color:#000; font-weight:normal; font-style:normal; text-decoration:none; font-family:Helvetica,Arial,sans-serif; font-size:12pt">
Hi Jeffrey,<br>
<br>
Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if installed)?<br>
<br>
Were you able to reproduce the issue on master (git) ?<br>
<br>
What are the exact order of command to reproduce the problem ?<br>
<br>
Thanks<br>
<br>
<div class="moz-cite-prefix">On 2016-03-08 06:43 PM, Jeffrey Chen wrote:<br>
</div>
<blockquote>
<div id="divtagdefaultwrapper" style="font-size:12pt; color:#000000; background-color:#FFFFFF; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi LTTng community:</p>
<p><br>
</p>
<p>We are using LTTng for our production environment. We have been noticing segfault problem when our process exit. We think the problem is due to the background thread continue to write traces, while LTTng cleanup its memory. We do not know how to fix this
issue without changing LTTng code base. I have written a very simple app that could repro the problem. Most of my codes are copy pasted from the LTTng doc sample. Is there any fix that the LTTng side could do?</p>
<p><br>
</p>
<p><br>
</p>
<p><b><span style="text-decoration:underline">hello.c</span></b><br>
</p>
<p>#include <stdio.h><br>
#include "hello-tp.h"<br>
#include <pthread.h><br>
<br>
<br>
void* doSomeThing(void *arg)<br>
{<br>
int x;<br>
for (x = 0; x < 100000; ++x) {<br>
tracepoint(hello_world, my_first_tracepoint, x, "test");<br>
}<br>
}<br>
<br>
int main(int argc, char *argv[])<br>
{<br>
int x;<br>
<br>
getchar();<br>
<br>
pthread_t inc_x_thread;<br>
<br>
if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {<br>
<br>
fprintf(stderr, "Error creating thread\n");<br>
return 1;<br>
}<br>
<br>
tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");<br>
<br>
return 0;<br>
}</p>
<p><br>
</p>
<p><b><span style="text-decoration:underline">hello-tp.c</span></b><br>
</p>
<p>#define TRACEPOINT_CREATE_PROBES<br>
#define TRACEPOINT_DEFINE<br>
<br>
#include "hello-tp.h"<br>
</p>
<p><br>
</p>
<p><b><span style="text-decoration:underline">hello-tp.h</span></b></p>
<p>#undef TRACEPOINT_PROVIDER<br>
#define TRACEPOINT_PROVIDER hello_world<br>
<br>
#undef TRACEPOINT_INCLUDE<br>
#define TRACEPOINT_INCLUDE "./hello-tp.h"<br>
<br>
#if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)<br>
#define _HELLO_TP_H<br>
<br>
#include <lttng/tracepoint.h><br>
<br>
TRACEPOINT_EVENT(<br>
hello_world,<br>
my_first_tracepoint,<br>
TP_ARGS(<br>
int, my_integer_arg,<br>
char*, my_string_arg<br>
),<br>
TP_FIELDS(<br>
ctf_string(my_string_field, my_string_arg)<br>
ctf_integer(int, my_integer_field, my_integer_arg)<br>
)<br>
)<br>
<br>
#endif /* _HELLO_TP_H */<br>
<br>
#include <lttng/tracepoint-event.h><br>
</p>
<p><br>
</p>
<p><b><span style="text-decoration:underline">Compile</span></b></p>
<p>gcc -c -I. hello-tp.c<br>
gcc -c hello.c<br>
gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread<br>
<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
</div>
<br>
<br>
<pre>_______________________________________________
lttng-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><a class="moz-txt-link-freetext" href="http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" target="_blank">http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br></pre>
</blockquote>
<br>
<pre class="moz-signature">--
Jonathan R. Julien
Efficios</pre>
<br>
_______________________________________________<br>
lttng-dev mailing list<br>
lttng-dev@lists.lttng.org<br>
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev<br>
</blockquote>
</div>
<br>
<div>-- <br>
</div>
<div>Mathieu Desnoyers<br>
EfficiOS Inc.<br>
http://www.efficios.com</div>
</div>
<br>
_______________________________________________<br>
lttng-dev mailing list<br>
lttng-dev@lists.lttng.org<br>
http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev<br>
</blockquote>
</div>
<div><br>
</div>
<div>-- <br>
</div>
<div>Mathieu Desnoyers<br>
EfficiOS Inc.<br>
http://www.efficios.com</div>
</div>
</div>
</div>
</div>
<br>
</blockquote>
</div>
<div><br>
</div>
<div>-- <br>
</div>
<div>Mathieu Desnoyers<br>
EfficiOS Inc.<br>
http://www.efficios.com</div>
</div>
</div>
</div>
</div>
<br>
</blockquote>
</div>
<br>
<div>-- <br>
</div>
<div>Mathieu Desnoyers<br>
EfficiOS Inc.<br>
http://www.efficios.com</div>
</div>
</div>
</div>
</div>
<br>
</blockquote>
</div>
<br>
<div>-- <br>
</div>
<div>Mathieu Desnoyers<br>
EfficiOS Inc.<br>
http://www.efficios.com</div>
</div>
</div>
</div>
</div>
<br>
</blockquote>
</div>
<div><br>
</div>
<div>-- <br>
</div>
<div>Mathieu Desnoyers<br>
EfficiOS Inc.<br>
http://www.efficios.com</div>
</div>
</div>
</div>
</div>
</body>
</html>