[lttng-dev] segfault when exiting process with background thread

[Mathieu Desnoyers]

----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cpthk at hotmail.com> wrote: 

> Thanks, Mathieu.

> Yes, the fix works as a workaround. This is also the same workaround we have
> now.

> It seems the problem is due to a race condition, that it checked whether "
> tracepoint_dlopen.rcu_read_lock_sym_bp " is not NULL. If not NULL, it calls to
> that. But, at this moment, memset cleared the memory.

> This fix is okay with us, but I am not certain if it would break other
> component, since I do not understand LTTng enough to tell. Thanks.

In addition to memset clearing that memory, dlclose() of the lttng-ust-tracepoint 
shared object will clear the memory containing the called functions. Therefore, 
we need to skip both steps of the destructor if we want threads to survive after 
this destructor execution. 

Thanks, 

Mathieu 

> From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> Sent: Thursday, March 10, 2016 10:52 AM
> To: Jeffrey Chen
> Cc: Jonathan Rajotte Julien; lttng-dev
> Subject: Re: segfault when exiting process with background thread
> Hi Jeffrey,

> I CC'd your @live.com address on the RFC patch. Please let me know if the
> approach
> is OK with you (and try it out).

> See:
> https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html

> Thanks,

> Mathieu

> ----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk at hotmail.com> wrote:

>> Thanks Mathieu.

>> Is there a plan at LTTng side to fix this issue? If so, we could wait for the
>> fix. If not, we will have to workaround the problem for now (probably by
>> applying your fix ourselves). Thanks.

>> From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
>> Sent: Tuesday, March 8, 2016 6:09 PM
>> To: Jonathan Rajotte Julien
>> Cc: lttng-dev
>> Subject: Re: [lttng-dev] segfault when exiting process with background thread
>> The attached patch works around the issue. The problem is
>> the tracepoint destructors that run in parallel with the thread
>> that still exists.

>> If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
>> object on dlclose of the instrumented code. Not sure if we should
>> care though...

>> Thoughts ?

>> Thanks,

>> Mathieu

>> ----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers
>> <mathieu.desnoyers at efficios.com> wrote:

>>> Reproduced it on master (ust and tools).

>>> Repro with:

>>> lttng create
>>> lttng enable-event -u -a
>>> lttng start

>>> then run hello a few times, it eventually segfaults in the spawned thread.

>>> Looking into it.

>>> Thanks,

>>> Mathieu

>>> ----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien
>>> <Jonathan.rajotte-julien at efficios.com> wrote:

>>>> Hi Jeffrey,

>>>> Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if
>>>> installed)?

>>>> Were you able to reproduce the issue on master (git) ?

>>>> What are the exact order of command to reproduce the problem ?

>>>> Thanks

>>>> On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

>>>>> Hi LTTng community:

>>>>> We are using LTTng for our production environment. We have been noticing
>>>>> segfault problem when our process exit. We think the problem is due to the
>>>>> background thread continue to write traces, while LTTng cleanup its memory. We
>>>>> do not know how to fix this issue without changing LTTng code base. I have
>>>>> written a very simple app that could repro the problem. Most of my codes are
>>>>> copy pasted from the LTTng doc sample. Is there any fix that the LTTng side
>>>>> could do?

>>>>> hello.c

>>>>> #include <stdio.h>
>>>>> #include "hello-tp.h"
>>>>> #include <pthread.h>

>>>>> void* doSomeThing(void *arg)
>>>>> {
>>>>> int x;
>>>>> for (x = 0; x < 100000; ++x) {
>>>>> tracepoint(hello_world, my_first_tracepoint, x, "test");
>>>>> }
>>>>> }

>>>>> int main(int argc, char *argv[])
>>>>> {
>>>>> int x;

>>>>> getchar();

>>>>> pthread_t inc_x_thread;

>>>>> if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

>>>>> fprintf(stderr, "Error creating thread\n");
>>>>> return 1;
>>>>> }

>>>>> tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

>>>>> return 0;
>>>>> }

>>>>> hello-tp.c

>>>>> #define TRACEPOINT_CREATE_PROBES
>>>>> #define TRACEPOINT_DEFINE

>>>>> #include "hello-tp.h"

>>>>> hello-tp.h

>>>>> #undef TRACEPOINT_PROVIDER
>>>>> #define TRACEPOINT_PROVIDER hello_world

>>>>> #undef TRACEPOINT_INCLUDE
>>>>> #define TRACEPOINT_INCLUDE "./hello-tp.h"

>>>>> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
>>>>> #define _HELLO_TP_H

>>>>> #include <lttng/tracepoint.h>

>>>>> TRACEPOINT_EVENT(
>>>>> hello_world,
>>>>> my_first_tracepoint,
>>>>> TP_ARGS(
>>>>> int, my_integer_arg,
>>>>> char*, my_string_arg
>>>>> ),
>>>>> TP_FIELDS(
>>>>> ctf_string(my_string_field, my_string_arg)
>>>>> ctf_integer(int, my_integer_field, my_integer_arg)
>>>>> )
>>>>> )

>>>>> #endif /* _HELLO_TP_H */

>>>>> #include <lttng/tracepoint-event.h>

>>>>> Compile

>>>>> gcc -c -I. hello-tp.c
>>>>> gcc -c hello.c
>>>>> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread

>>>>> _______________________________________________
>>>>> lttng-dev mailing list lttng-dev at lists.lttng.org
>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>>> --
>>>> Jonathan R. Julien
>>>> Efficios

>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> lttng-dev at lists.lttng.org
>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com

>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160310/d0f4ccf8/attachment.html>