[lttng-dev] segfault when exiting process with background thread

Tue Mar 15 19:00:18 UTC 2016

Commit merged into master, thanks for your feedback! 

Mathieu 

----- On Mar 15, 2016, at 2:48 PM, Jeffrey Chen <cpthk at hotmail.com> wrote: 

> Hi Mathieu:

> Will this fix be checked-in? Thanks.

> From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
> Sent: Thursday, March 10, 2016 3:27 PM
> To: Jeffrey Chen
> Cc: Jonathan Rajotte Julien; lttng-dev
> Subject: Re: segfault when exiting process with background thread

> ----- On Mar 10, 2016, at 5:59 PM, Jeffrey Chen <cpthk at hotmail.com> wrote:

>> Thanks, Mathieu.

>> Yes, the fix works as a workaround. This is also the same workaround we have
>> now.

>> It seems the problem is due to a race condition, that it checked whether "
>> tracepoint_dlopen.rcu_read_lock_sym_bp " is not NULL. If not NULL, it calls to
>> that. But, at this moment, memset cleared the memory.

>> This fix is okay with us, but I am not certain if it would break other
>> component, since I do not understand LTTng enough to tell. Thanks.

> In addition to memset clearing that memory, dlclose() of the
> lttng-ust-tracepoint
> shared object will clear the memory containing the called functions. Therefore,
> we need to skip both steps of the destructor if we want threads to survive after
> this destructor execution.

> Thanks,

> Mathieu

>> From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
>> Sent: Thursday, March 10, 2016 10:52 AM
>> To: Jeffrey Chen
>> Cc: Jonathan Rajotte Julien; lttng-dev
>> Subject: Re: segfault when exiting process with background thread
>> Hi Jeffrey,

>> I CC'd your @live.com address on the RFC patch. Please let me know if the
>> approach
>> is OK with you (and try it out).

>> See:
>> https://lists.lttng.org/pipermail/lttng-dev/2016-March/025608.html

>> Thanks,

>> Mathieu

>> ----- On Mar 10, 2016, at 1:35 PM, Jeffrey Chen <cpthk at hotmail.com> wrote:

>>> Thanks Mathieu.

>>> Is there a plan at LTTng side to fix this issue? If so, we could wait for the
>>> fix. If not, we will have to workaround the problem for now (probably by
>>> applying your fix ourselves). Thanks.

>>> From: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
>>> Sent: Tuesday, March 8, 2016 6:09 PM
>>> To: Jonathan Rajotte Julien
>>> Cc: lttng-dev
>>> Subject: Re: [lttng-dev] segfault when exiting process with background thread
>>> The attached patch works around the issue. The problem is
>>> the tracepoint destructors that run in parallel with the thread
>>> that still exists.

>>> If we remove this, we leak the liblttng-ust-tracepoint.so.0 shared
>>> object on dlclose of the instrumented code. Not sure if we should
>>> care though...

>>> Thoughts ?

>>> Thanks,

>>> Mathieu

>>> ----- On Mar 8, 2016, at 7:35 PM, Mathieu Desnoyers
>>> <mathieu.desnoyers at efficios.com> wrote:

>>>> Reproduced it on master (ust and tools).

>>>> Repro with:

>>>> lttng create
>>>> lttng enable-event -u -a
>>>> lttng start

>>>> then run hello a few times, it eventually segfaults in the spawned thread.

>>>> Looking into it.

>>>> Thanks,

>>>> Mathieu

>>>> ----- On Mar 8, 2016, at 7:12 PM, Jonathan Rajotte Julien
>>>> <Jonathan.rajotte-julien at efficios.com> wrote:

>>>>> Hi Jeffrey,

>>>>> Could you provide the version for lttng-ust, lttng-tools, lttng-modules (if
>>>>> installed)?

>>>>> Were you able to reproduce the issue on master (git) ?

>>>>> What are the exact order of command to reproduce the problem ?

>>>>> Thanks

>>>>> On 2016-03-08 06:43 PM, Jeffrey Chen wrote:

>>>>>> Hi LTTng community:

>>>>>> We are using LTTng for our production environment. We have been noticing
>>>>>> segfault problem when our process exit. We think the problem is due to the
>>>>>> background thread continue to write traces, while LTTng cleanup its memory. We
>>>>>> do not know how to fix this issue without changing LTTng code base. I have
>>>>>> written a very simple app that could repro the problem. Most of my codes are
>>>>>> copy pasted from the LTTng doc sample. Is there any fix that the LTTng side
>>>>>> could do?

>>>>>> hello.c

>>>>>> #include <stdio.h>
>>>>>> #include "hello-tp.h"
>>>>>> #include <pthread.h>

>>>>>> void* doSomeThing(void *arg)
>>>>>> {
>>>>>> int x;
>>>>>> for (x = 0; x < 100000; ++x) {
>>>>>> tracepoint(hello_world, my_first_tracepoint, x, "test");
>>>>>> }
>>>>>> }

>>>>>> int main(int argc, char *argv[])
>>>>>> {
>>>>>> int x;

>>>>>> getchar();

>>>>>> pthread_t inc_x_thread;

>>>>>> if(pthread_create(&inc_x_thread, NULL, doSomeThing, NULL)) {

>>>>>> fprintf(stderr, "Error creating thread\n");
>>>>>> return 1;
>>>>>> }

>>>>>> tracepoint(hello_world, my_first_tracepoint, 23, "hi there!");

>>>>>> return 0;
>>>>>> }

>>>>>> hello-tp.c

>>>>>> #define TRACEPOINT_CREATE_PROBES
>>>>>> #define TRACEPOINT_DEFINE

>>>>>> #include "hello-tp.h"

>>>>>> hello-tp.h

>>>>>> #undef TRACEPOINT_PROVIDER
>>>>>> #define TRACEPOINT_PROVIDER hello_world

>>>>>> #undef TRACEPOINT_INCLUDE
>>>>>> #define TRACEPOINT_INCLUDE "./hello-tp.h"

>>>>>> #if !defined(_HELLO_TP_H) || defined(TRACEPOINT_HEADER_MULTI_READ)
>>>>>> #define _HELLO_TP_H

>>>>>> #include <lttng/tracepoint.h>

>>>>>> TRACEPOINT_EVENT(
>>>>>> hello_world,
>>>>>> my_first_tracepoint,
>>>>>> TP_ARGS(
>>>>>> int, my_integer_arg,
>>>>>> char*, my_string_arg
>>>>>> ),
>>>>>> TP_FIELDS(
>>>>>> ctf_string(my_string_field, my_string_arg)
>>>>>> ctf_integer(int, my_integer_field, my_integer_arg)
>>>>>> )
>>>>>> )

>>>>>> #endif /* _HELLO_TP_H */

>>>>>> #include <lttng/tracepoint-event.h>

>>>>>> Compile

>>>>>> gcc -c -I. hello-tp.c
>>>>>> gcc -c hello.c
>>>>>> gcc -o hello hello.o hello-tp.o -llttng-ust -ldl -lpthread

>>>>>> _______________________________________________
>>>>>> lttng-dev mailing list lttng-dev at lists.lttng.org
>>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>>>> --
>>>>> Jonathan R. Julien
>>>>> Efficios

>>>>> _______________________________________________
>>>>> lttng-dev mailing list
>>>>> lttng-dev at lists.lttng.org
>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> http://www.efficios.com

>>>> _______________________________________________
>>>> lttng-dev mailing list
>>>> lttng-dev at lists.lttng.org
>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

> --
> Mathieu Desnoyers
> EfficiOS Inc.
> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160315/e33599a2/attachment.html>