[lttng-dev] Double free or corruption error (fasttop)

Mathieu Desnoyers mathieu.desnoyers at efficios.com
Mon Mar 19 15:26:13 EDT 2018 

----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <shehabyomn at gmail.com> wrote: 

> Yes, I tried with only one of those contexts and I still ran into the same
> problem.

What is the setting returned by 

cat /proc/sys/kernel/perf_event_paranoid 

on your system ? And do you run your test program as root or normal user ? 

Please CC the mailing list on your reply. 

Thanks, 

Mathieu 

> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
> shehabyomn at gmail.com ]

> On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers < [
> mailto:mathieu.desnoyers at efficios.com | mathieu.desnoyers at efficios.com ] >
> wrote:

>> ----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed < [
>> mailto:shehabyomn at gmail.com | shehabyomn at gmail.com ] > wrote:

>>> Hi Mathieu,

>>> Thank you very much for your reply.

>>> I manually built lttng-ust from source (commit #:
>>> 8a208943e21700211beee3ea64180a5a534c7d2a).

>>> This is how I set up the tracing session:
>>> 1- lttng create lu_ncb_8_native -o {path}
>>> 2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req
>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq
>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock
>>> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock
>>> 3- lttng add-context -u -t procname
>>> lttng add-context -u -t vpid
>>> lttng add-context -u -t pthread_id
>>> lttng add-context -u -t vtid
>>> lttng add-context -u -t ip
>>> lttng add-context -u -t perf:thread:cpu-cycles
>>> lttng add-context -u -t perf:thread:cycles
>>> lttng add-context -u -t perf:thread:instructions
>>> 4- lttng start
>>> 5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096
>>> -b32
>>> 6- lttng stop
>>> 7- lttng destroy

>> Can you reproduce if you remove the following contexts ?

>> lttng add-context -u -t perf:thread:cpu-cycles
>> lttng add-context -u -t perf:thread:cycles
>> lttng add-context -u -t perf:thread:instructions

>> And if you only keep a single of those contexts ?

>> Thanks,

>> Mathieu

>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: [ https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11# |
>>> shehabyomn at gmail.com ]

>>> On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers < [
>>> mailto:mathieu.desnoyers at efficios.com | mathieu.desnoyers at efficios.com ] >
>>> wrote:

>>>> ----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed < [
>>>> mailto:shehabyomn at gmail.com | shehabyomn at gmail.com ] > wrote:

>>>>> Hello All,

>>>>> I am trying to instrument a pthread application using the provided pthread
>>>>> wrapper, but I sometimes run into a "Double free or corruption error ( fasttop
>>>>> )" error.

>>>> Please provide more information about the version of lttng-ust you are using,
>>>> and how you setup
>>>> your tracing session.

>>>> Thanks,

>>>> Mathieu

>>>>> Here is a description of what I have tried and noticed:
>>>>> 1- The problem isn't consistent. It sometimes happen and sometimes works as
>>>>> expected.
>>>>> 2- From my experiments, the problem happens (more frequently at least) when
>>>>> adding performance counter contexts (I tried cycles, cpu _cycles and
>>>>> instructions).
>>>>> 3- I am testing using lu _ ncb from splash3 benchmark suite after setting LD _
>>>>> PRELOAD to use the pthread wrapper as described in the LTTng documents.
>>>>> 4- Here is the backtrace printed after exiting:
>>>>> ======= Backtrace : =========
>>>>> /lib64/ libc .so.6([Thread 0x7ffff5611700 ( LWP 97229) exited]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>> _destroy_context+0x35)[0x7ffff7471575]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng
>>>>> _session_destroy+0x21c)[0x7ffff747363c]
>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x1e906)[0x7ffff746d906]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>> +0x9f)[0x7ffff746dccf]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>> +0x9f)[0x7ffff746dccf]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ objd _ unref
>>>>> +0x9f)[0x7ffff746dccf]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _ abi
>>>>> _exit+0x68)[0x7ffff746ead8]
>>>>> / usr /local/lib/ liblttng - ust .so.0(+0x191d3)[0x7ffff74681d3]
>>>>> / usr /local/lib/ liblttng - ust .so.0( lttng _ ust _exit+0x67)[0x7ffff745ed57]
>>>>> /lib64/ ld - linux -x86-64.so.2(+0xf85a)[0x7ffff7dec85a]
>>>>> /lib64/ libc .so.6(+0x38a49)[0x7ffff6ca6a49]
>>>>> /lib64/ libc .so.6(+0x38a95)[0x7ffff6ca6a95]
>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401b51]
>>>>> /lib64/ libc .so.6(__ libc _start_main+0xf5)[0x7ffff6c8fb35]
>>>>> / aenao -99/elsayed9/ LTTng /data/scripts/ tmp / lu _ ncb [0x401c44]
>>>>> 5- Also, this is a backtrace I obtained from gdb :
>>>>> #0 0x00007ffff6eac1d7 in raise () from /lib64/ libc .so.6
>>>>> #1 0x00007ffff6ead8c8 in abort () from /lib64/ libc .so.6
>>>>> #2 0x00007ffff6eebf07 in __ libc _message () from /lib64/ libc .so.6
>>>>> #3 0x00007ffff6ef3503 in _int_free () from /lib64/ libc .so.6
>>>>> #4 0x00007ffff768ad25 in lttng _destroy_ perf _counter_field (
>>>>> field=<optimized out>) at lttng -context- perf -counters.c:418
>>>>> #5 0x00007ffff767a575 in lttng _destroy_context (
>>>>> ctx =0x7ffff0011090) at lttng -context.c:278
>>>>> #6 0x00007ffff767c63c in _ lttng _channel_ unmap (
>>>>> lttng _ chan =0x7ffff0010f40) at lttng -events.c:172
>>>>> #7 lttng _session_destroy (session=0x7ffff0000900)
>>>>> at lttng -events.c:247
>>>>> #8 0x00007ffff7676906 in lttng _release_session (
>>>>> objd =<optimized out>) at lttng - ust - abi .c:601
>>>>> #9 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=1,
>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>> #10 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=2,
>>>>> is_owner=<optimized out>) at lttng - ust - abi .c:216
>>>>> #11 0x00007ffff7676ccf in lttng _ ust _ objd _ unref (id=id at entry=18,
>>>>> is_owner=is_owner at entry=1) at lttng - ust - abi .c:216
>>>>> #12 0x00007ffff7677ad8 in objd _table_destroy ()
>>>>> at lttng - ust - abi .c:235
>>>>> #13 lttng _ ust _ abi _exit () at lttng - ust - abi .c:1002
>>>>> #14 0x00007ffff76711d3 in lttng _ ust _cleanup (exiting=1)
>>>>> at lttng - ust -comm.c:1807
>>>>> #15 0x00007ffff7667d57 in lttng _ ust _exit ()
>>>>> at lttng - ust -comm.c:1874
>>>>> #16 0x00007ffff7dec85a in _ dl _ fini ()
>>>>> from /lib64/ ld - linux -x86-64.so.2
>>>>> #17 0x00007ffff6eafa49 in __run_exit_handlers ()
>>>>> from /lib64/ libc .so.6
>>>>> #18 0x00007ffff6eafa95 in exit () from /lib64/ libc .so.6
>>>>> #19 0x0000000000401b51 in main ( argc =<optimized out>,
>>>>> argv =<optimized out>) at lu .c:368

>>>>> Any ideas, why this happens and how to fix it?

>>>>> Thanks,
>>>>> Shehab

>>>>> _______________________________________________
>>>>> lttng-dev mailing list
>>>>> [ mailto:lttng-dev at lists.lttng.org | lttng-dev at lists.lttng.org ]
>>>>> [ https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev |
>>>>> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev ]

>>>> --
>>>> Mathieu Desnoyers
>>>> EfficiOS Inc.
>>>> [ http://www.efficios.com/ | http://www.efficios.com ]

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> [ http://www.efficios.com/ | http://www.efficios.com ]

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20180319/0c853d46/attachment-0001.html>

More information about the lttng-dev mailing list