<div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Hello All,<br><br></div>I am trying to instrument a <span class="" style="" id=":3qh.1" tabindex="-1">pthread</span> application using the provided <span class="" style="" id=":3qh.2" tabindex="-1">pthread</span> wrapper, but I sometimes run into a "Double free or corruption error (<span class="" style="" id=":3qh.3" tabindex="-1">fasttop</span>)" error. <br><br></div>Here is a description of what I have tried and noticed:<br></div>1- The problem isn't consistent. It sometimes happen and sometimes works as expected.<br></div>2- From my experiments, the problem happens (more frequently at least) when adding performance counter contexts (I tried cycles, <span class="" style="" id=":3qh.4" tabindex="-1">cpu</span>_cycles and instructions).<br></div>3- I am testing using <span class="" style="" id=":3qh.5" tabindex="-1">lu</span>_<span class="" style="" id=":3qh.6" tabindex="-1">ncb</span> from splash3 benchmark suite after setting <span class="" style="" id=":3qh.7" tabindex="-1">LD</span>_<span class="" style="" id=":3qh.8" tabindex="-1">PRELOAD</span> to use the <span class="" style="" id=":3qh.9" tabindex="-1">pthread</span> wrapper as described in the <span class="" style="" id=":3qh.10" tabindex="-1">LTTng</span> documents.<br></div>4- Here is the <span class="" style="" id=":3qh.11" tabindex="-1">backtrace</span> printed after exiting:<br>======= <span class="" style="" id=":3qh.12" tabindex="-1">Backtrace</span>: =========<br>/lib64/<span class="" style="" id=":3qh.13" tabindex="-1">libc</span>.so.6([Thread 0x7ffff5611700 (<span class="" style="" id=":3qh.14" tabindex="-1">LWP</span> 97229) exited]<br>/<span class="" style="" id=":3qh.15" tabindex="-1">usr</span>/local/lib/<span class="" style="" id=":3qh.16" tabindex="-1">liblttng</span>-<span class="" style="" id=":3qh.17" tabindex="-1">ust</span>.so.0(<span class="" style="" id=":3qh.18" tabindex="-1">lttng</span>_destroy_context+0x35)[0x7ffff7471575]<br>/<span class="" style="" id=":3qh.19" tabindex="-1">usr</span>/local/lib/<span class="" style="" id=":3qh.20" tabindex="-1">liblttng</span>-<span class="" style="" id=":3qh.21" tabindex="-1">ust</span>.so.0(<span class="" style="" id=":3qh.22" tabindex="-1">lttng</span>_session_destroy+0x21c)[0x7ffff747363c]<br>/<span class="" style="" id=":3qh.23" tabindex="-1">usr</span>/local/lib/<span class="" style="" id=":3qh.24" tabindex="-1">liblttng</span>-<span class="" style="" id=":3qh.25" tabindex="-1">ust</span>.so.0(+0x1e906)[0x7ffff746d906]<br>/<span class="" style="" id=":3qh.26" tabindex="-1">usr</span>/local/lib/<span class="" style="" id=":3qh.27" tabindex="-1">liblttng</span>-<span class="" style="" id=":3qh.28" tabindex="-1">ust</span>.so.0(<span class="" style="" id=":3qh.29" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.30" tabindex="-1">ust</span>_<span class="" style="" id=":3qh.31" tabindex="-1">objd</span>_<span class="" style="" id=":3qh.32" tabindex="-1">unref</span>+0x9f)[0x7ffff746dccf]<br>/<span class="" style="" id=":3qh.33" tabindex="-1">usr</span>/local/lib/<span class="" style="" id=":3qh.34" tabindex="-1">liblttng</span>-<span class="" style="" id=":3qh.35" tabindex="-1">ust</span>.so.0(<span class="" style="" id=":3qh.36" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.37" tabindex="-1">ust</span>_<span class="" style="" id=":3qh.38" tabindex="-1">objd</span>_<span class="" style="" id=":3qh.39" tabindex="-1">unref</span>+0x9f)[0x7ffff746dccf]<br>/<span class="" style="" id=":3qh.40" tabindex="-1">usr</span>/local/lib/<span class="" style="" id=":3qh.41" tabindex="-1">liblttng</span>-<span class="" style="" id=":3qh.42" tabindex="-1">ust</span>.so.0(<span class="" style="" id=":3qh.43" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.44" tabindex="-1">ust</span>_<span class="" style="" id=":3qh.45" tabindex="-1">objd</span>_<span class="" style="" id=":3qh.46" tabindex="-1">unref</span>+0x9f)[0x7ffff746dccf]<br>/<span class="" style="" id=":3qh.47" tabindex="-1">usr</span>/local/lib/<span class="" style="" id=":3qh.48" tabindex="-1">liblttng</span>-<span class="" style="" id=":3qh.49" tabindex="-1">ust</span>.so.0(<span class="" style="" id=":3qh.50" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.51" tabindex="-1">ust</span>_<span class="" style="" id=":3qh.52" tabindex="-1">abi</span>_exit+0x68)[0x7ffff746ead8]<br>/<span class="" style="" id=":3qh.53" tabindex="-1">usr</span>/local/lib/<span class="" style="" id=":3qh.54" tabindex="-1">liblttng</span>-<span class="" style="" id=":3qh.55" tabindex="-1">ust</span>.so.0(+0x191d3)[0x7ffff74681d3]<br>/<span class="" style="" id=":3qh.56" tabindex="-1">usr</span>/local/lib/<span class="" style="" id=":3qh.57" tabindex="-1">liblttng</span>-<span class="" style="" id=":3qh.58" tabindex="-1">ust</span>.so.0(<span class="" style="" id=":3qh.59" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.60" tabindex="-1">ust</span>_exit+0x67)[0x7ffff745ed57]<br>/lib64/<span class="" style="" id=":3qh.61" tabindex="-1">ld</span>-<span class="" style="" id=":3qh.62" tabindex="-1">linux</span>-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]<br>/lib64/<span class="" style="" id=":3qh.63" tabindex="-1">libc</span>.so.6(+0x38a49)[0x7ffff6ca6a49]<br>/lib64/<span class="" style="" id=":3qh.64" tabindex="-1">libc</span>.so.6(+0x38a95)[0x7ffff6ca6a95]<br>/<span class="" style="" id=":3qh.65" tabindex="-1">aenao</span>-99/elsayed9/<span class="" style="" id=":3qh.66" tabindex="-1">LTTng</span>/data/scripts/<span class="" style="" id=":3qh.67" tabindex="-1">tmp</span>/<span class="" style="" id=":3qh.68" tabindex="-1">lu</span>_<span class="" style="" id=":3qh.69" tabindex="-1">ncb</span>[0x401b51]<br>/lib64/<span class="" style="" id=":3qh.70" tabindex="-1">libc</span>.so.6(__<span class="" style="" id=":3qh.71" tabindex="-1">libc</span>_start_main+0xf5)[0x7ffff6c8fb35]<br>/<span class="" style="" id=":3qh.72" tabindex="-1">aenao</span>-99/elsayed9/<span class="" style="" id=":3qh.73" tabindex="-1">LTTng</span>/data/scripts/<span class="" style="" id=":3qh.74" tabindex="-1">tmp</span>/<span class="" style="" id=":3qh.75" tabindex="-1">lu</span>_<span class="" style="" id=":3qh.76" tabindex="-1">ncb</span>[0x401c44]<br></div>5- Also, this is a <span class="" style="" id=":3qh.77" tabindex="-1">backtrace</span> I obtained from <span class="" style="" id=":3qh.78" tabindex="-1">gdb</span>:<br>#0  0x00007ffff6eac1d7 in raise () from /lib64/<span class="" style="" id=":3qh.79" tabindex="-1">libc</span>.so.6<br>#1  0x00007ffff6ead8c8 in abort () from /lib64/<span class="" style="" id=":3qh.80" tabindex="-1">libc</span>.so.6<br>#2  0x00007ffff6eebf07 in __<span class="" style="" id=":3qh.81" tabindex="-1">libc</span>_message () from /lib64/<span class="" style="" id=":3qh.82" tabindex="-1">libc</span>.so.6<br>#3  0x00007ffff6ef3503 in _int_free () from /lib64/<span class="" style="" id=":3qh.83" tabindex="-1">libc</span>.so.6<br>#4  0x00007ffff768ad25 in <span class="" style="" id=":3qh.84" tabindex="-1">lttng</span>_destroy_<span class="" style="" id=":3qh.85" tabindex="-1">perf</span>_counter_field (<br>    field=<optimized out>) at <span class="" style="" id=":3qh.86" tabindex="-1">lttng</span>-context-<span class="" style="" id=":3qh.87" tabindex="-1">perf</span>-counters.c:418<br>#5  0x00007ffff767a575 in <span class="" style="" id=":3qh.88" tabindex="-1">lttng</span>_destroy_context (<br>    <span class="" style="" id=":3qh.89" tabindex="-1">ctx</span>=0x7ffff0011090) at <span class="" style="" id=":3qh.90" tabindex="-1">lttng</span>-context.c:278<br>#6  0x00007ffff767c63c in _<span class="" style="" id=":3qh.91" tabindex="-1">lttng</span>_channel_<span class="" style="" id=":3qh.92" tabindex="-1">unmap</span> (<br>    <span class="" style="" id=":3qh.93" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.94" tabindex="-1">chan</span>=0x7ffff0010f40) at <span class="" style="" id=":3qh.95" tabindex="-1">lttng</span>-events.c:172<br>#7  <span class="" style="" id=":3qh.96" tabindex="-1">lttng</span>_session_destroy (session=0x7ffff0000900)<br>    at <span class="" style="" id=":3qh.97" tabindex="-1">lttng</span>-events.c:247<br>#8  0x00007ffff7676906 in <span class="" style="" id=":3qh.98" tabindex="-1">lttng</span>_release_session (<br>    <span class="" style="" id=":3qh.99" tabindex="-1">objd</span>=<optimized out>) at <span class="" style="" id=":3qh.100" tabindex="-1">lttng</span>-<span class="" style="" id=":3qh.101" tabindex="-1">ust</span>-<span class="" style="" id=":3qh.102" tabindex="-1">abi</span>.c:601<br>#9  0x00007ffff7676ccf in <span class="" style="" id=":3qh.103" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.104" tabindex="-1">ust</span>_<span class="" style="" id=":3qh.105" tabindex="-1">objd</span>_<span class="" style="" id=":3qh.106" tabindex="-1">unref</span> (id=1, <br>    is_owner=<optimized out>) at <span class="" style="" id=":3qh.107" tabindex="-1">lttng</span>-<span class="" style="" id=":3qh.108" tabindex="-1">ust</span>-<span class="" style="" id=":3qh.109" tabindex="-1">abi</span>.c:216<br>#10 0x00007ffff7676ccf in <span class="" style="" id=":3qh.110" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.111" tabindex="-1">ust</span>_<span class="" style="" id=":3qh.112" tabindex="-1">objd</span>_<span class="" style="" id=":3qh.113" tabindex="-1">unref</span> (id=2, <br>    is_owner=<optimized out>) at <span class="" style="" id=":3qh.114" tabindex="-1">lttng</span>-<span class="" style="" id=":3qh.115" tabindex="-1">ust</span>-<span class="" style="" id=":3qh.116" tabindex="-1">abi</span>.c:216<br>#11 0x00007ffff7676ccf in <span class="" style="" id=":3qh.117" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.118" tabindex="-1">ust</span>_<span class="" style="" id=":3qh.119" tabindex="-1">objd</span>_<span class="" style="" id=":3qh.120" tabindex="-1">unref</span> (id=id@entry=18, <br>    is_owner=is_owner@entry=1) at <span class="" style="" id=":3qh.121" tabindex="-1">lttng</span>-<span class="" style="" id=":3qh.122" tabindex="-1">ust</span>-<span class="" style="" id=":3qh.123" tabindex="-1">abi</span>.c:216<br>#12 0x00007ffff7677ad8 in <span class="" style="" id=":3qh.124" tabindex="-1">objd</span>_table_destroy ()<br>    at <span class="" style="" id=":3qh.125" tabindex="-1">lttng</span>-<span class="" style="" id=":3qh.126" tabindex="-1">ust</span>-<span class="" style="" id=":3qh.127" tabindex="-1">abi</span>.c:235<br>#13 <span class="" style="" id=":3qh.128" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.129" tabindex="-1">ust</span>_<span class="" style="" id=":3qh.130" tabindex="-1">abi</span>_exit () at <span class="" style="" id=":3qh.131" tabindex="-1">lttng</span>-<span class="" style="" id=":3qh.132" tabindex="-1">ust</span>-<span class="" style="" id=":3qh.133" tabindex="-1">abi</span>.c:1002<br>#14 0x00007ffff76711d3 in <span class="" style="" id=":3qh.134" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.135" tabindex="-1">ust</span>_cleanup (exiting=1)<br>    at <span class="" style="" id=":3qh.136" tabindex="-1">lttng</span>-<span class="" style="" id=":3qh.137" tabindex="-1">ust</span>-comm.c:1807<br>#15 0x00007ffff7667d57 in <span class="" style="" id=":3qh.138" tabindex="-1">lttng</span>_<span class="" style="" id=":3qh.139" tabindex="-1">ust</span>_exit ()<br>    at <span class="" style="" id=":3qh.140" tabindex="-1">lttng</span>-<span class="" style="" id=":3qh.141" tabindex="-1">ust</span>-comm.c:1874<br>#16 0x00007ffff7dec85a in _<span class="" style="" id=":3qh.142" tabindex="-1">dl</span>_<span class="" style="" id=":3qh.143" tabindex="-1">fini</span> ()<br>   from /lib64/<span class="" style="" id=":3qh.144" tabindex="-1">ld</span>-<span class="" style="" id=":3qh.145" tabindex="-1">linux</span>-x86-64.so.2<br>#17 0x00007ffff6eafa49 in __run_exit_handlers ()<br>   from /lib64/<span class="" style="" id=":3qh.146" tabindex="-1">libc</span>.so.6<br>#18 0x00007ffff6eafa95 in exit () from /lib64/<span class="" style="" id=":3qh.147" tabindex="-1">libc</span>.so.6<br>#19 0x0000000000401b51 in main (<span class="" style="" id=":3qh.148" tabindex="-1">argc</span>=<optimized out>, <br>    <span class="" style="" id=":3qh.149" tabindex="-1">argv</span>=<optimized out>) at <span class="" style="" id=":3qh.150" tabindex="-1">lu</span>.c:368<br><br></div>Any ideas, why this happens and how to fix it?<br><br></div>Thanks,<br></div><span class="" style="" id=":3qh.151" tabindex="-1">Shehab</span><br><div><div><div><div><br><br></div></div></div></div></div>