<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><div><span id="zwchr" data-marker="__DIVIDER__">----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote:<br></span></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Hello All,<br><br></div>I am trying to instrument a <span class="" style="" id=":3qh.1">pthread</span> application using the provided <span class="" style="" id=":3qh.2">pthread</span> wrapper, but I sometimes run into a "Double free or corruption error (<span class="" style="" id=":3qh.3">fasttop</span>)" error. </div></div></div></div></div></div></div></div></div></div></blockquote><div><br></div><div>Please provide more information about the version of lttng-ust you are using, and how you setup<br data-mce-bogus="1"></div><div>your tracing session.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Thanks,<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Mathieu<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div><div><div><div><div><div><div><div><div><br><br></div>Here is a description of what I have tried and noticed:<br></div>1- The problem isn't consistent. It sometimes happen and sometimes works as expected.<br></div>2- From my experiments, the problem happens (more frequently at least) when adding performance counter contexts (I tried cycles, <span class="" style="" id=":3qh.4">cpu</span>_cycles and instructions).<br></div>3- I am testing using <span class="" style="" id=":3qh.5">lu</span>_<span class="" style="" id=":3qh.6">ncb</span> from splash3 benchmark suite after setting <span class="" style="" id=":3qh.7">LD</span>_<span class="" style="" id=":3qh.8">PRELOAD</span> to use the <span class="" style="" id=":3qh.9">pthread</span> wrapper as described in the <span class="" style="" id=":3qh.10">LTTng</span> documents.<br></div>4- Here is the <span class="" style="" id=":3qh.11">backtrace</span> printed after exiting:<br>======= <span class="" style="" id=":3qh.12">Backtrace</span>: =========<br>/lib64/<span class="" style="" id=":3qh.13">libc</span>.so.6([Thread 0x7ffff5611700 (<span class="" style="" id=":3qh.14">LWP</span> 97229) exited]<br>/<span class="" style="" id=":3qh.15">usr</span>/local/lib/<span class="" style="" id=":3qh.16">liblttng</span>-<span class="" style="" id=":3qh.17">ust</span>.so.0(<span class="" style="" id=":3qh.18">lttng</span>_destroy_context+0x35)[0x7ffff7471575]<br>/<span class="" style="" id=":3qh.19">usr</span>/local/lib/<span class="" style="" id=":3qh.20">liblttng</span>-<span class="" style="" id=":3qh.21">ust</span>.so.0(<span class="" style="" id=":3qh.22">lttng</span>_session_destroy+0x21c)[0x7ffff747363c]<br>/<span class="" style="" id=":3qh.23">usr</span>/local/lib/<span class="" style="" id=":3qh.24">liblttng</span>-<span class="" style="" id=":3qh.25">ust</span>.so.0(+0x1e906)[0x7ffff746d906]<br>/<span class="" style="" id=":3qh.26">usr</span>/local/lib/<span class="" style="" id=":3qh.27">liblttng</span>-<span class="" style="" id=":3qh.28">ust</span>.so.0(<span class="" style="" id=":3qh.29">lttng</span>_<span class="" style="" id=":3qh.30">ust</span>_<span class="" style="" id=":3qh.31">objd</span>_<span class="" style="" id=":3qh.32">unref</span>+0x9f)[0x7ffff746dccf]<br>/<span class="" style="" id=":3qh.33">usr</span>/local/lib/<span class="" style="" id=":3qh.34">liblttng</span>-<span class="" style="" id=":3qh.35">ust</span>.so.0(<span class="" style="" id=":3qh.36">lttng</span>_<span class="" style="" id=":3qh.37">ust</span>_<span class="" style="" id=":3qh.38">objd</span>_<span class="" style="" id=":3qh.39">unref</span>+0x9f)[0x7ffff746dccf]<br>/<span class="" style="" id=":3qh.40">usr</span>/local/lib/<span class="" style="" id=":3qh.41">liblttng</span>-<span class="" style="" id=":3qh.42">ust</span>.so.0(<span class="" style="" id=":3qh.43">lttng</span>_<span class="" style="" id=":3qh.44">ust</span>_<span class="" style="" id=":3qh.45">objd</span>_<span class="" style="" id=":3qh.46">unref</span>+0x9f)[0x7ffff746dccf]<br>/<span class="" style="" id=":3qh.47">usr</span>/local/lib/<span class="" style="" id=":3qh.48">liblttng</span>-<span class="" style="" id=":3qh.49">ust</span>.so.0(<span class="" style="" id=":3qh.50">lttng</span>_<span class="" style="" id=":3qh.51">ust</span>_<span class="" style="" id=":3qh.52">abi</span>_exit+0x68)[0x7ffff746ead8]<br>/<span class="" style="" id=":3qh.53">usr</span>/local/lib/<span class="" style="" id=":3qh.54">liblttng</span>-<span class="" style="" id=":3qh.55">ust</span>.so.0(+0x191d3)[0x7ffff74681d3]<br>/<span class="" style="" id=":3qh.56">usr</span>/local/lib/<span class="" style="" id=":3qh.57">liblttng</span>-<span class="" style="" id=":3qh.58">ust</span>.so.0(<span class="" style="" id=":3qh.59">lttng</span>_<span class="" style="" id=":3qh.60">ust</span>_exit+0x67)[0x7ffff745ed57]<br>/lib64/<span class="" style="" id=":3qh.61">ld</span>-<span class="" style="" id=":3qh.62">linux</span>-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]<br>/lib64/<span class="" style="" id=":3qh.63">libc</span>.so.6(+0x38a49)[0x7ffff6ca6a49]<br>/lib64/<span class="" style="" id=":3qh.64">libc</span>.so.6(+0x38a95)[0x7ffff6ca6a95]<br>/<span class="" style="" id=":3qh.65">aenao</span>-99/elsayed9/<span class="" style="" id=":3qh.66">LTTng</span>/data/scripts/<span class="" style="" id=":3qh.67">tmp</span>/<span class="" style="" id=":3qh.68">lu</span>_<span class="" style="" id=":3qh.69">ncb</span>[0x401b51]<br>/lib64/<span class="" style="" id=":3qh.70">libc</span>.so.6(__<span class="" style="" id=":3qh.71">libc</span>_start_main+0xf5)[0x7ffff6c8fb35]<br>/<span class="" style="" id=":3qh.72">aenao</span>-99/elsayed9/<span class="" style="" id=":3qh.73">LTTng</span>/data/scripts/<span class="" style="" id=":3qh.74">tmp</span>/<span class="" style="" id=":3qh.75">lu</span>_<span class="" style="" id=":3qh.76">ncb</span>[0x401c44]<br></div>5- Also, this is a <span class="" style="" id=":3qh.77">backtrace</span> I obtained from <span class="" style="" id=":3qh.78">gdb</span>:<br>#0 0x00007ffff6eac1d7 in raise () from /lib64/<span class="" style="" id=":3qh.79">libc</span>.so.6<br>#1 0x00007ffff6ead8c8 in abort () from /lib64/<span class="" style="" id=":3qh.80">libc</span>.so.6<br>#2 0x00007ffff6eebf07 in __<span class="" style="" id=":3qh.81">libc</span>_message () from /lib64/<span class="" style="" id=":3qh.82">libc</span>.so.6<br>#3 0x00007ffff6ef3503 in _int_free () from /lib64/<span class="" style="" id=":3qh.83">libc</span>.so.6<br>#4 0x00007ffff768ad25 in <span class="" style="" id=":3qh.84">lttng</span>_destroy_<span class="" style="" id=":3qh.85">perf</span>_counter_field (<br> field=<optimized out>) at <span class="" style="" id=":3qh.86">lttng</span>-context-<span class="" style="" id=":3qh.87">perf</span>-counters.c:418<br>#5 0x00007ffff767a575 in <span class="" style="" id=":3qh.88">lttng</span>_destroy_context (<br><span class="" style="" id=":3qh.89">ctx</span>=0x7ffff0011090) at <span class="" style="" id=":3qh.90">lttng</span>-context.c:278<br>#6 0x00007ffff767c63c in _<span class="" style="" id=":3qh.91">lttng</span>_channel_<span class="" style="" id=":3qh.92">unmap</span> (<br><span class="" style="" id=":3qh.93">lttng</span>_<span class="" style="" id=":3qh.94">chan</span>=0x7ffff0010f40) at <span class="" style="" id=":3qh.95">lttng</span>-events.c:172<br>#7 <span class="" style="" id=":3qh.96">lttng</span>_session_destroy (session=0x7ffff0000900)<br> at <span class="" style="" id=":3qh.97">lttng</span>-events.c:247<br>#8 0x00007ffff7676906 in <span class="" style="" id=":3qh.98">lttng</span>_release_session (<br><span class="" style="" id=":3qh.99">objd</span>=<optimized out>) at <span class="" style="" id=":3qh.100">lttng</span>-<span class="" style="" id=":3qh.101">ust</span>-<span class="" style="" id=":3qh.102">abi</span>.c:601<br>#9 0x00007ffff7676ccf in <span class="" style="" id=":3qh.103">lttng</span>_<span class="" style="" id=":3qh.104">ust</span>_<span class="" style="" id=":3qh.105">objd</span>_<span class="" style="" id=":3qh.106">unref</span> (id=1, <br> is_owner=<optimized out>) at <span class="" style="" id=":3qh.107">lttng</span>-<span class="" style="" id=":3qh.108">ust</span>-<span class="" style="" id=":3qh.109">abi</span>.c:216<br>#10 0x00007ffff7676ccf in <span class="" style="" id=":3qh.110">lttng</span>_<span class="" style="" id=":3qh.111">ust</span>_<span class="" style="" id=":3qh.112">objd</span>_<span class="" style="" id=":3qh.113">unref</span> (id=2, <br> is_owner=<optimized out>) at <span class="" style="" id=":3qh.114">lttng</span>-<span class="" style="" id=":3qh.115">ust</span>-<span class="" style="" id=":3qh.116">abi</span>.c:216<br>#11 0x00007ffff7676ccf in <span class="" style="" id=":3qh.117">lttng</span>_<span class="" style="" id=":3qh.118">ust</span>_<span class="" style="" id=":3qh.119">objd</span>_<span class="" style="" id=":3qh.120">unref</span> (id=id@entry=18, <br> is_owner=is_owner@entry=1) at <span class="" style="" id=":3qh.121">lttng</span>-<span class="" style="" id=":3qh.122">ust</span>-<span class="" style="" id=":3qh.123">abi</span>.c:216<br>#12 0x00007ffff7677ad8 in <span class="" style="" id=":3qh.124">objd</span>_table_destroy ()<br> at <span class="" style="" id=":3qh.125">lttng</span>-<span class="" style="" id=":3qh.126">ust</span>-<span class="" style="" id=":3qh.127">abi</span>.c:235<br>#13 <span class="" style="" id=":3qh.128">lttng</span>_<span class="" style="" id=":3qh.129">ust</span>_<span class="" style="" id=":3qh.130">abi</span>_exit () at <span class="" style="" id=":3qh.131">lttng</span>-<span class="" style="" id=":3qh.132">ust</span>-<span class="" style="" id=":3qh.133">abi</span>.c:1002<br>#14 0x00007ffff76711d3 in <span class="" style="" id=":3qh.134">lttng</span>_<span class="" style="" id=":3qh.135">ust</span>_cleanup (exiting=1)<br> at <span class="" style="" id=":3qh.136">lttng</span>-<span class="" style="" id=":3qh.137">ust</span>-comm.c:1807<br>#15 0x00007ffff7667d57 in <span class="" style="" id=":3qh.138">lttng</span>_<span class="" style="" id=":3qh.139">ust</span>_exit ()<br> at <span class="" style="" id=":3qh.140">lttng</span>-<span class="" style="" id=":3qh.141">ust</span>-comm.c:1874<br>#16 0x00007ffff7dec85a in _<span class="" style="" id=":3qh.142">dl</span>_<span class="" style="" id=":3qh.143">fini</span> ()<br> from /lib64/<span class="" style="" id=":3qh.144">ld</span>-<span class="" style="" id=":3qh.145">linux</span>-x86-64.so.2<br>#17 0x00007ffff6eafa49 in __run_exit_handlers ()<br> from /lib64/<span class="" style="" id=":3qh.146">libc</span>.so.6<br>#18 0x00007ffff6eafa95 in exit () from /lib64/<span class="" style="" id=":3qh.147">libc</span>.so.6<br>#19 0x0000000000401b51 in main (<span class="" style="" id=":3qh.148">argc</span>=<optimized out>, <br><span class="" style="" id=":3qh.149">argv</span>=<optimized out>) at <span class="" style="" id=":3qh.150">lu</span>.c:368<br><br></div>Any ideas, why this happens and how to fix it?<br><br></div>Thanks,<br></div><span class="" style="" id=":3qh.151">Shehab</span><br><div><div><div><div><br><br></div></div></div></div></div><br>_______________________________________________<br>lttng-dev mailing list<br>lttng-dev@lists.lttng.org<br>https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev<br></blockquote></div><div><br></div><div data-marker="__SIG_POST__">-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</div></div></body></html>