<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div><span id="zwchr" data-marker="__DIVIDER__">----- On Mar 22, 2018, at 12:24 PM, Shehab Elsayed <shehabyomn@gmail.com> wrote:<br></span></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr">Actually, i am not sure if this would help. I have been trying to reproduce the problem on a different machine, but I can't. Even without the patches !!!!!</div></blockquote><div><br></div><div>Does it have the same glibc version ?<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Thanks,<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Mathieu<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><span data-mce-style="color: #888888;" style="color: #888888;" color="#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </span><span data-mce-style="color: #888888;" style="color: #888888;" color="#888888"><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></span><br data-mce-bogus="1"></div></div></div><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 8:13 PM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><span class=""><div><span id="m_1287507653871511028zwchr">----- On Mar 21, 2018, at 8:01 PM, Shehab Elsayed <<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>> wrote:<br></span></div></span><div><span class=""><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div><div>Just to clarify more what I meant by custom events; I have new tracepoints added in the source code of the benchmark. However, I don't enable the corresponding events when I do the actual tracing.<br><br></div>This is the sequence followed in building the benchmark:<br>gcc-7.2 -c -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112 -std=c11 -g -fno-strict-aliasing -DLTTNG_INST lu.c<br>gcc-7.2 -O2 -pthread -D_XOPEN_SOURCE=500 -D_POSIX_C_SOURCE=200112 -std=c11 -g -fno-strict-aliasing -DLTTNG_INST -o LU_NCB lu.o ../../instrumentation/lttng_tp/tp.o -lm -llttng-ust -ldl<br><br></div>LTTNG_INST is just a preprocessor flag I have and tp.o is my custom tracepoints</div></blockquote></span><div>Could you share a repository with your custom instrumentation on github, so I could<br></div><div>try it out ?<br></div><br><div>My current problem is that I cannot reproduce your issue on my end.<br></div><div><div class="h5"><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><br></div><div class="gmail_extra"><br clear="all"><div><div class="m_1287507653871511028gmail_signature"><div dir="ltr"><span style="color:#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </span><span style="color:#888888"><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></span><br></div></div></div><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 7:55 PM, Shehab Elsayed <span dir="ltr"><<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Still running into same problem. I attached the debug trace I got after applying the 2 patches.<br><br></div><div>The benchmark I am running also includes some custom created tracepoints. I am not adding the events being traced in the files I have provided. Do you think this might be causing a problem when I have tracpoints from 2 different packages?<br></div></div><div class="gmail_extra"><span><br clear="all"><div><div class="m_1287507653871511028m_1944747827822555580gmail_signature"><div dir="ltr"><span style="color:#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </span><span style="color:#888888"><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></span><br></div></div></div>
<br></span><div><div class="m_1287507653871511028h5"><div class="gmail_quote">On Wed, Mar 21, 2018 at 4:22 PM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><span><div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362zwchr">----- On Mar 21, 2018, at 1:55 PM, Shehab Elsayed <<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>> wrote:<br></span></div></span><div><span><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div><div><div><div><div><div>I am so sorry for the late replies.<br><br></div>I have tried the first patch you sent and the problem still happens (although seemingly less frequently especially with debugging enabled!!!!). I have attached the output I got from one of the erroneous runs.<br><br></div>I will try the updated patch and let you know how it goes.<br><br></div>Also, I am not sure if these points make any difference:<br></div>1- The error actually happens at the end of the application. It actually finishes execution, but then something goes wrong.<br></div>2- I run into this problem only for some of the benchmarks (or at least the problems happens much less frequently for others that I didn't run into it, not yet)<br><br></div>Thanks you very much, and sorry again for the late replies.</div></blockquote><br></span><div>No worries! Looking through your log, I notice that pthread set cancel state has problems when<br></div><div>called from application threads. We do not restore the original thread's cancel state. Can you try</div><div>with the attached patch applied on top of the previous one ?<br></div><div><div class="m_1287507653871511028m_1944747827822555580h5"><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><br></div><div class="gmail_extra"><br clear="all"><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362gmail_signature"><div dir="ltr"><span style="color:#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </span><span style="color:#888888"><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></span><br></div></div></div><br><div class="gmail_quote">On Wed, Mar 21, 2018 at 1:27 PM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><span><div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210zwchr">----- On Mar 20, 2018, at 5:42 PM, Mathieu Desnoyers <<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>> wrote:<br></span></div></span><div><span><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210zwchr">----- On Mar 20, 2018, at 4:58 PM, Mathieu Desnoyers <<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>> wrote:<br></span></div><div><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210zwchr">----- On Mar 20, 2018, at 12:07 PM, Mathieu Desnoyers <<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>> wrote:<br></span></div><div><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><br><div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210zwchr">----- On Mar 19, 2018, at 4:21 PM, Shehab Elsayed <<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>> wrote:<br></span></div><div><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div>I did "echo "-1" > <span class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210gmail-m_5949702530543711213gmail-im">/proc/sys/kernel/perf_event_paranoid</span>" and made sure the value was actually written by "cat /proc/sys/kernel/perf_event_paranoid"<br><br></div><div>It executed normally 2 times but then got the same error.</div></div></blockquote><br><div>Can you provide the output when reproducing the issue with the<br></div><div>LTTNG_UST_DEBUG=1 environment variable set when starting<br></div><div>your test program ?</div></div></div></blockquote><div>I just noticed something that might explain what goes wrong here:<br></div><br><div>lttng-context-perf-counters.c: add_thread_field() grabs the ust_lock(). Pthread mutex<br></div><div>in your case is instrumented with the pthread wrapper. This "add_thread_field" is invoked<br></div><div>the first time the perf counter is hit by each given thread. When this happens, the<br></div><div>instrumented pthread mutex will try to call into the instrumentation tracepoint again,<br></div><div>which will call add_thread_field() (again), and so on until we reach the libringbuffer<br></div><div>"lib_ring_buffer_nesting" threshold of 4 calls deep.<br></div><br><div>I suspect this situation where we recursively call add_thread_field is unexpected,<br></div><div>which may trigger your double-free here.<br></div><br><div>Will investigate more.</div></div></div></blockquote><div>Can you try with the attached patch applied ?</div></div></div></blockquote></span><div>Here is an updated v2 of the patch which tests the notrace tls counter sooner (before evaluating<br></div><div>filter). Please let me know how it goes.<br></div><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362h5"><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><br><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div><br><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><br><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div><br><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div><br><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><br></div><div class="gmail_extra"><br clear="all"><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210gmail_signature"><div dir="ltr"><span style="color:#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </span><span style="color:#888888"><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></span><br></div></div></div><br><div class="gmail_quote">On Mon, Mar 19, 2018 at 4:01 PM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><span><br><br><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213zwchr">----- On Mar 19, 2018, at 3:53 PM, Shehab Elsayed <<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>> wrote:<br></span></span><div><span><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div><div><span class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213gmail-im">cat /proc/sys/kernel/perf_event_paranoid</span> ---> returns 1<br><br></div>I run the program as a normal user<br><br></div>The glibc version I get by running "ldd --version" is 2.17</div></blockquote><br></span><div>Can you reproduce the issue after you do this as root ?<br></div><br><div>echo "-1" > <span class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213gmail-im">/proc/sys/kernel/perf_event_paranoid</span><br></div><br><div>Based on this documentation of the Linux kernel:<br></div><br><div>Documentation/sysctl/kernel.txt:<br></div><br><div>perf_event_paranoid:<br><br>Controls use of the performance events system by unprivileged<br>users (without CAP_SYS_ADMIN). The default value is 2.<br><br> -1: Allow use of (almost) all events by all users<br> Ignore mlock limit after perf_event_mlock_kb without CAP_IPC_LOCK<br>>=0: Disallow ftrace function tracepoint by users without CAP_SYS_ADMIN<br> Disallow raw tracepoint access by users without CAP_SYS_ADMIN<br>>=1: Disallow CPU event access by users without CAP_SYS_ADMIN<br>>=2: Disallow kernel profiling by users without CAP_SYS_ADMIN<br><br></div><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210h5"><div>Thanks,<br></div><br><div>Mathieu<br></div><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><br></div><div class="gmail_extra"><br clear="all"><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213gmail_signature"><div dir="ltr"><span style="color:#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </span><span style="color:#888888"><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></span><br></div></div></div><br><div class="gmail_quote">On Mon, Mar 19, 2018 at 3:31 PM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><span><div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608zwchr">---- On Mar 19, 2018, at 3:26 PM, Mathieu Desnoyers <<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>> wrote:<br></span></div></span><div><span><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608zwchr">----- On Mar 19, 2018, at 2:26 PM, Shehab Elsayed <<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>> wrote:<br></span></div><div><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr">Yes, I tried with only one of those contexts and I still ran into the same problem.</div></blockquote><div>What is the setting returned by<br></div><br><div>cat /proc/sys/kernel/perf_event_paranoid<br></div><br><div>on your system ? And do you run your test program as root or normal user ?<br></div><br><div>Please CC the mailing list on your reply.</div></div></div></blockquote><br></span><div>I will also need to know what glibc version you have on your system.<br></div><br><div>Thanks,<br></div><br><div>Mathieu<br></div><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213h5"><br><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div><br><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><br></div><div class="gmail_extra"><br clear="all"><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608gmail_signature"><div dir="ltr"><span style="color:#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </span><span style="color:#888888"><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></span><br></div></div></div><br><div class="gmail_quote">On Mon, Mar 19, 2018 at 2:24 PM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><span><div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122zwchr">----- On Mar 19, 2018, at 12:36 PM, Shehab Elsayed <<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>> wrote:<br></span></div></span><div><span><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div><div><div><div><div><div><div>Hi Mathieu,<br><br>Thank you very much for your reply.<br><br>I manually built lttng-ust from source (commit #: 8a208943e21700211beee3ea64180a5a534c7d2a).<br><br></div>This is how I set up the tracing session:<br></div>1- lttng create lu_ncb_8_native -o {path}<br></div>2- lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_req<br> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_acq<br> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_trylock<br> lttng enable-event --userspace lttng_ust_pthread:pthread_mutex_lock_unlock<br></div>3- lttng add-context -u -t procname<br> lttng add-context -u -t vpid<br> lttng add-context -u -t pthread_id<br> lttng add-context -u -t vtid<br> lttng add-context -u -t ip<br> lttng add-context -u -t perf:thread:cpu-cycles<br> lttng add-context -u -t perf:thread:cycles<br> lttng add-context -u -t perf:thread:instructions<br></div>4- lttng start<br>5- LD_PRELOAD=/usr/local/lib/liblttng-ust-pthread-wrapper.so ./lu_ncb -p8 -n8096 -b32<br></div>6- lttng stop<br></div>7- lttng destroy</div></blockquote><br></span><div>Can you reproduce if you remove the following contexts ?<br></div><span><br><div> lttng add-context -u -t perf:thread:cpu-cycles<br> lttng add-context -u -t perf:thread:cycles<br> lttng add-context -u -t perf:thread:instructions</div><br></span><div>And if you only keep a single of those contexts ?<br></div><br><div>Thanks,<br></div><br><div>Mathieu<br></div><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608h5"><br><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><br><br><br></div><div class="gmail_extra"><br clear="all"><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122gmail_signature"><div dir="ltr"><span style="color:#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </span><span style="color:#888888"><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></span><br></div></div></div><br><div class="gmail_quote">On Mon, Mar 19, 2018 at 12:21 PM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><span><br><br><div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982zwchr">----- On Mar 16, 2018, at 5:37 PM, Shehab Elsayed <<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>> wrote:<br></span></div></span><div><span><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div>Hello All,<br><br></div>I am trying to instrument a <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.1">pthread</span> application using the provided <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.2">pthread</span> wrapper, but I sometimes run into a "Double free or corruption error (<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.3">fasttop</span>)" error. </div></div></div></div></div></div></div></div></div></div></blockquote><br></span><div>Please provide more information about the version of lttng-ust you are using, and how you setup<br></div><div>your tracing session.<br></div><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122h5"><div dir="ltr"><div><div><div><div><div><div><div><div><div><br><br></div>Here is a description of what I have tried and noticed:<br></div>1- The problem isn't consistent. It sometimes happen and sometimes works as expected.<br></div>2- From my experiments, the problem happens (more frequently at least) when adding performance counter contexts (I tried cycles, <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.4">cpu</span>_cycles and instructions).<br></div>3- I am testing using <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.5">lu</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.6">ncb</span> from splash3 benchmark suite after setting <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.7">LD</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.8">PRELOAD</span> to use the <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.9">pthread</span> wrapper as described in the <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.10">LTTng</span> documents.<br></div>4- Here is the <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.11">backtrace</span> printed after exiting:<br>======= <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.12">Backtrace</span>: =========<br>/lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.13">libc</span>.so.6([Thread 0x7ffff5611700 (<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.14">LWP</span> 97229) exited]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.15">usr</span>/local/lib/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.16">liblttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.17">ust</span>.so.0(<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.18">lttng</span>_destroy_context+0x35)[0x7ffff7471575]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.19">usr</span>/local/lib/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.20">liblttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.21">ust</span>.so.0(<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.22">lttng</span>_session_destroy+0x21c)[0x7ffff747363c]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.23">usr</span>/local/lib/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.24">liblttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.25">ust</span>.so.0(+0x1e906)[0x7ffff746d906]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.26">usr</span>/local/lib/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.27">liblttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.28">ust</span>.so.0(<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.29">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.30">ust</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.31">objd</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.32">unref</span>+0x9f)[0x7ffff746dccf]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.33">usr</span>/local/lib/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.34">liblttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.35">ust</span>.so.0(<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.36">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.37">ust</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.38">objd</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.39">unref</span>+0x9f)[0x7ffff746dccf]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.40">usr</span>/local/lib/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.41">liblttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.42">ust</span>.so.0(<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.43">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.44">ust</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.45">objd</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.46">unref</span>+0x9f)[0x7ffff746dccf]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.47">usr</span>/local/lib/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.48">liblttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.49">ust</span>.so.0(<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.50">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.51">ust</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.52">abi</span>_exit+0x68)[0x7ffff746ead8]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.53">usr</span>/local/lib/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.54">liblttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.55">ust</span>.so.0(+0x191d3)[0x7ffff74681d3]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.56">usr</span>/local/lib/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.57">liblttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.58">ust</span>.so.0(<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.59">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.60">ust</span>_exit+0x67)[0x7ffff745ed57]<br>/lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.61">ld</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.62">linux</span>-x86-64.so.2(+0xf85a)[0x7ffff7dec85a]<br>/lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.63">libc</span>.so.6(+0x38a49)[0x7ffff6ca6a49]<br>/lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.64">libc</span>.so.6(+0x38a95)[0x7ffff6ca6a95]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.65">aenao</span>-99/elsayed9/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.66">LTTng</span>/data/scripts/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.67">tmp</span>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.68">lu</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.69">ncb</span>[0x401b51]<br>/lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.70">libc</span>.so.6(__<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.71">libc</span>_start_main+0xf5)[0x7ffff6c8fb35]<br>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.72">aenao</span>-99/elsayed9/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.73">LTTng</span>/data/scripts/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.74">tmp</span>/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.75">lu</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.76">ncb</span>[0x401c44]<br></div>5- Also, this is a <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.77">backtrace</span> I obtained from <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.78">gdb</span>:<br>#0 0x00007ffff6eac1d7 in raise () from /lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.79">libc</span>.so.6<br>#1 0x00007ffff6ead8c8 in abort () from /lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.80">libc</span>.so.6<br>#2 0x00007ffff6eebf07 in __<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.81">libc</span>_message () from /lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.82">libc</span>.so.6<br>#3 0x00007ffff6ef3503 in _int_free () from /lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.83">libc</span>.so.6<br>#4 0x00007ffff768ad25 in <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.84">lttng</span>_destroy_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.85">perf</span>_counter_field (<br> field=<optimized out>) at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.86">lttng</span>-context-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.87">perf</span>-counters.c:418<br>#5 0x00007ffff767a575 in <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.88">lttng</span>_destroy_context (<br><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.89">ctx</span>=0x7ffff0011090) at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.90">lttng</span>-context.c:278<br>#6 0x00007ffff767c63c in _<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.91">lttng</span>_channel_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.92">unmap</span> (<br><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.93">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.94">chan</span>=0x7ffff0010f40) at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.95">lttng</span>-events.c:172<br>#7 <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.96">lttng</span>_session_destroy (session=0x7ffff0000900)<br> at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.97">lttng</span>-events.c:247<br>#8 0x00007ffff7676906 in <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.98">lttng</span>_release_session (<br><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.99">objd</span>=<optimized out>) at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.100">lttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.101">ust</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.102">abi</span>.c:601<br>#9 0x00007ffff7676ccf in <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.103">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.104">ust</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.105">objd</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.106">unref</span> (id=1, <br> is_owner=<optimized out>) at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.107">lttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.108">ust</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.109">abi</span>.c:216<br>#10 0x00007ffff7676ccf in <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.110">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.111">ust</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.112">objd</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.113">unref</span> (id=2, <br> is_owner=<optimized out>) at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.114">lttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.115">ust</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.116">abi</span>.c:216<br>#11 0x00007ffff7676ccf in <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.117">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.118">ust</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.119">objd</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.120">unref</span> (id=id@entry=18, <br> is_owner=is_owner@entry=1) at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.121">lttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.122">ust</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.123">abi</span>.c:216<br>#12 0x00007ffff7677ad8 in <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.124">objd</span>_table_destroy ()<br> at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.125">lttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.126">ust</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.127">abi</span>.c:235<br>#13 <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.128">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.129">ust</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.130">abi</span>_exit () at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.131">lttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.132">ust</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.133">abi</span>.c:1002<br>#14 0x00007ffff76711d3 in <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.134">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.135">ust</span>_cleanup (exiting=1)<br> at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.136">lttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.137">ust</span>-comm.c:1807<br>#15 0x00007ffff7667d57 in <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.138">lttng</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.139">ust</span>_exit ()<br> at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.140">lttng</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.141">ust</span>-comm.c:1874<br>#16 0x00007ffff7dec85a in _<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.142">dl</span>_<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.143">fini</span> ()<br> from /lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.144">ld</span>-<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.145">linux</span>-x86-64.so.2<br>#17 0x00007ffff6eafa49 in __run_exit_handlers ()<br> from /lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.146">libc</span>.so.6<br>#18 0x00007ffff6eafa95 in exit () from /lib64/<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.147">libc</span>.so.6<br>#19 0x0000000000401b51 in main (<span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.148">argc</span>=<optimized out>, <br><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.149">argv</span>=<optimized out>) at <span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.150">lu</span>.c:368<br><br></div>Any ideas, why this happens and how to fix it?<br><br></div>Thanks,<br></div><span id="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122m_-3928334406880738982:3qh.151">Shehab</span><br><div><div><div><div><br><br></div></div></div></div></div><br></div></div>_______________________________________________<br>lttng-dev mailing list<br><a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br><a href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" target="_blank">https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><span class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122HOEnZb"><span style="color:#888888"><br></span></span></blockquote></div><span class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608m_-2379611272366386122HOEnZb"><span style="color:#888888"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></span></span></div></div></blockquote></div></div><br></blockquote></div></div></div><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213m_4191731141621744608h5"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></div></div></div></div></blockquote></div></div><br></blockquote></div><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></div><br></blockquote></div></div></div><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210m_5949702530543711213h5"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></div></div></div></div></blockquote></div></div><br></blockquote></div></div></div><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362m_-8739052298097068210h5"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></div></div></div></div></blockquote></div></div><br></blockquote></div><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></div><br></blockquote></div><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></div><br></blockquote></div><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></div><br></blockquote></div></div></div><div><div class="m_1287507653871511028m_1944747827822555580m_-4375542666408511362h5"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></div></div></div></div></blockquote></div></div><br></blockquote></div></div></div><div><div class="m_1287507653871511028m_1944747827822555580h5"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></div></div></div></div></blockquote></div><br></div></div></div></blockquote></div></div><br></blockquote></div></div></div><div><div class="h5"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br data-mce-bogus="1"></div></div></div></div></div></blockquote></div></div><br></blockquote></div><div><br></div><div data-marker="__SIG_POST__">-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</div></div></body></html>