<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>I came across something some time ago that I think might solve this problem. It goes back to making sure that the version of the synchronization function called from the wrapper is the same that would be called without the wrapper.</div><div><br></div><div>To specify the version the following instruction can be used:</div><div> ....... = dlvsym(RTLD_NEXT, "function_name", "GLIBC_2.3.2")</div><div><br></div><div>To know which version is normally used, you can use the following:</div><div> objdump -t <bin_name> | grep <func_name></div><div>where <bin_name> is the name of the program to be run, and <func_name> is the name of the function to be called from the wrapper.<br></div><div><br></div><div>For more information, check this link: <br> <a href="http://blog.fesnel.com/blog/2009/08/25/preloading-with-multiple-symbol-versions/">http://blog.fesnel.com/blog/2009/08/25/preloading-with-multiple-symbol-versions/</a></div><div><br></div><div>I hope this helps.<br></div><div><br></div><div><br></div><div><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jul 5, 2018 at 5:36 PM Shehab Elsayed <<a href="mailto:shehabyomn@gmail.com">shehabyomn@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Any suggestions where I should investigate to fix this bug? Also, please let me know whether the git repo I shared is sufficient.</div><div><br></div><div>Thanks,</div><div>Shehab<br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail-m_-8697634148803366064gmail_signature"><div dir="ltr"><font color="#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </font><font color="#888888"><a value="+20124498360"><span dir="ltr"></span></a><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></font></div></div></div>
<br><div class="gmail_quote">On Tue, May 29, 2018 at 10:36 AM, Shehab Elsayed <span dir="ltr"><<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I was wondering if you had a chance to look over this issue. Thanks!<br></div><div class="gmail_extra"><span><br clear="all"><div><div class="gmail-m_-8697634148803366064m_-827947492261404759gmail_signature"><div dir="ltr"><font color="#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </font><font color="#888888"><a value="+20124498360"><span dir="ltr"></span></a><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></font></div></div></div>
<br></span><div><div class="gmail-m_-8697634148803366064h5"><div class="gmail_quote">On Thu, May 3, 2018 at 4:37 PM, Shehab Elsayed <span dir="ltr"><<a href="mailto:shehabyomn@gmail.com" target="_blank">shehabyomn@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks for the reply. Here is a link to the repo <a href="https://github.com/ShehabElsayed/LTTng_debugging.git" target="_blank">https://github.com/ShehabElsayed/LTTng_debugging.git</a><br><br></div>It contains the LTTng source code I am using and one of the benchmarks that is causing problems.<br></div><div class="gmail_extra"><br clear="all"><div><div class="gmail-m_-8697634148803366064m_-827947492261404759m_-5569970550666406520gmail_signature"><div dir="ltr"><font color="#888888">Shehab Y. Elsayed, MSc.<br>
PhD Student<br>The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering<br>University of Toronto </font><font color="#888888"><a value="+20124498360"><span dir="ltr"></span></a><br>
E-mail: <a href="https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#" target="_blank">shehabyomn@gmail.com</a></font></div></div></div><div><div class="gmail-m_-8697634148803366064m_-827947492261404759h5">
<br><div class="gmail_quote">On Tue, May 1, 2018 at 10:39 AM, Jonathan Rajotte-Julien <span dir="ltr"><<a href="mailto:jonathan.rajotte-julien@efficios.com" target="_blank">jonathan.rajotte-julien@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Shehab,<br>
<br>
Please provide a link to a lttng-ust git tree or an actual patch (git format-patch).<br>
This will help any person who might be interested in helping you including<br>
myself.<br>
<br>
Cheers<br>
<div><div class="gmail-m_-8697634148803366064m_-827947492261404759m_-5569970550666406520h5"><br>
On Tue, May 01, 2018 at 09:31:52AM -0400, Shehab Elsayed wrote:<br>
> Hi,<br>
> <br>
> I am trying to create wrappers for pthreads conditional variables functions<br>
> (wait, signal and broadcast) to insert tracepoints. I followed the same<br>
> approach done for the mutex functions already provided with lttng, however<br>
> I am running into some problems. Mainly the applications seem to get stuck<br>
> when the conditional variable wrappers are enabled.<br>
> <br>
> Here is my implementation for the wrapper functions:<br>
> int pthread_cond_wait(pthread_cond_t *condition, pthread_mutex_t<br>
> *mutex)<br>
> <br>
> {<br>
> <br>
> static int (*cond_wait)(pthread_cond_t *, pthread_mutex_t<br>
> *);<br>
> int retval;<br>
> <br>
> <br>
> <br>
> if (!cond_wait)<br>
> {<br>
> <br>
> cond_wait = dlsym(RTLD_NEXT, "pthread_cond_wait");<br>
> <br>
> if (!cond_wait)<br>
> {<br>
> <br>
> if (thread_in_trace)<br>
> {<br>
> <br>
> <br>
> abort();<br>
> <br>
> <br>
> }<br>
> <br>
> fprintf(stderr, "unable to initialize pthread wrapper<br>
> library.\n");<br>
> return EINVAL;<br>
> <br>
> <br>
> }<br>
> <br>
> <br>
> }<br>
> <br>
> if (thread_in_trace)<br>
> {<br>
> <br>
> return cond_wait(condition, mutex);<br>
> <br>
> <br>
> }<br>
> <br>
> <br>
> <br>
> thread_in_trace =<br>
> 1;<br>
> <br>
> tracepoint(lttng_ust_pthread, pthread_cond_wait_begin, condition,<br>
> mutex,<br>
> <br>
> LTTNG_UST_CALLER_IP());<br>
> <br>
> retval = cond_wait(condition, mutex);<br>
> <br>
> tracepoint(lttng_ust_pthread, pthread_cond_wait_end, condition,<br>
> mutex,<br>
> <br>
> LTTNG_UST_CALLER_IP());<br>
> <br>
> thread_in_trace =<br>
> 0;<br>
> <br>
> return retval;<br>
> <br>
> }<br>
> <br>
> <br>
> <br>
> int pthread_cond_signal(pthread_cond_t<br>
> *condition)<br>
> <br>
> {<br>
> <br>
> static int (*cond_signal)(pthread_cond_t<br>
> *);<br>
> <br>
> int retval;<br>
> <br>
> <br>
> <br>
> if (!cond_signal)<br>
> {<br>
> <br>
> cond_signal = dlsym(RTLD_NEXT, "pthread_cond_signal");<br>
> <br>
> if (!cond_signal)<br>
> {<br>
> <br>
> if (thread_in_trace)<br>
> {<br>
> <br>
> <br>
> abort();<br>
> <br>
> <br>
> }<br>
> <br>
> fprintf(stderr, "unable to initialize pthread wrapper<br>
> library.\n");<br>
> return EINVAL;<br>
> <br>
> <br>
> }<br>
> <br>
> <br>
> }<br>
> <br>
> if (thread_in_trace)<br>
> {<br>
> <br>
> return cond_signal(condition);<br>
> <br>
> <br>
> }<br>
> <br>
> <br>
> <br>
> thread_in_trace =<br>
> 1;<br>
> <br>
> tracepoint(lttng_ust_pthread, pthread_cond_signal_begin,<br>
> condition,<br>
> LTTNG_UST_CALLER_IP());<br>
> <br>
> retval = cond_signal(condition);<br>
> <br>
> tracepoint(lttng_ust_pthread, pthread_cond_signal_end,<br>
> condition,<br>
> LTTNG_UST_CALLER_IP());<br>
> <br>
> thread_in_trace =<br>
> 0;<br>
> <br>
> return retval;<br>
> <br>
> }<br>
> <br>
> <br>
> <br>
> int pthread_cond_broadcast(pthread_cond_t<br>
> *condition)<br>
> <br>
> {<br>
> <br>
> static int (*cond_broadcast)(pthread_cond_t<br>
> *);<br>
> int retval;<br>
> <br>
> <br>
> <br>
> if (!cond_broadcast)<br>
> {<br>
> <br>
> cond_broadcast = dlsym(RTLD_NEXT, "pthread_cond_broadcast");<br>
> <br>
> if (!cond_broadcast)<br>
> {<br>
> <br>
> if (thread_in_trace)<br>
> {<br>
> <br>
> <br>
> abort();<br>
> <br>
> <br>
> }<br>
> <br>
> fprintf(stderr, "unable to initialize pthread wrapper<br>
> library.\n");<br>
> return EINVAL;<br>
> <br>
> <br>
> }<br>
> <br>
> <br>
> }<br>
> <br>
> if (thread_in_trace)<br>
> {<br>
> <br>
> return cond_broadcast(condition);<br>
> <br>
> <br>
> }<br>
> <br>
> <br>
> <br>
> thread_in_trace =<br>
> 1;<br>
> <br>
> tracepoint(lttng_ust_pthread, pthread_cond_broadcast_begin,<br>
> condition,<br>
> LTTNG_UST_CALLER_IP());<br>
> <br>
> retval = cond_broadcast(condition);<br>
> <br>
> tracepoint(lttng_ust_pthread, pthread_cond_broadcast_end,<br>
> condition,<br>
> LTTNG_UST_CALLER_IP());<br>
> <br>
> thread_in_trace =<br>
> 0;<br>
> <br>
> return retval;<br>
> <br>
> }<br>
> <br>
> Things I have tried:<br>
> 1- Comment out pthread_cond_wait function --> segmentation fault<br>
> 2- Comment out pthread_cond_signal and _broadcast --> stuck (I can see the<br>
> program is still running but nor forward progress is being made) compared<br>
> to the case when the wrappers are not preloaded.<br>
> 3- Comment out all tracepoint related code (basically the wrapper just call<br>
> the corresponding pthreads function) --> same results as points 1 and 2.<br>
> <br>
> Any idea what might be causing this or how I could debug this problem?<br>
> <br>
> Thank you very much in advance.<br>
> <br>
> Best Regards,<br>
> Shehab<br>
<br>
> int pthread_cond_wait(pthread_cond_t *condition, pthread_mutex_t *mutex) <br>
> { <br>
> static int (*cond_wait)(pthread_cond_t *, pthread_mutex_t *); <br>
> int retval; <br>
> <br>
> if (!cond_wait) { <br>
> cond_wait = dlsym(RTLD_NEXT, "pthread_cond_wait"); <br>
> if (!cond_wait) { <br>
> if (thread_in_trace) { <br>
> abort(); <br>
> } <br>
> fprintf(stderr, "unable to initialize pthread wrapper library.\n"); <br>
> return EINVAL; <br>
> } <br>
> } <br>
> if (thread_in_trace) { <br>
> return cond_wait(condition, mutex); <br>
> } <br>
> <br>
> thread_in_trace = 1; <br>
> tracepoint(lttng_ust_pthread, pthread_cond_wait_begin, condition, mutex, <br>
> LTTNG_UST_CALLER_IP()); <br>
> retval = cond_wait(condition, mutex); <br>
> tracepoint(lttng_ust_pthread, pthread_cond_wait_end, condition, mutex, <br>
> LTTNG_UST_CALLER_IP()); <br>
> thread_in_trace = 0; <br>
> return retval; <br>
> } <br>
> <br>
> int pthread_cond_signal(pthread_cond_t *condition) <br>
> { <br>
> static int (*cond_signal)(pthread_cond_t *); <br>
> int retval; <br>
> <br>
> if (!cond_signal) { <br>
> cond_signal = dlsym(RTLD_NEXT, "pthread_cond_signal"); <br>
> if (!cond_signal) { <br>
> if (thread_in_trace) { <br>
> abort(); <br>
> } <br>
> fprintf(stderr, "unable to initialize pthread wrapper library.\n"); <br>
> return EINVAL; <br>
> } <br>
> } <br>
> if (thread_in_trace) { <br>
> return cond_signal(condition); <br>
> } <br>
> <br>
> thread_in_trace = 1; <br>
> tracepoint(lttng_ust_pthread, pthread_cond_signal_begin, condition, <br>
> LTTNG_UST_CALLER_IP()); <br>
> retval = cond_signal(condition); <br>
> tracepoint(lttng_ust_pthread, pthread_cond_signal_end, condition, <br>
> LTTNG_UST_CALLER_IP()); <br>
> thread_in_trace = 0; <br>
> return retval; <br>
> } <br>
> <br>
> int pthread_cond_broadcast(pthread_cond_t *condition) <br>
> { <br>
> static int (*cond_broadcast)(pthread_cond_t *); <br>
> int retval; <br>
> <br>
> if (!cond_broadcast) { <br>
> cond_broadcast = dlsym(RTLD_NEXT, "pthread_cond_broadcast"); <br>
> if (!cond_broadcast) { <br>
> if (thread_in_trace) { <br>
> abort(); <br>
> } <br>
> fprintf(stderr, "unable to initialize pthread wrapper library.\n"); <br>
> return EINVAL; <br>
> } <br>
> } <br>
> if (thread_in_trace) { <br>
> return cond_broadcast(condition); <br>
> } <br>
> <br>
> thread_in_trace = 1; <br>
> tracepoint(lttng_ust_pthread, pthread_cond_broadcast_begin, condition, <br>
> LTTNG_UST_CALLER_IP()); <br>
> retval = cond_broadcast(condition); <br>
> tracepoint(lttng_ust_pthread, pthread_cond_broadcast_end, condition, <br>
> LTTNG_UST_CALLER_IP()); <br>
> thread_in_trace = 0; <br>
> return retval; <br>
> } <br>
<br>
</div></div>> _______________________________________________<br>
> lttng-dev mailing list<br>
> <a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br>
> <a href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" rel="noreferrer" target="_blank">https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br>
<span class="gmail-m_-8697634148803366064m_-827947492261404759m_-5569970550666406520HOEnZb"><font color="#888888"><br>
<br>
-- <br>
Jonathan Rajotte-Julien<br>
EfficiOS<br>
</font></span></blockquote></div><br></div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</blockquote></div>