[lttng-dev] Tracepoints for pthreads conditional variables
Shehab Elsayed
shehabyomn at gmail.com
Fri Apr 26 16:44:16 EDT 2019
I came across something some time ago that I think might solve this
problem. It goes back to making sure that the version of the
synchronization function called from the wrapper is the same that would be
called without the wrapper.
To specify the version the following instruction can be used:
....... = dlvsym(RTLD_NEXT, "function_name", "GLIBC_2.3.2")
To know which version is normally used, you can use the following:
objdump -t <bin_name> | grep <func_name>
where <bin_name> is the name of the program to be run, and <func_name> is
the name of the function to be called from the wrapper.
For more information, check this
link:
http://blog.fesnel.com/blog/2009/08/25/preloading-with-multiple-symbol-versions/
I hope this helps.
On Thu, Jul 5, 2018 at 5:36 PM Shehab Elsayed <shehabyomn at gmail.com> wrote:
> Any suggestions where I should investigate to fix this bug? Also, please
> let me know whether the git repo I shared is sufficient.
>
> Thanks,
> Shehab
>
> Shehab Y. Elsayed, MSc.
> PhD Student
> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
> University of Toronto
> E-mail: shehabyomn at gmail.com
> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>
> On Tue, May 29, 2018 at 10:36 AM, Shehab Elsayed <shehabyomn at gmail.com>
> wrote:
>
>> I was wondering if you had a chance to look over this issue. Thanks!
>>
>> Shehab Y. Elsayed, MSc.
>> PhD Student
>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>> University of Toronto
>> E-mail: shehabyomn at gmail.com
>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>
>> On Thu, May 3, 2018 at 4:37 PM, Shehab Elsayed <shehabyomn at gmail.com>
>> wrote:
>>
>>> Thanks for the reply. Here is a link to the repo
>>> https://github.com/ShehabElsayed/LTTng_debugging.git
>>>
>>> It contains the LTTng source code I am using and one of the benchmarks
>>> that is causing problems.
>>>
>>> Shehab Y. Elsayed, MSc.
>>> PhD Student
>>> The Edwards S. Rogers Sr. Dept. of Electrical and Computer Engineering
>>> University of Toronto
>>> E-mail: shehabyomn at gmail.com
>>> <https://webmail.rice.edu/imp/message.php?mailbox=INBOX&index=11#>
>>>
>>> On Tue, May 1, 2018 at 10:39 AM, Jonathan Rajotte-Julien <
>>> jonathan.rajotte-julien at efficios.com> wrote:
>>>
>>>> Hi Shehab,
>>>>
>>>> Please provide a link to a lttng-ust git tree or an actual patch (git
>>>> format-patch).
>>>> This will help any person who might be interested in helping you
>>>> including
>>>> myself.
>>>>
>>>> Cheers
>>>>
>>>> On Tue, May 01, 2018 at 09:31:52AM -0400, Shehab Elsayed wrote:
>>>> > Hi,
>>>> >
>>>> > I am trying to create wrappers for pthreads conditional variables
>>>> functions
>>>> > (wait, signal and broadcast) to insert tracepoints. I followed the
>>>> same
>>>> > approach done for the mutex functions already provided with lttng,
>>>> however
>>>> > I am running into some problems. Mainly the applications seem to get
>>>> stuck
>>>> > when the conditional variable wrappers are enabled.
>>>> >
>>>> > Here is my implementation for the wrapper functions:
>>>> > int pthread_cond_wait(pthread_cond_t *condition, pthread_mutex_t
>>>> > *mutex)
>>>> >
>>>> > {
>>>> >
>>>> > static int (*cond_wait)(pthread_cond_t *, pthread_mutex_t
>>>> > *);
>>>> > int retval;
>>>> >
>>>> >
>>>> >
>>>> > if (!cond_wait)
>>>> > {
>>>> >
>>>> > cond_wait = dlsym(RTLD_NEXT, "pthread_cond_wait");
>>>> >
>>>> > if (!cond_wait)
>>>> > {
>>>> >
>>>> > if (thread_in_trace)
>>>> > {
>>>> >
>>>> >
>>>> > abort();
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > fprintf(stderr, "unable to initialize pthread wrapper
>>>> > library.\n");
>>>> > return EINVAL;
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > if (thread_in_trace)
>>>> > {
>>>> >
>>>> > return cond_wait(condition, mutex);
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> > thread_in_trace =
>>>> > 1;
>>>> >
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_wait_begin, condition,
>>>> > mutex,
>>>> >
>>>> > LTTNG_UST_CALLER_IP());
>>>> >
>>>> > retval = cond_wait(condition, mutex);
>>>> >
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_wait_end, condition,
>>>> > mutex,
>>>> >
>>>> > LTTNG_UST_CALLER_IP());
>>>> >
>>>> > thread_in_trace =
>>>> > 0;
>>>> >
>>>> > return retval;
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> > int pthread_cond_signal(pthread_cond_t
>>>> > *condition)
>>>> >
>>>> > {
>>>> >
>>>> > static int (*cond_signal)(pthread_cond_t
>>>> > *);
>>>> >
>>>> > int retval;
>>>> >
>>>> >
>>>> >
>>>> > if (!cond_signal)
>>>> > {
>>>> >
>>>> > cond_signal = dlsym(RTLD_NEXT, "pthread_cond_signal");
>>>> >
>>>> > if (!cond_signal)
>>>> > {
>>>> >
>>>> > if (thread_in_trace)
>>>> > {
>>>> >
>>>> >
>>>> > abort();
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > fprintf(stderr, "unable to initialize pthread wrapper
>>>> > library.\n");
>>>> > return EINVAL;
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > if (thread_in_trace)
>>>> > {
>>>> >
>>>> > return cond_signal(condition);
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> > thread_in_trace =
>>>> > 1;
>>>> >
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_signal_begin,
>>>> > condition,
>>>> > LTTNG_UST_CALLER_IP());
>>>> >
>>>> > retval = cond_signal(condition);
>>>> >
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_signal_end,
>>>> > condition,
>>>> > LTTNG_UST_CALLER_IP());
>>>> >
>>>> > thread_in_trace =
>>>> > 0;
>>>> >
>>>> > return retval;
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> > int pthread_cond_broadcast(pthread_cond_t
>>>> > *condition)
>>>> >
>>>> > {
>>>> >
>>>> > static int (*cond_broadcast)(pthread_cond_t
>>>> > *);
>>>> > int retval;
>>>> >
>>>> >
>>>> >
>>>> > if (!cond_broadcast)
>>>> > {
>>>> >
>>>> > cond_broadcast = dlsym(RTLD_NEXT, "pthread_cond_broadcast");
>>>> >
>>>> > if (!cond_broadcast)
>>>> > {
>>>> >
>>>> > if (thread_in_trace)
>>>> > {
>>>> >
>>>> >
>>>> > abort();
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > fprintf(stderr, "unable to initialize pthread wrapper
>>>> > library.\n");
>>>> > return EINVAL;
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> > if (thread_in_trace)
>>>> > {
>>>> >
>>>> > return cond_broadcast(condition);
>>>> >
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> > thread_in_trace =
>>>> > 1;
>>>> >
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_broadcast_begin,
>>>> > condition,
>>>> > LTTNG_UST_CALLER_IP());
>>>> >
>>>> > retval = cond_broadcast(condition);
>>>> >
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_broadcast_end,
>>>> > condition,
>>>> > LTTNG_UST_CALLER_IP());
>>>> >
>>>> > thread_in_trace =
>>>> > 0;
>>>> >
>>>> > return retval;
>>>> >
>>>> > }
>>>> >
>>>> > Things I have tried:
>>>> > 1- Comment out pthread_cond_wait function --> segmentation fault
>>>> > 2- Comment out pthread_cond_signal and _broadcast --> stuck (I can
>>>> see the
>>>> > program is still running but nor forward progress is being made)
>>>> compared
>>>> > to the case when the wrappers are not preloaded.
>>>> > 3- Comment out all tracepoint related code (basically the wrapper
>>>> just call
>>>> > the corresponding pthreads function) --> same results as points 1 and
>>>> 2.
>>>> >
>>>> > Any idea what might be causing this or how I could debug this problem?
>>>> >
>>>> > Thank you very much in advance.
>>>> >
>>>> > Best Regards,
>>>> > Shehab
>>>>
>>>> > int pthread_cond_wait(pthread_cond_t *condition, pthread_mutex_t
>>>> *mutex)
>>>> > {
>>>>
>>>>
>>>>
>>>> > static int (*cond_wait)(pthread_cond_t *, pthread_mutex_t *);
>>>>
>>>> > int retval;
>>>>
>>>> >
>>>>
>>>> > if (!cond_wait) {
>>>>
>>>> > cond_wait = dlsym(RTLD_NEXT, "pthread_cond_wait");
>>>>
>>>> > if (!cond_wait) {
>>>>
>>>> > if (thread_in_trace) {
>>>>
>>>> > abort();
>>>>
>>>> > }
>>>>
>>>> > fprintf(stderr, "unable to initialize pthread wrapper
>>>> library.\n");
>>>> > return EINVAL;
>>>>
>>>> > }
>>>>
>>>> > }
>>>>
>>>> > if (thread_in_trace) {
>>>>
>>>> > return cond_wait(condition, mutex);
>>>>
>>>> > }
>>>>
>>>> >
>>>>
>>>> > thread_in_trace = 1;
>>>>
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_wait_begin, condition,
>>>> mutex,
>>>> > LTTNG_UST_CALLER_IP());
>>>>
>>>> > retval = cond_wait(condition, mutex);
>>>>
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_wait_end, condition,
>>>> mutex,
>>>> > LTTNG_UST_CALLER_IP());
>>>>
>>>> > thread_in_trace = 0;
>>>>
>>>> > return retval;
>>>>
>>>> > }
>>>>
>>>> >
>>>>
>>>> > int pthread_cond_signal(pthread_cond_t *condition)
>>>>
>>>> > {
>>>>
>>>> > static int (*cond_signal)(pthread_cond_t *);
>>>>
>>>> > int retval;
>>>>
>>>> >
>>>>
>>>> > if (!cond_signal) {
>>>>
>>>> > cond_signal = dlsym(RTLD_NEXT, "pthread_cond_signal");
>>>>
>>>> > if (!cond_signal) {
>>>>
>>>> > if (thread_in_trace) {
>>>>
>>>> > abort();
>>>>
>>>> > }
>>>>
>>>> > fprintf(stderr, "unable to initialize pthread wrapper
>>>> library.\n");
>>>> > return EINVAL;
>>>>
>>>> > }
>>>>
>>>> > }
>>>>
>>>> > if (thread_in_trace) {
>>>>
>>>> > return cond_signal(condition);
>>>>
>>>> > }
>>>>
>>>> >
>>>>
>>>> > thread_in_trace = 1;
>>>>
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_signal_begin,
>>>> condition,
>>>> > LTTNG_UST_CALLER_IP());
>>>>
>>>> > retval = cond_signal(condition);
>>>>
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_signal_end, condition,
>>>>
>>>> > LTTNG_UST_CALLER_IP());
>>>>
>>>> > thread_in_trace = 0;
>>>>
>>>> > return retval;
>>>>
>>>> > }
>>>>
>>>> >
>>>>
>>>> > int pthread_cond_broadcast(pthread_cond_t *condition)
>>>>
>>>> > {
>>>>
>>>> > static int (*cond_broadcast)(pthread_cond_t *);
>>>>
>>>> > int retval;
>>>>
>>>> >
>>>>
>>>> > if (!cond_broadcast) {
>>>>
>>>> > cond_broadcast = dlsym(RTLD_NEXT, "pthread_cond_broadcast");
>>>>
>>>> > if (!cond_broadcast) {
>>>>
>>>> > if (thread_in_trace) {
>>>>
>>>> > abort();
>>>>
>>>> > }
>>>>
>>>> > fprintf(stderr, "unable to initialize pthread wrapper
>>>> library.\n");
>>>> > return EINVAL;
>>>>
>>>> > }
>>>>
>>>> > }
>>>>
>>>> > if (thread_in_trace) {
>>>>
>>>> > return cond_broadcast(condition);
>>>>
>>>> > }
>>>>
>>>> >
>>>>
>>>> > thread_in_trace = 1;
>>>>
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_broadcast_begin,
>>>> condition,
>>>> > LTTNG_UST_CALLER_IP());
>>>>
>>>> > retval = cond_broadcast(condition);
>>>>
>>>> > tracepoint(lttng_ust_pthread, pthread_cond_broadcast_end,
>>>> condition,
>>>> > LTTNG_UST_CALLER_IP());
>>>>
>>>> > thread_in_trace = 0;
>>>>
>>>> > return retval;
>>>>
>>>> > }
>>>>
>>>> > _______________________________________________
>>>> > lttng-dev mailing list
>>>> > lttng-dev at lists.lttng.org
>>>> > https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>>
>>>>
>>>> --
>>>> Jonathan Rajotte-Julien
>>>> EfficiOS
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20190426/ff11e4e9/attachment-0001.html>
More information about the lttng-dev
mailing list