[lttng-dev] Issue with dynamic load and UST

[Mathieu Desnoyers]

* Suchakra Sharma (suchakrapani.sharma at polymtl.ca) wrote:
> Hi,
> 
> Zifei recently did a demo of dynamic tracing using ust with dyninst [1] The traces are being generated 
> and I did some experiments by adding some more 'tracepoint types' too [2]. This seemed to be a promising 
> approach but then there's an error as follows when I start tracing on a fresh rebooted machine (if a 
> normal session of static userspace tracing is done once, the problem doesn't occur):
> 
> libust[3429/3429]: Error: Timed out waiting for lttng-sessiond (in lttng_ust_init() at lttng-ust-comm.c:1072)
> 
> This occurred even after separating the tracepoints [3] Backtrace is at [4]. I was able to reproduce it on my 
> two machines with UST 2.1.2 and dyninst 8.1 (Fedora 18) 
> 
> Is it possible to circumvent this problem somehow? (it seems to be an inherent issue with dlopen w/ sessiond 
> but I may be wrong) 
> 
> 
> [1] https://gist.github.com/anonymous/5599765 
> [2] https://gist.github.com/tuxology/5602019
> [3] https://gist.github.com/anonymous/5605050
> [4] https://gist.github.com/5kg/5604931

This appears to be a deadlock between the ust_lock and an internal lock
in glibc's dynamic linker.

See this commit:

commit eddd8d5d1a04887d1979417f1aca6c2f109bfe50
Author: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
Date:   Thu Mar 1 12:00:04 2012 -0500

    Fix: fix deadlock with dlopen() lttng-ust
    
    Pre-"fault" the libuuid TLS outside of the ust lock to ensure that we
    never deadlock with the linker mutex in libc. This mutex is held both
    for TLS fixup of dlopen'd libs, and across execution of all
    constructors. We therefore have a reverse use of this mutex wrt ust
    lock, that we fix by forcing the TLS fixup to happen very early in the
    lttng-ust constructor.
    
    [fixed merge from dev branch]
    
    Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>

Good luck!

Mathieu


> 
> 
> --
> Suchakra
> 
> 
> 
> 
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com