[lttng-dev] UST Hanging at Tracepoint

[Mathieu Desnoyers]

* Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote:
> * Mathieu Desnoyers (mathieu.desnoyers at efficios.com) wrote:
> > * Burton, Michael (mburton at ciena.com) wrote:
> > > Mathieu,
> > > 
> > > I think there is a deadlock scenario in UST, which has been causing my problem.
> > 
> > Good catch !
> > 
> > > 
> > > sessiond is started as root:
> > >     - creates global sockets ONLY
> > >     - DOES NOT CREATE shm in $HOME/lttng-ust-wait-<uid>
> > > 
> > > application linked against ust is run as root:
> > >   - in lttng_ust_init constructor
> > >   - ust_listener_thread  (local_apps)
> > >     - fails to connect to local_apps in $HOME/.lttng (as expected)
> > >     - prev_connect_failed=1
> > >     - ust_unlock()
> > >     - restart
> > >     - wait_for_sessiond()
> > >  -->  - ust_lock()
> > >  |      - get_map_shm()
> > >  |        - get_wait_shm()
> > > DEADLOCK    - shm_open() FAILS (not created by sessiond when run by root)
> > >  |            - fork() (trying to create shared memory itself)
> > >  |              - ust_before_fork()
> > >  ------------>    - ust_lock()
> > > 
> > > 
> > > You should be able to create this with an empty main, with no
> > > tracepoints.  As long as sessiond is started as root so
> > > $HOME/lttng-ust-wait-<uid> is not created.  You can also make the
> > > lttng-ust constructor (lttng_ust_init) wait forever and then you'll be
> > > able to see the deadlock in gdb without even leaving the
> > > lttng_ust_init constructor.
> > 
> > Ah, I see. This deadlock is caused by the interaction between
> > liblttng-ust-libc-wrapper and liblttng-ust (the fork override is
> > performed by liblttng-ust-libc-wrapper). I'll have to look into this
> > issue further. More on that soon.
> 
> Actually, I meant that it comes from interaction between
> liblttng-ust-fork and liblttng-ust (sorry about the confusion above).
> I'll see if I can provide a direct access to the fork() symbol to
> liblttng-ust, using a weak symbol so it is defined even when
> liblttng-ust-fork is not linked.
> 
> More on that soon...

Fixed by master commit:

commit e8508a495280a1396a0319a157409a7ea61ffc5a
Author: Mathieu Desnoyers <mathieu.desnoyers at efficios.com>
Date:   Wed Jun 13 04:26:34 2012 -0400

    Fix: liblttng-ust-fork deadlock

and merged into stable-2.0.

Thanks for pointing out this important bug.

Mathieu

-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com