[lttng-dev] Crash on first run of target using liblttng-ust-cyg-profile.so, but subsequent runs succeed

[Michael]

Mathieu Desnoyers <mathieu.desnoyers <at> efficios.com> writes:

> 
> 
> 
> 
> ----- On Feb 12, 2016, at 3:21 PM, Mathieu Desnoyers 
<mathieu.desnoyers <at> efficios.com> wrote:
> 
> ----- On Feb 12, 2016, at 12:02 PM, Mathieu Desnoyers 
<mathieu.desnoyers <at> efficios.com> wrote:
> 
> 
> ---- On Feb 11, 2016, at 9:01 AM, Sean Heelan <seanheelan <at> 
gmail.com> wrote:
> 
> Hi all,I am running a target within a Docker instance, and tracing 
function execution using the latest LTTng release (2.7). The commands I 
am issuing look as follows:
> ----
> 
> 
> lttng create cc_session -o bla
> lttng enable-event --userspace lttng_ust_cyg_profile:func_entry
> lttng start
> LD_PRELOAD=liblttng-ust-cyg-profile.so target
> lttng stop
> lttng destroy
> ----
> 
> When the target is executed it aborts with the following error:
> ----
> php: lttng-ust-comm.c:1582: lttng_ust_init: Assertion `!ret' failed.
> 
> ----
> If I rerun the command it then works fine. In fact, simply doing the 
following within the Docker container demonstrates the issue:
> ----
> 
> LD_PRELOAD=liblttng-ust-cyg-profile.so ls
> LD_PRELOAD=liblttng-ust-cyg-profile.so ls
> 
> ----
> The first 'ls' will fail at the same point mentioned above, while the 
second will succeed. Off the top of my head I'm struggling to come up 
with an explanation as to what impact the first execution using 
LD_PRELOAD would have on the second. Does it impact a shared lib cache 
in some way, which I'm unaware of?
> Any assistance would be appreciated!
> 
> It appears that sem_timedwait() returns an unexpected error.
> 
> Can you add a ERROR("sem_timedwait"); just before the assert at line 
1582 in your liblttng-ust/lttng-ust-comm.c
> 
> and show us the output ?
> 
> Also, what is the value of your LTTNG_UST_REGISTER_TIMEOUT env. var. ?
> 
> 
> Also, what architecture, Linux distribution, and kernel
> 
> version are you using ?
> 
> 
> 
> I pushed the following fix in lttng-ust:
> 
> 
> commit 5cf81d53e305c81517e48e0dc91620a916cdc0ccAuthor: Mathieu 
Desnoyers <mathieu.desnoyers <at> efficios.com>Date:   Fri Feb 12 
15:44:10 2016 -0500    Fix: handle negative range for 
LTTNG_UST_REGISTER_TIMEOUT        We should not consider values below -1 
as valid timeout values, this is    is unexpected and could lead to 
EINVAL errors returned by sem_timedwait.        Signed-off-by: Mathieu 
Desnoyers <mathieu.desnoyers <at> efficios.com>
> 
> 
> Let me know if it helps,
> 
> 
> Thanks,
> 
> 
> Mathieu
> 

I am seeing the same issue that Sean originally reported within Docker 
with the latest fix Mathieu made.  Setting LTTNG_UST_REGISTER_TIMEOUT to 
any positive number cause the assert to fail. This behavior change 
appears to have been introduced with Docker 1.10.2.  I logged a Docker 
issue for this - https://github.com/docker/docker/issues/20818.