[lttng-dev] Crash on first run of target using liblttng-ust-cyg-profile.so, but subsequent runs succeed

[Mathieu Desnoyers]

----- On Mar 1, 2016, at 11:35 AM, Michael msimons at microsoft.com wrote:

> Mathieu Desnoyers <mathieu.desnoyers <at> efficios.com> writes:
> 
>> 
>> 
>> 
>> 
>> ----- On Feb 12, 2016, at 3:21 PM, Mathieu Desnoyers
> <mathieu.desnoyers <at> efficios.com> wrote:
>> 
>> ----- On Feb 12, 2016, at 12:02 PM, Mathieu Desnoyers
> <mathieu.desnoyers <at> efficios.com> wrote:
>> 
>> 
>> ---- On Feb 11, 2016, at 9:01 AM, Sean Heelan <seanheelan <at>
> gmail.com> wrote:
>> 
>> Hi all,I am running a target within a Docker instance, and tracing
> function execution using the latest LTTng release (2.7). The commands I
> am issuing look as follows:
>> ----
>> 
>> 
>> lttng create cc_session -o bla
>> lttng enable-event --userspace lttng_ust_cyg_profile:func_entry
>> lttng start
>> LD_PRELOAD=liblttng-ust-cyg-profile.so target
>> lttng stop
>> lttng destroy
>> ----
>> 
>> When the target is executed it aborts with the following error:
>> ----
>> php: lttng-ust-comm.c:1582: lttng_ust_init: Assertion `!ret' failed.
>> 
>> ----
>> If I rerun the command it then works fine. In fact, simply doing the
> following within the Docker container demonstrates the issue:
>> ----
>> 
>> LD_PRELOAD=liblttng-ust-cyg-profile.so ls
>> LD_PRELOAD=liblttng-ust-cyg-profile.so ls
>> 
>> ----
>> The first 'ls' will fail at the same point mentioned above, while the
> second will succeed. Off the top of my head I'm struggling to come up
> with an explanation as to what impact the first execution using
> LD_PRELOAD would have on the second. Does it impact a shared lib cache
> in some way, which I'm unaware of?
>> Any assistance would be appreciated!
>> 
>> It appears that sem_timedwait() returns an unexpected error.
>> 
>> Can you add a ERROR("sem_timedwait"); just before the assert at line
> 1582 in your liblttng-ust/lttng-ust-comm.c
>> 
>> and show us the output ?
>> 
>> Also, what is the value of your LTTNG_UST_REGISTER_TIMEOUT env. var. ?
>> 
>> 
>> Also, what architecture, Linux distribution, and kernel
>> 
>> version are you using ?
>> 
>> 
>> 
>> I pushed the following fix in lttng-ust:
>> 
>> 
>> commit 5cf81d53e305c81517e48e0dc91620a916cdc0ccAuthor: Mathieu
> Desnoyers <mathieu.desnoyers <at> efficios.com>Date:   Fri Feb 12
> 15:44:10 2016 -0500    Fix: handle negative range for
> LTTNG_UST_REGISTER_TIMEOUT        We should not consider values below -1
> as valid timeout values, this is    is unexpected and could lead to
> EINVAL errors returned by sem_timedwait.        Signed-off-by: Mathieu
> Desnoyers <mathieu.desnoyers <at> efficios.com>
>> 
>> 
>> Let me know if it helps,
>> 
>> 
>> Thanks,
>> 
>> 
>> Mathieu
>> 
> 
> I am seeing the same issue that Sean originally reported within Docker
> with the latest fix Mathieu made.  Setting LTTNG_UST_REGISTER_TIMEOUT to
> any positive number cause the assert to fail. This behavior change
> appears to have been introduced with Docker 1.10.2.  I logged a Docker
> issue for this - https://github.com/docker/docker/issues/20818.

This is really weird. It looks like the futex system call would
be blacklisted for some reason, which is really unexpected. This
assert was due to an unexpected return value from sem_timedwait.

Michael Jeanson will do some testing with seccomp and lttng-ust.
We'll try to confirm which system calls lttng-ust is using when
tracing is active and inactive, so we can provide a system call
dependency list to our users.

We'll also revisit the failure modes for unavailable system calls:
we should not assert, but rather print an error to the user, and
bail out.

If you can share with us the glibc version you are using, and
the seccomp whitelist your docker environment is using, it would
be very helpful in reproducing the issue.

Thanks,

Mathieu


> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com