[lttng-dev] Questions about LTTNG_UST_REGISTER_TIMEOUT

[David OShea]

Hi Martin, all,

> -----Original Message-----
> From: Martin Ünsal [mailto:martinunsal at gmail.com]
> Sent: Saturday, 27 July 2013 4:59 AM
> To: lttng-dev at lists.lttng.org
> Subject: [lttng-dev] Questions about LTTNG_UST_REGISTER_TIMEOUT
> 
> Hi all,
> 
> I am a new lttng-ust user. I would like to leave tracepoints compiled
> into production code. I have verified that inactive tracepoints have
> extremely low overhead, but I am concerned about startup time and
> LTTNG_UST_REGISTER_TIMEOUT. Our typical use case is that we are not
> tracing and there is no lttng-sessiond, we enable tracing only in
> development or QA.
> 
> The specific questions:
> 
> 1) What scenarios might the timeout come into play? I have tested with
> and without lttng-sessiond running and I have seen no more than 10ms
> overhead during app startup caused by lttng. That's no problem, but if
> there is some unforeseen scenario where I introduce 3s overhead I am in
> a world of hurt!

I believe I have seen lttng-sessiond end up in some kind of bad state that caused my traced application hit the timeout - I'm fairly certain that there weren't a large number of other processes talking to lttng-sessiond at the time.  I'm not sure how it got into this state; I know a process with UST trace providers had crashed, perhaps that affected lttng-sessiond somehow.  I can reproduce similar behaviour by sending lttng-sessiond the STOP signal, so perhaps lttng-sessiond deadlocked or something.

I found https://bugs.lttng.org/issues/545, my interpretation of which is that once lttng-sessiond is in the above state, you could end up with your applications completely freezing on startup.  The example in that bug report was spawning a large number of processes simultaneously, but I simply used:

for i in $(seq 1 100); do LTTNG_UST_REGISTER_TIMEOUT=500 LD_PRELOAD=<one of my dynamic trace providers> sleep 1; done

and saw the 62nd invocation of 'sleep' freeze forever.

So whilst I'll be applying the fix for the above bug, I will definitely be setting LTTNG_UST_REGISTER_TIMEOUT=0 too just in case there are any more bugs like this lurking :)

Incidentally I'm still using LTTng 2.1.x.

Regards,
David

----------------------------------------------------------------------
The information contained in this transmission may be confidential. Any disclosure, copying, or further distribution of confidential information is not permitted unless such privilege is explicitly granted in writing by Quantum. Quantum reserves the right to have electronic communications, including email and attachments, sent across its networks filtered through anti virus and spam software programs and retain such messages in order to comply with applicable data security and retention requirements. Quantum is not responsible for the proper and complete transmission of the substance of this communication or for any delay in its receipt.