[lttng-dev] Lttng Soft Hang issue

[Jérémie Galarneau]

Would you mind testing Mathieu's patch?

I have rebased it on stable-2.6:
git clone https://github.com/jgalar/lttng-tools.git -b hang-fix

Thanks,
Jérémie


On Mon, Jul 6, 2015 at 11:23 AM, Jérémie Galarneau <
jeremie.galarneau at efficios.com> wrote:

> Hi,
>
> What do you observe when this happens? Does the session daemon become
> unresponsive?
>
> Jérémie
>
> On Wed, May 27, 2015 at 6:19 AM, Aravind HT <aravind.ht at gmail.com> wrote:
>
>> Request someone to kindly help with this. Blocked at this point, unable
>> to continue as any application crash leads to lttng not working.
>>
>> Thanks,
>> Aravind.
>>
>> On Thu, May 21, 2015 at 12:18 AM, Aravind HT <aravind.ht at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>>
>>>
>>> I have recently started trying lttng 2.6 for a few applications on my
>>> Ubuntu 12.04 and I noticed that the health check on sessiond and consumerd
>>> failed soon after starting the session.
>>>
>>>
>>> On investigating, I found that thread_manage_consumer() had exited
>>> causing an overall health check failure.
>>>
>>>
>>> Here are the sequence of steps that I found contributed to
>>> *thread_manage_consumer()* exiting.
>>>
>>>
>>> 1.       In *thread_manage_apps() : 1558 , ust_app_unregister(pollfd)* is
>>> being called. This happens when there is an error detected by *revents
>>> = LTTNG_POLL_GETEV(&events, i)*
>>>
>>> My initial guess here is that as one of my apps has crashed, producing a *LPOLLERR
>>> | LPOLLHUP | LPOLLRDHUP *to be generated for *epoll()* causing
>>> *ust_app_unregister()* to be called for that app.
>>>
>>>
>>>
>>> 2.       In *ust_app_unregister():3154 , close_metadata(registry,
>>> ua_sess->consumer)  *in which* registry->metadata_closed = 1 *is set*.*
>>>
>>>
>>>
>>> (2.a) Note:* close_metadata() *also calls* consumer_close_metadata() *which
>>> sends* LTTNG_CONSUMER_CLOSE_METADATA *and* metadata_key *to the
>>> consumerd to stop it from further dealing with the concerned app. Somehow
>>> this doesn’t seem to help*.*
>>>
>>>
>>>
>>> 3.       Next, I see that the *thread_manage_consumer():1353* for some
>>> reason has ignored the above 2.a and gets to do request/reply for that app
>>> by calling*ust_consumer_metadata_request():491 ->
>>> ust_app_push_metadat(ust_reg, socket,1) *which at line 460 checks for
>>> *registry->metadata_closed* and returns an *–EPIPE*
>>>
>>>
>>>
>>> 4.       This *–EPIPE* error cascades all the way back up to
>>> *thread_manage_consumer():1353* at which point
>>> *thread_manage_consumer()* decides to exit causing health_check() to
>>> fail.
>>>
>>>
>>>
>>>
>>>
>>> So it looks like under some scenario, an application crash could cause
>>> the lttng some problems.
>>>
>>>
>>>
>>> I think a possible fix for this scenario, is to instead of 4, send an
>>> *ERROR* message back to *consumerd()* . This could be done from
>>> *ust_consumer_metadata_request()* call. Can someone please let me know
>>> if this is correct and shed more light on the issue ?
>>>
>>>
>>>
>>>
>>>
>>> Please forgive if there are any guideline omissions for posting here
>>> from my part. This is my first post.
>>>
>>>
>>> Regards,
>>>
>>> Aravind.
>>>
>>
>>
>> _______________________________________________
>> lttng-dev mailing list
>> lttng-dev at lists.lttng.org
>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>
>>
>
>
> --
> Jérémie Galarneau
> EfficiOS Inc.
> http://www.efficios.com
>



-- 
Jérémie Galarneau
EfficiOS Inc.
http://www.efficios.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20150706/bf977fea/attachment.html>