[lttng-dev] Lttng Soft Hang issue

[Aravind HT]

Sure, I will try the fix and update.

On Mon, Jul 6, 2015 at 10:12 PM, Jérémie Galarneau <
jeremie.galarneau at efficios.com> wrote:

> Would you mind testing Mathieu's patch?
>
> I have rebased it on stable-2.6:
> git clone https://github.com/jgalar/lttng-tools.git -b hang-fix
>
> Thanks,
> Jérémie
>
>
> On Mon, Jul 6, 2015 at 11:23 AM, Jérémie Galarneau <
> jeremie.galarneau at efficios.com> wrote:
>
>> Hi,
>>
>> What do you observe when this happens? Does the session daemon become
>> unresponsive?
>>
>> Jérémie
>>
>> On Wed, May 27, 2015 at 6:19 AM, Aravind HT <aravind.ht at gmail.com> wrote:
>>
>>> Request someone to kindly help with this. Blocked at this point, unable
>>> to continue as any application crash leads to lttng not working.
>>>
>>> Thanks,
>>> Aravind.
>>>
>>> On Thu, May 21, 2015 at 12:18 AM, Aravind HT <aravind.ht at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>>
>>>>
>>>> I have recently started trying lttng 2.6 for a few applications on my
>>>> Ubuntu 12.04 and I noticed that the health check on sessiond and consumerd
>>>> failed soon after starting the session.
>>>>
>>>>
>>>> On investigating, I found that thread_manage_consumer() had exited
>>>> causing an overall health check failure.
>>>>
>>>>
>>>> Here are the sequence of steps that I found contributed to
>>>> *thread_manage_consumer()* exiting.
>>>>
>>>>
>>>> 1.       In *thread_manage_apps() : 1558 , ust_app_unregister(pollfd)* is
>>>> being called. This happens when there is an error detected by *revents
>>>> = LTTNG_POLL_GETEV(&events, i)*
>>>>
>>>> My initial guess here is that as one of my apps has crashed, producing
>>>> a *LPOLLERR | LPOLLHUP | LPOLLRDHUP *to be generated for *epoll()*
>>>>  causing *ust_app_unregister()* to be called for that app.
>>>>
>>>>
>>>>
>>>> 2.       In *ust_app_unregister():3154 , close_metadata(registry,
>>>> ua_sess->consumer)  *in which* registry->metadata_closed = 1 *is set*.*
>>>>
>>>>
>>>>
>>>> (2.a) Note:* close_metadata() *also calls* consumer_close_metadata() *which
>>>> sends* LTTNG_CONSUMER_CLOSE_METADATA *and* metadata_key *to the
>>>> consumerd to stop it from further dealing with the concerned app. Somehow
>>>> this doesn’t seem to help*.*
>>>>
>>>>
>>>>
>>>> 3.       Next, I see that the *thread_manage_consumer():1353* for some
>>>> reason has ignored the above 2.a and gets to do request/reply for that app
>>>> by calling*ust_consumer_metadata_request():491 ->
>>>> ust_app_push_metadat(ust_reg, socket,1) *which at line 460 checks for
>>>> *registry->metadata_closed* and returns an *–EPIPE*
>>>>
>>>>
>>>>
>>>> 4.       This *–EPIPE* error cascades all the way back up to
>>>> *thread_manage_consumer():1353* at which point
>>>> *thread_manage_consumer()* decides to exit causing health_check() to
>>>> fail.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> So it looks like under some scenario, an application crash could cause
>>>> the lttng some problems.
>>>>
>>>>
>>>>
>>>> I think a possible fix for this scenario, is to instead of 4, send an
>>>> *ERROR* message back to *consumerd()* . This could be done from
>>>> *ust_consumer_metadata_request()* call. Can someone please let me know
>>>> if this is correct and shed more light on the issue ?
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Please forgive if there are any guideline omissions for posting here
>>>> from my part. This is my first post.
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Aravind.
>>>>
>>>
>>>
>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
>>>
>>>
>>
>>
>> --
>> Jérémie Galarneau
>> EfficiOS Inc.
>> http://www.efficios.com
>>
>
>
>
> --
> Jérémie Galarneau
> EfficiOS Inc.
> http://www.efficios.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20150708/6c489c58/attachment.html>