<div dir="ltr">Do you have a specific changeset in mind when you say you applied the changes mentioned in that thread?<div><br></div><div>Do you simply revert the following commit?</div><div><a href="https://github.com/lttng/lttng-tools/commit/1dc0526df43f2b5f86ef451e4c0331445346b15f">https://github.com/lttng/lttng-tools/commit/1dc0526df43f2b5f86ef451e4c0331445346b15f</a></div><div><br></div><div>Thanks,</div><div>Jérémie<br><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jul 11, 2015 at 3:26 AM, Aravind HT <span dir="ltr"><<a href="mailto:aravind.ht@gmail.com" target="_blank">aravind.ht@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">To try this fix, I also needed the changes talked about in <a href="http://lists.lttng.org/pipermail/lttng-dev/2015-July/024689.html" target="_blank">http://lists.lttng.org/pipermail/lttng-dev/2015-July/024689.html</a> , otherwise, I would see relayd coring.<div>Once I took them I could see that the soft hang issue was no longer reproducible. </div><div><br></div><div>Thanks for helping.</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 8, 2015 at 9:49 AM, Aravind HT <span dir="ltr"><<a href="mailto:aravind.ht@gmail.com" target="_blank">aravind.ht@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Sure, I will try the fix and update.</div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 6, 2015 at 10:12 PM, Jérémie Galarneau <span dir="ltr"><<a href="mailto:jeremie.galarneau@efficios.com" target="_blank">jeremie.galarneau@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Would you mind testing Mathieu's patch?<div><br></div><div>I have rebased it on stable-2.6:</div><div><div>git clone <a href="https://github.com/jgalar/lttng-tools.git" target="_blank">https://github.com/jgalar/lttng-tools.git</a> -b hang-fix<br></div></div><div><br></div><div><div>Thanks,</div><div>Jérémie</div></div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 6, 2015 at 11:23 AM, Jérémie Galarneau <span dir="ltr"><<a href="mailto:jeremie.galarneau@efficios.com" target="_blank">jeremie.galarneau@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>What do you observe when this happens? Does the session daemon become unresponsive?</div><div><br></div><div>Jérémie</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Wed, May 27, 2015 at 6:19 AM, Aravind HT <span dir="ltr"><<a href="mailto:aravind.ht@gmail.com" target="_blank">aravind.ht@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><div dir="ltr">Request someone to kindly help with this. Blocked at this point, unable to continue as any application crash leads to lttng not working.<div><br></div><div>Thanks,</div><div>Aravind.</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 21, 2015 at 12:18 AM, Aravind HT <span dir="ltr"><<a href="mailto:aravind.ht@gmail.com" target="_blank">aravind.ht@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><p class="MsoNormal" style="font-size:12.8000001907349px">Hi,</p><p class="MsoNormal" style="font-size:12.8000001907349px"> </p><p class="MsoNormal" style="font-size:12.8000001907349px">I have recently started trying lttng 2.6 for a few applications on my Ubuntu 12.04 and I noticed that the health check on sessiond and consumerd failed soon after starting the session.</p><p class="MsoNormal" style="font-size:12.8000001907349px"><br></p><p class="MsoNormal" style="font-size:12.8000001907349px">On investigating, I found that thread_manage_consumer() had exited causing an overall health check failure.</p><p class="MsoNormal" style="font-size:12.8000001907349px"><br></p><p class="MsoNormal" style="font-size:12.8000001907349px">Here are the sequence of steps that I found contributed to <b>thread_manage_consumer()</b> exiting.</p><p class="MsoNormal" style="font-size:12.8000001907349px"></p><p class="MsoNormal" style="font-size:12.8000001907349px"><br></p><p style="font-size:12.8000001907349px">1.<span style="font-stretch:normal;font-size:7pt;font-family:'Times New Roman'"> </span>In <b>thread_manage_apps() : 1558 , ust_app_unregister(pollfd)</b> is being called. This happens when there is an error detected by <b>revents = LTTNG_POLL_GETEV(&events, i)</b></p><p style="font-size:12.8000001907349px">My initial guess here is that as one of my apps has crashed, producing a <b>LPOLLERR | LPOLLHUP | LPOLLRDHUP </b>to be generated for <b>epoll()</b> causing <b>ust_app_unregister()</b> to be called for that app.</p><p style="font-size:12.8000001907349px"> </p><p style="font-size:12.8000001907349px">2.<span style="font-stretch:normal;font-size:7pt;font-family:'Times New Roman'"> </span>In <b>ust_app_unregister():3154 , close_metadata(registry, ua_sess->consumer) </b>in which<b> registry->metadata_closed = 1 </b>is set<b>.</b></p><p style="font-size:12.8000001907349px;text-indent:0.5in"> </p><p style="font-size:12.8000001907349px;text-indent:0.5in">(2.a) Note:<b> close_metadata() </b>also calls<b> consumer_close_metadata() </b>which sends<b> LTTNG_CONSUMER_CLOSE_METADATA </b>and<b> metadata_key </b>to the consumerd to stop it from further dealing with the concerned app. Somehow this doesn’t seem to help<b>.</b></p><p style="font-size:12.8000001907349px"><b> </b></p><p style="font-size:12.8000001907349px">3.<span style="font-stretch:normal;font-size:7pt;font-family:'Times New Roman'"> </span>Next, I see that the <b>thread_manage_consumer():1353</b> for some reason has ignored the above 2.a and gets to do request/reply for that app by calling<b>ust_consumer_metadata_request():491 -> ust_app_push_metadat(ust_reg, socket,1) </b>which at line 460 checks for <b>registry->metadata_closed</b> and returns an <b>–EPIPE</b></p><p class="MsoNormal" style="font-size:12.8000001907349px"> </p><p style="font-size:12.8000001907349px">4.<span style="font-stretch:normal;font-size:7pt;font-family:'Times New Roman'"> </span>This <b>–EPIPE</b> error cascades all the way back up to <b>thread_manage_consumer():1353</b> at which point <b>thread_manage_consumer()</b> decides to exit causing health_check() to fail.</p><p style="font-size:12.8000001907349px"> </p><p class="MsoNormal" style="font-size:12.8000001907349px"> </p><p class="MsoNormal" style="font-size:12.8000001907349px">So it looks like under some scenario, an application crash could cause the lttng some problems.</p><p class="MsoNormal" style="font-size:12.8000001907349px"> </p><p class="MsoNormal" style="font-size:12.8000001907349px">I think a possible fix for this scenario, is to instead of 4, send an <b>ERROR</b> message back to <b>consumerd()</b> . This could be done from <b>ust_consumer_metadata_request()</b> call. Can someone please let me know if this is correct and shed more light on the issue ?</p><p class="MsoNormal" style="font-size:12.8000001907349px"> </p><p class="MsoNormal" style="font-size:12.8000001907349px"> </p><p class="MsoNormal" style="font-size:12.8000001907349px">Please forgive if there are any guideline omissions for posting here from my part. This is my first post.</p><p class="MsoNormal" style="font-size:12.8000001907349px"><br></p><p class="MsoNormal" style="font-size:12.8000001907349px">Regards,</p><p class="MsoNormal" style="font-size:12.8000001907349px">Aravind.</p></div>
</blockquote></div><br></div>
</div></div><br></div></div>_______________________________________________<br>
lttng-dev mailing list<br>
<a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br>
<a href="http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" rel="noreferrer" target="_blank">http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div>Jérémie Galarneau<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Jérémie Galarneau<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a></div>
</div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Jérémie Galarneau<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a></div>
</div></div></div>