<div dir="ltr">Yes, I have both 32 bit as well as 64 bit apps running.</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 6, 2015 at 8:23 PM, Jérémie Galarneau <span dir="ltr"><<a href="mailto:jeremie.galarneau@efficios.com" target="_blank">jeremie.galarneau@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Are you running both 32 and 64-bits apps at once?<span class="HOEnZb"><font color="#888888"><div><br></div><div>Jérémie</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 6, 2015 at 12:10 AM, Aravind HT <span dir="ltr"><<a href="mailto:aravind.ht@gmail.com" target="_blank">aravind.ht@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">This is how I am creating the session and using <div><p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black"><br></span></p><p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black">lttng-relayd -o
/var/tmp/lttng-traces -d</span></p>
<p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black">lttng-sessiond
--consumerd32-path /usr/lib/lttng/libexec/lttng-consumerd --consumerd32-libdir
/usr/lib/ --consumerd64-path /usr/lib64/lttng/libexec/lttng-consumerd
--consumerd64-libdir /usr/lib64/ -b --no-kernel</span></p>
<p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black"> </span></p>
<p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black">lttng --no-sessiond
create mysession --live 2000000 -U net://<a href="http://128.0.0.4" target="_blank">128.0.0.4</a></span></p>
<p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black">lttng enable-channel mychannel
--userspace --buffers-pid -s mysession --tracefile-size 2000000
--tracefile-count 2</span></p>
<p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black">lttng enable-event --channel
mychannel --userspace --all </span><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(35,35,35)">--exclude 'ust_libc:*'</span><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black"> -s
mysession</span></p>
<p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black">lttng start mysession</span></p>
<p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black">lttng list</span></p><p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black"><br></span></p><p style="margin:0in 0in 0.0001pt"><font color="#000000" face="Helvetica, sans-serif"><span style="font-size:12px">At present, I am not trying to enable new channels. The above is pretty much what I do.</span></font></p><p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black"><br></span></p><p style="margin:0in 0in 0.0001pt"><span style="font-size:9pt;font-family:Helvetica,sans-serif;color:black"><br></span></p></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jul 4, 2015 at 1:26 AM, Jérémie Galarneau <span dir="ltr"><<a href="mailto:jeremie.galarneau@efficios.com" target="_blank">jeremie.galarneau@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Can you share a bit more information about the session configurations in your use case?<div>Are you creating and destroying new sessions, tracing in per-pid or uid mode, enabling new channels, etc.</div><div><br></div><div>Also, is this in "live" mode or are you just streaming to the relay daemon? (lttng create --set-url net://somewhere)</div><span><font color="#888888"><div><br></div></font></span><div><span><font color="#888888">Jérémie</font></span><div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 3, 2015 at 3:30 PM, Jérémie Galarneau <span dir="ltr"><<a href="mailto:jeremie.galarneau@efficios.com" target="_blank">jeremie.galarneau@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Fri, Jul 3, 2015 at 1:48 PM, Aravind HT <span dir="ltr"><<a href="mailto:aravind.ht@gmail.com" target="_blank">aravind.ht@gmail.com</a>></span> wrote:<br></span><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><span><div>Thanks for replying.</div><div><br></div><div>I have a daemon which i'm using for some CLI application, this forks whenever invoked but never exec()'s and am using lttng for tracing this and a few other apps.</div><div>I have a script that I run to stress test this daemon; this I think means lttng is also being stressed.</div><div>As soon as I run this script, after a few streams are created, I sometimes see a soft hang as mentioned in my previous thread <a href="http://lists.lttng.org/pipermail/lttng-dev/2015-May/024533.html" style="font-size:12.8000001907349px" target="_blank">http://lists.lttng.org/pipermail/lttng-dev/2015-May/024533.html</a></div><div>or </div><div>I see lttng-relayd coring.</div><div><br></div><div>In short, an app that calls quite a bit of fork and exits which is being traced with other normal apps that run forever should re-create the problem.</div><div><br></div><div>To run valgrind, use the -DVALGRIND when building, your can grep the code for what this does.</div><div>Kill the valgrind proc with -TERM to produce the logs.</div><div><br></div></span></div></blockquote><div><br></div><div>Doh, of course! :-)</div><span><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div></div><div>This is the command and the options that I use to run valgrind</div><div><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">valgrind --tool=memcheck --leak-check=full -v
--log-file=/tmp/test --trace-children=yes /usr/bin/lttng-relayd -o
/var/log/lttng-traces -d</span><span style="color:black"></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><br></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><br></span></p><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">I have tried the fix detailed in the checkin </span><a href="http://git.lttng.org/?p=lttng-tools.git;a=commitdiff;h=cd2ef1ef1d54ced9e4d0d03b865bb7fc6a905f80" style="font-family:Calibri,sans-serif;font-size:11pt" target="_blank">http://git.lttng.org/?p=lttng-tools.git;a=commitdiff;h=cd2ef1ef1d54ced9e4d0d03b865bb7fc6a905f80</a></p><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">and it seems to work for me. This however only solves the relayd core, but I still see the "soft hang" happening.</span></p></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><br></span></p><p class="MsoNormal"><font color="#1f497d" face="Calibri, sans-serif"><span style="font-size:14.6666669845581px">Can some one please tell me how the above fix is missing from the source ?</span></font></p></div></div></blockquote><div><br></div></span><div><div>Somehow it appears this was reverted by this commit:</div><div><a href="https://github.com/lttng/lttng-tools/commit/1dc0526df43f2b5f86ef451e4c0331445346b15f" target="_blank">https://github.com/lttng/lttng-tools/commit/1dc0526df43f2b5f86ef451e4c0331445346b15f</a><br></div><div><br></div><div>I'm trying to reproduce the problem here.</div><span><font color="#888888"><div><br></div><div>Jérémie</div></font></span></div><div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><br></span></p><p class="MsoNormal">Regards,<span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><br></span></p><p class="MsoNormal">Aravind.</p><p class="MsoNormal"><br></p></div><div><br></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 3, 2015 at 8:36 PM, Jérémie Galarneau <span dir="ltr"><<a href="mailto:jeremie.galarneau@efficios.com" target="_blank">jeremie.galarneau@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>Thanks for the detailed report. Can you provide a sequence of steps that leads to this problem?</div><div>Also, did you modify lttng-relayd at all to be able to run valgrind? I seem to remember valgrind not supporting clone().</div><div><br></div><div>Thanks,</div><div>Jérémie</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div>On Thu, Jul 2, 2015 at 12:42 PM, Aravind HT <span dir="ltr"><<a href="mailto:aravind.ht@gmail.com" target="_blank">aravind.ht@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><div dir="ltr">Possible Solution :<div><br></div><div>The following changes seem to be missing from the actual code</div><div><p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><a href="http://git.lttng.org/?p=lttng-tools.git;a=commitdiff;h=cd2ef1ef1d54ced9e4d0d03b865bb7fc6a905f80" target="_blank">http://git.lttng.org/?p=lttng-tools.git;a=commitdiff;h=cd2ef1ef1d54ced9e4d0d03b865bb7fc6a905f80</a></span></p><p class="MsoNormal"><br></p><p class="MsoNormal">only the connection.h changes have been absorbed, while the main.c changes left out.</p></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 29, 2015 at 11:51 PM, Aravind HT <span dir="ltr"><<a href="mailto:aravind.ht@gmail.com" target="_blank">aravind.ht@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I ran valgrind on lttng-relayd for a soft hang issue that I had reported earlier.</div><div><a href="http://lists.lttng.org/pipermail/lttng-dev/2015-May/024533.html" target="_blank">http://lists.lttng.org/pipermail/lttng-dev/2015-May/024533.html</a><br></div><div><br></div><div>Below are the list of errors that I found.</div><div><br></div><div><div>==793== Thread 4:</div><div>==793== Invalid write of size 8</div><div>==793== at 0x406038: cds_list_add (list.h:53)</div><div>==793== by 0x409442: queue_stream (main.c:1175)</div><div>==793== by 0x409AB7: relay_add_stream (main.c:1274)</div><div>==793== by 0x40C47A: relay_process_control (main.c:2099)</div><div>==793== by 0x40DBB1: relay_thread_worker (main.c:2574)</div><div>==793== by 0x5BDAFE2: start_thread (pthread_create.c:312)</div><div>==793== by 0x5ED8AFC: clone (clone.S:111)</div><div>==793== Address 0x6e09040 is 64 bytes inside a block of size 312 free'd</div><div>==793== at 0x4C2898C: free (in /usr/lib64/valgrind/vgpreload_memcheck-amd64-linux.so)</div><div>==793== by 0x418EF8: rcu_destroy_stream (stream.c:33)</div><div>==793== by 0x5079C94: ??? (in /usr/lib64/liburcu.so.2.0.0)</div><div>==793== by 0x5BDAFE2: start_thread (pthread_create.c:312)</div><div>==793== by 0x5ED8AFC: clone (clone.S:111)</div><div>==793==</div></div><div><br></div><div>Are there any known bugs/fixes for these ?</div><div><br></div><div><br></div><div>Regards,</div><div>Aravind.</div><div><br></div></div>
</blockquote></div><br></div>
</div></div><br></div></div>_______________________________________________<br>
lttng-dev mailing list<br>
<a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br>
<a href="http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" rel="noreferrer" target="_blank">http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br>
<br></blockquote></div><span><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div>Jérémie Galarneau<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a></div>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div></div></div><div><div><br><br clear="all"><div><br></div>-- <br><div>Jérémie Galarneau<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a></div>
</div></div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Jérémie Galarneau<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a></div>
</div></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Jérémie Galarneau<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a></div>
</div>
</div></div></blockquote></div><br></div>