<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div><span id="zwchr" data-marker="__DIVIDER__">----- On May 30, 2016, at 9:16 AM, Vijay Anand <vjanandr85@gmail.com> wrote:<br></span></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_default" style="font-size:small"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">Hi Mathieu, Jonathan,</span></div><div class="gmail_default" style="font-size:small"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif"><br></span></div><div class="gmail_default" style="font-size:small"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">I have written python scripts to simulate our requirements.</span></div><div class="gmail_default"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif"><a href="https://github.com/vjanandr/sampleP/tree/master/lttng" target="_blank">https://github.com/vjanandr/sampleP/tree/master/lttng</a><br><br></span></div><div class="gmail_default"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">Please refer to the README file for further details about the scripts.</span></div><div class="gmail_default"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif"><br></span></div><div class="gmail_default"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">Mathieu, I understand your recommendation to use per UID buffers instead of per PID buffers.</span></div><div class="gmail_default"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">But I dont seem to understand your suggestion to use filters...</span></div><div class="gmail_default"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif"><br></span></div><div class="gmail_default"><div style="color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;font-size:16px">>>>></div><div style="color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;font-size:16px">Why ? You can always collect trace data into buffers shared across processes (per-uid buffers)</div><div style="color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;font-size:16px">and filter after the fact.</div><div><<<<<</div><div><br>Did you mean to say write all traces to a file but filter them while reading based on maybe process ID ? </div></div></div></blockquote><div>Yes, this is what I mean,<br></div><div><br data-mce-bogus="1"></div><div>Thanks,<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Mathieu<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_extra"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif"><br clear="all"></span><div><div><div dir="ltr"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">Regards,</span><div><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">Vijay</span></div></div></div></div><br><div class="gmail_quote">On Fri, May 27, 2016 at 1:16 AM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"><span><br><br><span>----- On May 26, 2016, at 8:23 AM, Vijay Anand <<a href="mailto:vjanandr85@gmail.com" target="_blank">vjanandr85@gmail.com</a>> wrote:<br></span></span><div><span><blockquote style="border-left-width:2px;border-left-style:solid;border-left-color:rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><span style="color:rgb(0,0,0)">Could anyone please let us know how do we go about this ?</span></div><div class="gmail_extra"><span style="color:rgb(0,0,0)"><br clear="all"></span><div><div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:verdana,sans-serif">Regards,</span><div><span style="color:rgb(0,0,0);font-family:verdana,sans-serif">Vijay</span></div></div></div></div><br><div class="gmail_quote">On Tue, May 24, 2016 at 11:31 AM, Vijay Anand <span dir="ltr"><<a href="mailto:vjanandr85@gmail.com" target="_blank">vjanandr85@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div style="font-family:verdana,sans-serif;font-size:small"><span style="color:rgb(0,0,0)">Hello Folks,</span></div><div style="font-family:verdana,sans-serif;font-size:small"><span style="color:rgb(0,0,0)"><br></span></div><div style="font-family:verdana,sans-serif;font-size:small"><span style="color:rgb(0,0,0)">We have been evaluating LTTNG for use in our production systems for user space tracing. We have evaluated most of the features supported by LTTNG and very much see a value add LTTNG brings into our debugging infrastructure.</span></div><div><ul><li><span style="color:rgb(0,0,0)"><span style="font-family:verdana,sans-serif">Our current requirement is to trace Userspace programs running on linux. </span><br></span></li><li><span style="color:rgb(0,0,0)"><span style="font-family:verdana,sans-serif">Each of the linux processes define their own tracepoint providers.</span></span></li></ul></div></div></blockquote></div></div></div></blockquote><br></span><div>Sounds like a good design.<br></div><span><br><blockquote style="border-left-width:2px;border-left-style:solid;border-left-color:rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><ul><li><span style="color:rgb(0,0,0)"><br></span></li><li><span style="color:rgb(0,0,0)"><span style="font-family:verdana,sans-serif">We would like to trace event histories of each process independently.</span></span></li></ul></div></div></blockquote></div></div></div></blockquote><br></span><div>Why ? You can always collect trace data into buffers shared across processes (per-uid buffers)</div><div>and filter after the fact.</div><span><br><blockquote style="border-left-width:2px;border-left-style:solid;border-left-color:rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><ul><li><span style="color:rgb(0,0,0)"><br></span></li><li><span style="color:rgb(0,0,0)"><span style="font-family:verdana,sans-serif">We could potentially have 1000s of such processes running simultaneously.</span></span></li></ul></div></div></blockquote></div></div></div></blockquote><br></span><div>Especially with that many processes, having per-process buffers will degrade your cache<br></div><div>locality.<br></div><span><br><blockquote style="border-left-width:2px;border-left-style:solid;border-left-color:rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><ul><li><span style="color:rgb(0,0,0)"><br></span></li><li><span style="font-family:verdana,sans-serif"><span style="color:rgb(0,0,0)">We concluded on using a session/channel to trace one tracepoint provider corresponding to a unique process. </span></span></li><ul><li><span style="font-family:verdana,sans-serif"><span style="color:rgb(0,0,0)">But I understand we could also create one system wide session and use channels to trace each of the providers. Either of the approaches seems to work for us.</span></span></li></ul></ul></div></div></blockquote></div></div></div></blockquote></span><div>Both approach will kill you cache locality with that many process. I don't recommend either<br></div><div>of the two approaches you refer to above. You might want to consider sharing your buffers</div><div>across processes.</div><span><br><blockquote style="border-left-width:2px;border-left-style:solid;border-left-color:rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><ul><li><span style="font-family:verdana,sans-serif"><span style="color:rgb(0,0,0)">But upon evaluating I see that we could create only 25 active process-sessions and not traces from all the processes are logged.</span></span></li></ul></div></div></blockquote></div></div></div></blockquote></span><div>We will need much more details on your tracing configuration setup (exact list of commands you<br></div><div>issue to create your tracing sessions).<br></div><br><div>Also please detail what you mean by "<span style="font-family:verdana,sans-serif"><span style="color:rgb(0,0,0)">not traces from all the processes are logged".</span></span><br></div><div><span style="font-family:verdana,sans-serif"><span style="color:rgb(0,0,0)">What commands do you issue, what is the exact setup, and what is the output<br></span></span></div><div><span style="font-family:verdana,sans-serif"><span style="color:rgb(0,0,0)">you observe (exact listing) which leads you to reach this conclusion.<br></span></span></div><br><div>Thanks,<br></div><br><div>Mathieu<br></div><span><br><blockquote style="border-left-width:2px;border-left-style:solid;border-left-color:rgb(16,16,255);margin-left:5px;padding-left:5px;color:rgb(0,0,0);font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><ul><ul><li><span style="font-family:verdana,sans-serif"><span style="color:rgb(0,0,0)">Please note I have tried increasing the buffer size and the number of buffers. This doesn't help.</span></span></li><li><span style="font-family:verdana,sans-serif"><span style="color:rgb(0,0,0)">Each of the process trace 52 events at and interval of 1 second each.</span></span></li></ul><li><span style="color:rgb(0,0,0);font-family:verdana,sans-serif">I have evaluated this with lttng in session,live and snapshot modes and I have not been getting favourable.</span></li></ul><span style="color:rgb(0,0,0)"><span style="font-family:verdana,sans-serif">Could you folks share the scale numbers that LTTNG supports, especially when it comes to tracing user space programs ?</span><br></span></div><div style="font-family:verdana,sans-serif;font-size:small"><span style="color:rgb(0,0,0)"><br></span></div><div><div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:verdana,sans-serif">Regards,</span><div><span style="color:rgb(0,0,0);font-family:verdana,sans-serif">Vijay<div style="font-family:verdana,sans-serif;font-size:small;display:inline"></div></span></div></div></div></div></div></blockquote></div></div></div><br></blockquote></span></div><span><span style="color: #888888;" data-mce-style="color: #888888;" color="#888888"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br data-mce-bogus="1"></div></span></span></div></div></blockquote></div></div></div><br></blockquote></div><div><br></div><div data-marker="__SIG_POST__">-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</div></div></body></html>