<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div><br></div><div><br></div><span id="zwchr" data-marker="__DIVIDER__">----- On May 26, 2016, at 8:23 AM, Vijay Anand <vjanandr85@gmail.com> wrote:<br></span><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">Could anyone please let us know how do we go about this ?</span></div><div class="gmail_extra"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><br clear="all"></span><div><div class="gmail_signature"><div dir="ltr"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">Regards,</span><div><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">Vijay</span></div></div></div></div><br><div class="gmail_quote">On Tue, May 24, 2016 at 11:31 AM, Vijay Anand <span dir="ltr"><<a href="mailto:vjanandr85@gmail.com" target="_blank">vjanandr85@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div style="font-family:verdana,sans-serif;font-size:small"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">Hello Folks,</span></div><div style="font-family:verdana,sans-serif;font-size:small"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><br></span></div><div style="font-family:verdana,sans-serif;font-size:small"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">We have been evaluating LTTNG for use in our production systems for user space tracing. We have evaluated most of the features supported by LTTNG and very much see a value add LTTNG brings into our debugging infrastructure.</span></div><div><ul><li><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><span style="font-family:verdana,sans-serif">Our current requirement is to trace Userspace programs running on linux. </span><br></span></li><li><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><span style="font-family:verdana,sans-serif">Each of the linux processes define their own tracepoint providers.</span></span></li></ul></div></div></blockquote></div></div></div></blockquote><div><br></div><div>Sounds like a good design.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><ul><li><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><br></span></li><li><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><span style="font-family:verdana,sans-serif">We would like to trace event histories of each process independently.</span></span></li></ul></div></div></blockquote></div></div></div></blockquote><div><br></div><div>Why ? You can always collect trace data into buffers shared across processes (per-uid buffers)</div><div>and filter after the fact.</div><div><br data-mce-bogus="1"></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><ul><li><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><br></span></li><li><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><span style="font-family:verdana,sans-serif">We could potentially have 1000s of such processes running simultaneously.</span></span></li></ul></div></div></blockquote></div></div></div></blockquote><div><br></div><div>Especially with that many processes, having per-process buffers will degrade your cache<br data-mce-bogus="1"></div><div>locality.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><ul><li><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><br></span></li><li><span style="font-family:verdana,sans-serif"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">We concluded on using a session/channel to trace one tracepoint provider corresponding to a unique process. </span></span></li><ul><li><span style="font-family:verdana,sans-serif"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">But I understand we could also create one system wide session and use channels to trace each of the providers. Either of the approaches seems to work for us.</span></span></li></ul></ul></div></div></blockquote></div></div></div></blockquote><div>Both approach will kill you cache locality with that many process. I don't recommend either<br></div><div>of the two approaches you refer to above. You might want to consider sharing your buffers</div><div>across processes.</div><div><br data-mce-bogus="1"></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><ul><li><span style="font-family:verdana,sans-serif"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">But upon evaluating I see that we could create only 25 active process-sessions and not traces from all the processes are logged.</span></span></li></ul></div></div></blockquote></div></div></div></blockquote><div>We will need much more details on your tracing configuration setup (exact list of commands you<br></div><div>issue to create your tracing sessions).<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Also please detail what you mean by "<span style="font-family: verdana,sans-serif;" data-mce-style="font-family: verdana,sans-serif;"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">not traces from all the processes are logged".</span></span><br data-mce-bogus="1"></div><div><span style="font-family: verdana,sans-serif;" data-mce-style="font-family: verdana,sans-serif;"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">What commands do you issue, what is the exact setup, and what is the output<br></span></span></div><div><span style="font-family: verdana,sans-serif;" data-mce-style="font-family: verdana,sans-serif;"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">you observe (exact listing) which leads you to reach this conclusion.<br data-mce-bogus="1"></span></span></div><div><br data-mce-bogus="1"></div><div>Thanks,<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Mathieu<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><ul><ul><li><span style="font-family:verdana,sans-serif"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">Please note I have tried increasing the buffer size and the number of buffers. This doesn't help.</span></span></li><li><span style="font-family:verdana,sans-serif"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000">Each of the process trace 52 events at and interval of 1 second each.</span></span></li></ul><li><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">I have evaluated this with lttng in session,live and snapshot modes and I have not been getting favourable.</span></li></ul><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><span style="font-family:verdana,sans-serif">Could you folks share the scale numbers that LTTNG supports, especially when it comes to tracing user space programs ?</span><br></span></div><div style="font-family:verdana,sans-serif;font-size:small"><span style="color: #000000;" data-mce-style="color: #000000;" color="#000000"><br></span></div><div><div><div dir="ltr"><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">Regards,</span><div><span style="color: #000000; font-family: verdana,sans-serif;" data-mce-style="color: #000000; font-family: verdana,sans-serif;" color="#000000" face="verdana, sans-serif">Vijay<div style="font-family:verdana,sans-serif;font-size:small;display:inline"></div></span></div></div></div></div></div></blockquote></div></div></div><br></blockquote></div><br><div data-marker="__SIG_POST__">-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</div></div></body></html>