<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hi,</div><div><br data-mce-bogus="1"></div><div>The timestamp source use by the lttng kernel and ust tracers is the</div><div>monotonic clock. See clock_gettime(2) "CLOCK_MONOTONIC".<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>However, both tracers sample both CLOCK_REALTIME and<br data-mce-bogus="1"></div><div>CLOCK_MONOTONIC at trace start to find the relationship</div><div>(offset) of the monotonic clock with Epoch.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>If you start your tracing early at boot, before NTP has updated<br data-mce-bogus="1"></div><div>CLOCK_REALTIME, you might be depending on an unreliable<br data-mce-bogus="1"></div><div>offset from Epoch (e.g. system thinks it is in 1970, or your BIOS<br data-mce-bogus="1"></div><div>clock is a few hours wrong).<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>One way to work around this is to use the new "lttng metadata regenerate"<br data-mce-bogus="1"></div><div>command after NTP has updated your realtime clock. So you basically<br></div><div>start tracing with a "wrong" offset from Epoch, and then the lttng metadata<br data-mce-bogus="1"></div><div>regenerate command will fixup your metadata with the correct offset.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>We will merge the metadata regenerate command into lttng-tools master</div><div>branch next week.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Meanwhile, you can of course use the trace even though your offset from<br></div><div>Epoch is not right, but it can make correlation with system logs (e.g. syslogd)</div><div>a bit more cumbersome.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Thanks,<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Mathieu<br data-mce-bogus="1"></div><div><br></div><div><br></div><div><span id="zwchr" data-marker="__DIVIDER__">----- On Feb 26, 2016, at 5:03 PM, Martin Townsend <mtownsend1973@gmail.com> wrote:<br></span></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><div dir="ltr"><div><div>Hi Mathieu,<br><br></div>Seeing the relationship between userspace and kernel would definitely be an advantage for this problem. As I will be looking into a lot of timing information and out of interest where does lttng get it's timestamps from?<br><br>Once I have updated everything else to work happily with systemd I'll start to look into this. I'll start by seeing if I can bring up the lttng-sessiond user-space process within my custom init process and if this proves unsuccessful I will look into a custom lttng-module that's built into the kernel.<br><br></div><div>Cheers,<br></div><div>Martin.<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Feb 26, 2016 at 8:39 PM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><span class=""><br><br><span>----- On Feb 26, 2016, at 1:01 PM, Mathieu Desnoyers <<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>> wrote:<br></span></span><div><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div>Hi Martin,<br></div><br><span class=""><div>I would say that the main advantages of using LTTng in this case</div><div>would be on the tooling side, with availability of Trace Compass</div><div>and LTTng Analyses projects to navigate in the resulting CTF</div><div>traces, and get high-level overview of the various metrics that</div><div>could slow down your system.<br></div><br><div>LTTng would also allow you to correlate your kernel trace with<br></div><div>a user-space trace gathered with lttng-ust, which gives you<br></div><div>deeper insight into your applications.</div></span></div></blockquote><br><div>One more benefit of using lttng as builtin tracer rather than<br></div><div>ftrace is that LTTng gathers much more information about the<br></div><div>system call input/output parameters than ftrace. This can be<br></div><div>useful to get a better understanding of what userspace is<br></div><div>doing with the kernel.<br></div><br><div>Thanks,<br></div><br><div>Mathieu<br></div><div><div class="h5"><br><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><br><br><div>Thanks,<br></div><br><div>Mathieu<br></div><br><span>----- On Feb 26, 2016, at 3:41 AM, Martin Townsend <<a href="mailto:mtownsend1973@gmail.com" target="_blank">mtownsend1973@gmail.com</a>> wrote:<br></span><div><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div dir="ltr">Hi Mathieu,<br><div>Thanks for the info, I didn't know about ftrace, I will take a look at this. </div><br><div>Out of interest would there be any benefits from using a built-in lttng module over ftrace?</div><br><div>Cheers,<br>Martin.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 25, 2016 at 9:18 PM, Mathieu Desnoyers <span dir="ltr"><<a href="mailto:mathieu.desnoyers@efficios.com" target="_blank">mathieu.desnoyers@efficios.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div>Hi Martin,<br></div><br><div>The main limitation LTTng currently has for early boot tracing is that<br></div><div>you need to first spawn a lttng-sessiond user-space process, and setup<br></div><div>tracing, before you can actually do any tracing. As long as you can<br></div><div>fit within those constraints, you should be OK.<br></div><br><div>If you really want to trace earlier than that, you might have to create<br></div><div>a dedicated early-boot tracing module that would setup tracing<br></div><div>buffers into a "dummy" session which exists only within lttng-modules,</div><div>and then allow sessiond to later hook on those buffers when user-space</div><div>is ready. Nothing exists for this at the moment. Note that since</div><div>lttng-modules master (upcoming 2.8), you can now build lttng-modules</div><div>into your kernel image, this might be useful for you. See the "kernel</div><div>built-in support" section in</div><div><a href="https://github.com/lttng/lttng-modules/blob/master/README.md" target="_blank">https://github.com/lttng/lttng-modules/blob/master/README.md</a><br></div><br><div>Since LTTng 2.0, we have left early boot tracing to other tools, such</div><div>as Ftrace, which target kernel developers use-cases, and focused</div><div>more on tracing of the system in its execution phases which are more</div><div>relevant to application developers.<br></div><br><div>If you want to go ahead and create a LTTng modules module that<br></div><div>allow early boot tracing, I'd be happy to provide ideas and review.<br></div><br><div>Thanks,<br></div><br><div>Mathieu<br></div><div><div><br><span>----- On Feb 25, 2016, at 3:56 PM, Martin Townsend <<a href="mailto:mtownsend1973@gmail.com" target="_blank">mtownsend1973@gmail.com</a>> wrote:<br></span></div></div><div><blockquote style="border-left:2px solid #1010ff;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt"><div><div><div dir="ltr"><div><div>Hi,<br><br></div>This is a bit of a long shot but does LTTng allow you trace boot?<br><br>I'm seeing a weird problem where if I boot with systemd-bootchart if boots faster than just using systemd as the init process. I created my own init process based on systemd-bootchart and worked out it was down to the fact it called nanosleep, so I now have my own init process which hands over to systemd and creates a child that nanosleeps for the boot duration. I would really like to trace/profile the scheduler and hrtimers understand what's happening and try and get a proper fix :) Even if it means a bit of hacking kernel/LTTng, I would be willing to do this.<br><br></div>Many Thanks, Martin.<br></div><br></div></div>_______________________________________________<br>lttng-dev mailing list<br><a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br><a href="http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" target="_blank">http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><span><span style="color:#888888"><br></span></span></blockquote></div><span><span style="color:#888888"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br></div></span></span></div></div></blockquote></div></div><br></blockquote></div><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br data-mce-bogus="1"></div></div><br>_______________________________________________<br>lttng-dev mailing list<br><a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br><a href="http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" target="_blank">http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br></blockquote></div></div></div><div><div class="h5"><br><div>-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br><a href="http://www.efficios.com" target="_blank">http://www.efficios.com</a><br data-mce-bogus="1"></div></div></div></div></div></blockquote></div></div><br></blockquote></div><div><br></div><div data-marker="__SIG_POST__">-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</div></div></body></html>