[lttng-dev] Tracing/Profiling boot

Fri Feb 26 17:19:46 EST 2016

Hi, 

The timestamp source use by the lttng kernel and ust tracers is the 
monotonic clock. See clock_gettime(2) "CLOCK_MONOTONIC". 

However, both tracers sample both CLOCK_REALTIME and 
CLOCK_MONOTONIC at trace start to find the relationship 
(offset) of the monotonic clock with Epoch. 

If you start your tracing early at boot, before NTP has updated 
CLOCK_REALTIME, you might be depending on an unreliable 
offset from Epoch (e.g. system thinks it is in 1970, or your BIOS 
clock is a few hours wrong). 

One way to work around this is to use the new "lttng metadata regenerate" 
command after NTP has updated your realtime clock. So you basically 
start tracing with a "wrong" offset from Epoch, and then the lttng metadata 
regenerate command will fixup your metadata with the correct offset. 

We will merge the metadata regenerate command into lttng-tools master 
branch next week. 

Meanwhile, you can of course use the trace even though your offset from 
Epoch is not right, but it can make correlation with system logs (e.g. syslogd) 
a bit more cumbersome. 

Thanks, 

Mathieu 

----- On Feb 26, 2016, at 5:03 PM, Martin Townsend <mtownsend1973 at gmail.com> wrote: 

> Hi Mathieu,

> Seeing the relationship between userspace and kernel would definitely be an
> advantage for this problem. As I will be looking into a lot of timing
> information and out of interest where does lttng get it's timestamps from?

> Once I have updated everything else to work happily with systemd I'll start to
> look into this. I'll start by seeing if I can bring up the lttng-sessiond
> user-space process within my custom init process and if this proves
> unsuccessful I will look into a custom lttng-module that's built into the
> kernel.

> Cheers,
> Martin.

> On Fri, Feb 26, 2016 at 8:39 PM, Mathieu Desnoyers <
> mathieu.desnoyers at efficios.com > wrote:

>> ----- On Feb 26, 2016, at 1:01 PM, Mathieu Desnoyers <
>> mathieu.desnoyers at efficios.com > wrote:

>>> Hi Martin,

>>> I would say that the main advantages of using LTTng in this case
>>> would be on the tooling side, with availability of Trace Compass
>>> and LTTng Analyses projects to navigate in the resulting CTF
>>> traces, and get high-level overview of the various metrics that
>>> could slow down your system.

>>> LTTng would also allow you to correlate your kernel trace with
>>> a user-space trace gathered with lttng-ust, which gives you
>>> deeper insight into your applications.

>> One more benefit of using lttng as builtin tracer rather than
>> ftrace is that LTTng gathers much more information about the
>> system call input/output parameters than ftrace. This can be
>> useful to get a better understanding of what userspace is
>> doing with the kernel.

>> Thanks,

>> Mathieu

>>> Thanks,

>>> Mathieu

>>> ----- On Feb 26, 2016, at 3:41 AM, Martin Townsend < mtownsend1973 at gmail.com >
>>> wrote:

>>>> Hi Mathieu,
>>>> Thanks for the info, I didn't know about ftrace, I will take a look at this.

>>>> Out of interest would there be any benefits from using a built-in lttng module
>>>> over ftrace?

>>>> Cheers,
>>>> Martin.

>>>> On Thu, Feb 25, 2016 at 9:18 PM, Mathieu Desnoyers <
>>>> mathieu.desnoyers at efficios.com > wrote:

>>>>> Hi Martin,

>>>>> The main limitation LTTng currently has for early boot tracing is that
>>>>> you need to first spawn a lttng-sessiond user-space process, and setup
>>>>> tracing, before you can actually do any tracing. As long as you can
>>>>> fit within those constraints, you should be OK.

>>>>> If you really want to trace earlier than that, you might have to create
>>>>> a dedicated early-boot tracing module that would setup tracing
>>>>> buffers into a "dummy" session which exists only within lttng-modules,
>>>>> and then allow sessiond to later hook on those buffers when user-space
>>>>> is ready. Nothing exists for this at the moment. Note that since
>>>>> lttng-modules master (upcoming 2.8), you can now build lttng-modules
>>>>> into your kernel image, this might be useful for you. See the "kernel
>>>>> built-in support" section in
>>>>> https://github.com/lttng/lttng-modules/blob/master/README.md

>>>>> Since LTTng 2.0, we have left early boot tracing to other tools, such
>>>>> as Ftrace, which target kernel developers use-cases, and focused
>>>>> more on tracing of the system in its execution phases which are more
>>>>> relevant to application developers.

>>>>> If you want to go ahead and create a LTTng modules module that
>>>>> allow early boot tracing, I'd be happy to provide ideas and review.

>>>>> Thanks,

>>>>> Mathieu

>>>>> ----- On Feb 25, 2016, at 3:56 PM, Martin Townsend < mtownsend1973 at gmail.com >
>>>>> wrote:

>>>>>> Hi,

>>>>>> This is a bit of a long shot but does LTTng allow you trace boot?

>>>>>> I'm seeing a weird problem where if I boot with systemd-bootchart if boots
>>>>>> faster than just using systemd as the init process. I created my own init
>>>>>> process based on systemd-bootchart and worked out it was down to the fact it
>>>>>> called nanosleep, so I now have my own init process which hands over to systemd
>>>>>> and creates a child that nanosleeps for the boot duration. I would really like
>>>>>> to trace/profile the scheduler and hrtimers understand what's happening and try
>>>>>> and get a proper fix :) Even if it means a bit of hacking kernel/LTTng, I would
>>>>>> be willing to do this.

>>>>>> Many Thanks, Martin.

>>>>>> _______________________________________________
>>>>>> lttng-dev mailing list
>>>>>> lttng-dev at lists.lttng.org
>>>>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>>>>> --
>>>>> Mathieu Desnoyers
>>>>> EfficiOS Inc.
>>>>> http://www.efficios.com

>>> --
>>> Mathieu Desnoyers
>>> EfficiOS Inc.
>>> http://www.efficios.com

>>> _______________________________________________
>>> lttng-dev mailing list
>>> lttng-dev at lists.lttng.org
>>> http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

>> --
>> Mathieu Desnoyers
>> EfficiOS Inc.
>> http://www.efficios.com

-- 
Mathieu Desnoyers 
EfficiOS Inc. 
http://www.efficios.com 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20160226/4c38e3f3/attachment.html>