<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:#7030A0;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:windowtext">Thanks very much Mathieu. Appreciate the help.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext">-Brian<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="color:windowtext"><o:p> </o:p></span></a></p>
<span style="mso-bookmark:_MailEndCompose"></span>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Mathieu Desnoyers [mailto:mathieu.desnoyers@efficios.com]
<br>
<b>Sent:</b> Wednesday, March 16, 2016 5:16 PM<br>
<b>To:</b> Brian Robbins <brianrob@microsoft.com><br>
<b>Cc:</b> lttng-dev <lttng-dev@lists.lttng.org>; Lee Culver <leculver@microsoft.com><br>
<b>Subject:</b> Re: [lttng-dev] CTF Compact Header Timestamps<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Hi Brian,</span><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">You should think about those 27 bits as the "new lowest significant bits"<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">of the 64-bit clock.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Quoting the CTF spec to understand how to deal with 27-bit overflow<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">between two consecutive time-stamps in the same stream (here N = 27):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">"For a N-bit integer type referring to a clock, if the integer overflows compared to the N low order bits of the clock prior value found in the same stream, then it is assumed that
one, and only one, overflow occurred. It is therefore important that events encoding time on a small number of bits happen frequently enough to detect when more than one N-bit overflow occurs."<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Those 27 bits are _not_ an offset from the previous timestamp. They update the<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">current notion of time (based on all prior timestamps read for this stream) by<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">setting the low 27 bits to the new value. If that value happens to be lower than the<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">previous 27 low bits of the current time, this means we had one 27-bit overflow, so<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">we need to increment the 28th bit by one.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">In each packet header, you will find the full 64-bit timestamps, so we always<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">have a full 64-bit base from which to start.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">It is up to the tracer (lttng) to ensure that there is never more than a single 27-bit<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">overflow preceding a 27-bit compact timestamp. If lttng detects that there would<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">be more than a single 27-bit overflow prior to a timestamp, it will use an extended<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">header which stores the full 64-bit of the new timestamp.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">We rely on this time-compaction scheme rather than offsetting from the prior<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">64-bit timestamp because it allows us to weaken the sequential inter-dependency<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">between consecutive events, which is useful for lock-free ring buffers.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Let me know if you still have questions,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Thanks,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">Mathieu<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Arial",sans-serif;color:black">----- On Mar 16, 2016, at 6:55 PM, Brian Robbins <<a href="mailto:brianrob@microsoft.com">brianrob@microsoft.com</a>> wrote:<br>
<br>
<o:p></o:p></span></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #1010FF 1.5pt;padding:0in 0in 0in 4.0pt;margin-left:3.75pt;margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:windowtext">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">I’m forwarding this on-behalf of one of my colleagues who is trying to parse CTF. Any help you can provide is greatly appreciated.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Thanks.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext">-Brian</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:windowtext"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Lee Culver
<br>
<b>Sent:</b> Wednesday, March 16, 2016 11:06 AM<br>
<b>To:</b> Brian Robbins <<a href="mailto:brianrob@microsoft.com">brianrob@microsoft.com</a>><br>
<b>Subject:</b> CTF Compact Header Timestamps</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Event headers in LTTng can have a 27-bit clock or a 64-bit clock. I have been able to properly map the 64-bit clock back to a real timestamp, but I’m not sure how to interpret the 27-bit clock value. I’ve dug through the CTF specification
but can’t find the relevant text. I have pasted the relevant metadata at the bottom of this email.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">For 64-bit clocks, this calculation gets us to the proper DateTime in C#:<o:p></o:p></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> </span><o:p></o:p></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> var clock = _metadata.Clocks.Single();</span><o:p></o:p></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> var offset = new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc).AddSeconds(clock.Offset / clock.Frequency);</span><o:p></o:p></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> </span><o:p></o:p></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> long ticks = (long)(header.Timestamp * 10000000.0 / clock.Frequency);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> var time = new DateTime(offset.Ticks + ticks, DateTimeKind.Utc);</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">However, for the ‘compact’ variant (see below), we end up with a timestamp that’s far lower than the frequency, such as ‘0x55363b9’ and I’m simply not sure how to interpret this. It does NOT seem to be an offset from the previous event
since if I treat it as such we get events that are out of order. One such example is the following:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> Event #11 - Extended header timestamp (64-bit): 0x00098a754555ea28<o:p></o:p></p>
<p class="MsoNormal" style="text-indent:.5in">Event #12 - Compact header timestamp (27-bit): 0x04ecc38c<o:p></o:p></p>
<p class="MsoNormal" style="text-indent:.5in">Event #13 - Extended header timestamp (64-bit): 0x00098a75369cc5e4<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">As you can see, Event[11].Timestamp + Event[12].Timestamp > Event[13].Timestamp. Which means it cannot be a simple “offset from previous event”, since we would be getting events out of order.<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Here is the relevant metadata:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">clock {</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> name = "monotonic";</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> uuid = "9fb57333-ada2-4c73-a7ce-334f46437c74";</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> description = "Monotonic Clock";</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> freq = 1000000000; /* Frequency, in Hz */</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> /* clock value offset from Epoch is: offset * (1/freq) */</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> offset = 1451946230228988436;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">};</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">typealias integer {</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> size = 27; align = 1; signed = false;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> map = clock.monotonic.value;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">} := uint27_clock_monotonic_t;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">typealias integer {</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> size = 64; align = 8; signed = false;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> map = clock.monotonic.value;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">} := uint64_clock_monotonic_t;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">struct event_header_compact {</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> enum : uint5_t { compact = 0 ... 30, extended = 31 } id;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> variant <id> {</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> struct {</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> uint27_clock_monotonic_t timestamp;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> } compact;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> struct {</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> uint32_t id;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> uint64_clock_monotonic_t timestamp;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> } extended;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> } v;</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">} align(8);</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">How should we be treating these <span style="font-size:9.5pt;font-family:Consolas">
uint27_clock_monotonic_t</span> values?<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal">-Lee Culver<o:p></o:p></p>
<p class="MsoNormal">CLR Reliability and Performance<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Helvetica",sans-serif;color:black"><br>
_______________________________________________<br>
lttng-dev mailing list<br>
<a href="mailto:lttng-dev@lists.lttng.org">lttng-dev@lists.lttng.org</a><br>
<a href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev">https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><o:p></o:p></span></p>
</blockquote>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black">--
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Arial",sans-serif;color:black">Mathieu Desnoyers<br>
EfficiOS Inc.<br>
<a href="http://www.efficios.com">http://www.efficios.com</a><o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>