<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hi Brian,</div><div><br data-mce-bogus="1"></div><div>You should think about those 27 bits as the "new lowest significant bits"</div><div>of the 64-bit clock.<br></div><div><br data-mce-bogus="1"></div><div>Quoting the CTF spec to understand how to deal with 27-bit overflow</div><div>between two consecutive time-stamps in the same stream (here N = 27):<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>"For a N-bit integer type referring to a clock, if the integer overflows compared to the N low order bits of the clock prior value found in the same stream, then it is assumed that one, and only one, overflow occurred. It is therefore important that events encoding time on a small number of bits happen frequently enough to detect when more than one N-bit overflow occurs."<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Those 27 bits are _not_ an offset from the previous timestamp. They update the<br data-mce-bogus="1"></div><div>current notion of time (based on all prior timestamps read for this stream) by<br data-mce-bogus="1"></div><div>setting the low 27 bits to the new value. If that value happens to be lower than the<br data-mce-bogus="1"></div><div>previous 27 low bits of the current time, this means we had one 27-bit overflow, so<br data-mce-bogus="1"></div><div>we need to increment the 28th bit by one.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>In each packet header, you will find the full 64-bit timestamps, so we always<br data-mce-bogus="1"></div><div>have a full 64-bit base from which to start.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>It is up to the tracer (lttng) to ensure that there is never more than a single 27-bit<br data-mce-bogus="1"></div><div>overflow preceding a 27-bit compact timestamp. If lttng detects that there would<br data-mce-bogus="1"></div><div>be more than a single 27-bit overflow prior to a timestamp, it will use an extended<br data-mce-bogus="1"></div><div>header which stores the full 64-bit of the new timestamp.<br data-mce-bogus="1"></div><div><br></div><div>We rely on this time-compaction scheme rather than offsetting from the prior<br data-mce-bogus="1"></div><div>64-bit timestamp because it allows us to weaken the sequential inter-dependency<br data-mce-bogus="1"></div><div>between consecutive events, which is useful for lock-free ring buffers.<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Let me know if you still have questions,<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Thanks,<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div>Mathieu<br data-mce-bogus="1"></div><div><br data-mce-bogus="1"></div><div><span id="zwchr" data-marker="__DIVIDER__">----- On Mar 16, 2016, at 6:55 PM, Brian Robbins <brianrob@microsoft.com> wrote:<br></span></div><div data-marker="__QUOTED_TEXT__"><blockquote style="border-left:2px solid #1010FF;margin-left:5px;padding-left:5px;color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;">
<style><!--
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:#7030A0;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="color:windowtext">Hello,</span></p>
<p class="MsoNormal"><span style="color:windowtext"> </span></p>
<p class="MsoNormal"><span style="color:windowtext">I’m forwarding this on-behalf of one of my colleagues who is trying to parse CTF. Any help you can provide is greatly appreciated.</span></p>
<p class="MsoNormal"><span style="color:windowtext"> </span></p>
<p class="MsoNormal"><span style="color:windowtext">Thanks.</span></p>
<p class="MsoNormal"><span style="color:windowtext">-Brian</span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="color:windowtext"> </span></a><br data-mce-bogus="1"></p>
<span style="mso-bookmark:_MailEndCompose"></span>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Lee Culver
<br>
<b>Sent:</b> Wednesday, March 16, 2016 11:06 AM<br>
<b>To:</b> Brian Robbins <brianrob@microsoft.com><br>
<b>Subject:</b> CTF Compact Header Timestamps</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Event headers in LTTng can have a 27-bit clock or a 64-bit clock. I have been able to properly map the 64-bit clock back to a real timestamp, but I’m not sure how to interpret the 27-bit clock value. I’ve dug through the CTF specification
but can’t find the relevant text. I have pasted the relevant metadata at the bottom of this email.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">For 64-bit clocks, this calculation gets us to the proper DateTime in C#:</p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> </span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> var clock = _metadata.Clocks.Single();</span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> var offset = new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc).AddSeconds(clock.Offset / clock.Frequency);</span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> </span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> long ticks = (long)(header.Timestamp * 10000000.0 / clock.Frequency);</span></p>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:Consolas;background:white;mso-highlight:white"> var time = new DateTime(offset.Ticks + ticks, DateTimeKind.Utc);</span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">However, for the ‘compact’ variant (see below), we end up with a timestamp that’s far lower than the frequency, such as ‘0x55363b9’ and I’m simply not sure how to interpret this. It does NOT seem to be an offset from the previous event
since if I treat it as such we get events that are out of order. One such example is the following:</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> Event #11 - Extended header timestamp (64-bit): 0x00098a754555ea28</p>
<p class="MsoNormal" style="text-indent:.5in">Event #12 - Compact header timestamp (27-bit): 0x04ecc38c</p>
<p class="MsoNormal" style="text-indent:.5in">Event #13 - Extended header timestamp (64-bit): 0x00098a75369cc5e4</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">As you can see, Event[11].Timestamp + Event[12].Timestamp > Event[13].Timestamp. Which means it cannot be a simple “offset from previous event”, since we would be getting events out of order.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Here is the relevant metadata:</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">clock {</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> name = "monotonic";</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> uuid = "9fb57333-ada2-4c73-a7ce-334f46437c74";</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> description = "Monotonic Clock";</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> freq = 1000000000; /* Frequency, in Hz */</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> /* clock value offset from Epoch is: offset * (1/freq) */</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> offset = 1451946230228988436;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">};</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> </span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> </span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">typealias integer {</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> size = 27; align = 1; signed = false;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> map = clock.monotonic.value;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">} := uint27_clock_monotonic_t;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> </span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">typealias integer {</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> size = 64; align = 8; signed = false;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> map = clock.monotonic.value;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">} := uint64_clock_monotonic_t;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> </span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">struct event_header_compact {</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> enum : uint5_t { compact = 0 ... 30, extended = 31 } id;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> variant <id> {</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> struct {</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> uint27_clock_monotonic_t timestamp;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> } compact;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> struct {</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> uint32_t id;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> uint64_clock_monotonic_t timestamp;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> } extended;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext"> } v;</span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.5pt;font-family:Consolas;color:windowtext">} align(8);</span><span style="color:windowtext"></span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">How should we be treating these <span style="font-size:9.5pt;font-family:Consolas">
uint27_clock_monotonic_t</span> values?</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thanks!</p>
<p class="MsoNormal">-Lee Culver</p>
<p class="MsoNormal">CLR Reliability and Performance</p>
<p class="MsoNormal"> </p>
</div>
<br>_______________________________________________<br>lttng-dev mailing list<br>lttng-dev@lists.lttng.org<br>https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev<br></blockquote></div><div><br></div><div data-marker="__SIG_POST__">-- <br></div><div>Mathieu Desnoyers<br>EfficiOS Inc.<br>http://www.efficios.com</div></div></body></html>