[lttng-dev] CTF Compact Header Timestamps

[Brian Robbins]

Thanks very much Mathieu.  Appreciate the help.

-Brian

From: Mathieu Desnoyers [mailto:mathieu.desnoyers at efficios.com]
Sent: Wednesday, March 16, 2016 5:16 PM
To: Brian Robbins <brianrob at microsoft.com>
Cc: lttng-dev <lttng-dev at lists.lttng.org>; Lee Culver <leculver at microsoft.com>
Subject: Re: [lttng-dev] CTF Compact Header Timestamps

Hi Brian,

You should think about those 27 bits as the "new lowest significant bits"
of the 64-bit clock.

Quoting the CTF spec to understand how to deal with 27-bit overflow
between two consecutive time-stamps in the same stream (here N = 27):

"For a N-bit integer type referring to a clock, if the integer overflows compared to the N low order bits of the clock prior value found in the same stream, then it is assumed that one, and only one, overflow occurred. It is therefore important that events encoding time on a small number of bits happen frequently enough to detect when more than one N-bit overflow occurs."

Those 27 bits are _not_ an offset from the previous timestamp. They update the
current notion of time (based on all prior timestamps read for this stream) by
setting the low 27 bits to the new value. If that value happens to be lower than the
previous 27 low bits of the current time, this means we had one 27-bit overflow, so
we need to increment the 28th bit by one.

In each packet header, you will find the full 64-bit timestamps, so we always
have a full 64-bit base from which to start.

It is up to the tracer (lttng) to ensure that there is never more than a single 27-bit
overflow preceding a 27-bit compact timestamp. If lttng detects that there would
be more than a single 27-bit overflow prior to a timestamp, it will use an extended
header which stores the full 64-bit of the new timestamp.

We rely on this time-compaction scheme rather than offsetting from the prior
64-bit timestamp because it allows us to weaken the sequential inter-dependency
between consecutive events, which is useful for lock-free ring buffers.

Let me know if you still have questions,

Thanks,

Mathieu

----- On Mar 16, 2016, at 6:55 PM, Brian Robbins <brianrob at microsoft.com<mailto:brianrob at microsoft.com>> wrote:

Hello,

I’m forwarding this on-behalf of one of my colleagues who is trying to parse CTF.  Any help you can provide is greatly appreciated.

Thanks.
-Brian

From: Lee Culver
Sent: Wednesday, March 16, 2016 11:06 AM
To: Brian Robbins <brianrob at microsoft.com<mailto:brianrob at microsoft.com>>
Subject: CTF Compact Header Timestamps

Event headers in LTTng can have a 27-bit clock or a 64-bit clock.  I have been able to properly map the 64-bit clock back to a real timestamp, but I’m not sure how to interpret the 27-bit clock value.  I’ve dug through the CTF specification but can’t find the relevant text.  I have pasted the relevant metadata at the bottom of this email.

For 64-bit clocks, this calculation gets us to the proper DateTime in C#:

            var clock = _metadata.Clocks.Single();
            var offset = new DateTime(1970, 1, 1, 0, 0, 0, 0, DateTimeKind.Utc).AddSeconds(clock.Offset / clock.Frequency);

            long ticks = (long)(header.Timestamp * 10000000.0 / clock.Frequency);
            var time = new DateTime(offset.Ticks + ticks, DateTimeKind.Utc);

However, for the ‘compact’ variant (see below), we end up with a timestamp that’s far lower than the frequency, such as ‘0x55363b9’ and I’m simply not sure how to interpret this.  It does NOT seem to be an offset from the previous event since if I treat it as such we get events that are out of order.  One such example is the following:

                Event #11 - Extended header timestamp (64-bit): 0x00098a754555ea28
Event #12 - Compact header timestamp (27-bit): 0x04ecc38c
Event #13 - Extended header timestamp (64-bit): 0x00098a75369cc5e4

As you can see, Event[11].Timestamp + Event[12].Timestamp > Event[13].Timestamp.  Which means it cannot be a simple “offset from previous event”, since we would be getting events out of order.

Here is the relevant metadata:

clock {
       name = "monotonic";
       uuid = "9fb57333-ada2-4c73-a7ce-334f46437c74";
       description = "Monotonic Clock";
       freq = 1000000000; /* Frequency, in Hz */
       /* clock value offset from Epoch is: offset * (1/freq) */
       offset = 1451946230228988436;
};


typealias integer {
       size = 27; align = 1; signed = false;
       map = clock.monotonic.value;
} := uint27_clock_monotonic_t;

typealias integer {
       size = 64; align = 8; signed = false;
       map = clock.monotonic.value;
} := uint64_clock_monotonic_t;

struct event_header_compact {
       enum : uint5_t { compact = 0 ... 30, extended = 31 } id;
       variant <id> {
              struct {
                     uint27_clock_monotonic_t timestamp;
              } compact;
              struct {
                     uint32_t id;
                     uint64_clock_monotonic_t timestamp;
              } extended;
       } v;
} align(8);

How should we be treating these uint27_clock_monotonic_t values?

Thanks!
-Lee Culver
CLR Reliability and Performance


_______________________________________________
lttng-dev mailing list
lttng-dev at lists.lttng.org<mailto:lttng-dev at lists.lttng.org>
https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev

--
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.lttng.org/pipermail/lttng-dev/attachments/20160317/3b916209/attachment-0001.html>