[lttng-dev] Using lttng-ust with xenomai

[Norbert Lange]

Am Fr., 22. Nov. 2019 um 20:00 Uhr schrieb Mathieu Desnoyers
<mathieu.desnoyers at efficios.com>:
>
> ----- On Nov 22, 2019, at 12:44 PM, Norbert Lange nolange79 at gmail.com wrote:
>
> > Am Fr., 22. Nov. 2019 um 16:52 Uhr schrieb Jan Kiszka <jan.kiszka at siemens.com>:
> >>
> >> On 22.11.19 16:42, Mathieu Desnoyers wrote:
>
> [...]
>
> >
> >
> >> >
> >> > That's indeed a good point. I suspect membarrier may not send any IPI
> >> > to Xenomai threads (that would have to be confirmed). I suspect the
> >> > latency introduced by this IPI would be unwanted.
> >>
> >> Is an "IPI" a POSIX signal here? Or are real IPI that delivers an
> >> interrupt to Linux on another CPU? The latter would still be possible,
> >> but it would be delayed until all Xenomai threads on that core eventual
> >> took a break (which should happen a couple of times per second under
> >> normal conditions - 100% RT load is an illegal application state).
> >
> > Not POSIX, some inter-thread interrupts. point is the syscall waits
> > for the set of
> > registered *running* Linux threads.
>
> Just a small clarification: the PRIVATE membarrier command does not *wait*
> for other threads, but it rather ensures that all other running threads
> have had IPIs that issue memory barriers before it returns.

Ok, normal linux IRQs have to wait till Xenomai gives the cores back,
hence the waiting.

>
> >> >
> >> > Another thing to make sure is to have a glibc and Linux kernel which perform
> >> > clock_gettime() as vDSO for the monotonic clock, because you don't want a
> >> > system call there. If that does not work for you, you can alternatively
> >> > implement your own lttng-ust and lttng-modules clock plugin .so/.ko to override
> >> > the clock used by lttng, and for instance use TSC directly. See for instance
> >> > the lttng-ust(3) LTTNG_UST_CLOCK_PLUGIN environment variable.
> >>
> >> clock_gettime & Co for a Xenomai application is syscall-free as well.
> >
> > Yes, and that gave me a deadlock already, if a library us not compiled
> > for Xenomai,
> > it will either use the syscall (and you detect that immediatly) or it
> > will work most of the time,
> > and lock up once in a while if a Linux thread took the "writer lock"
> > of the VDSO structures
> > and your high priority xenomai thread is busy waiting infinitely.
> >
> > Only sane approach would be to use either the xenomai function directly,
> > or recreate the function (rdtsc + interpolation on x86).
> > Either compiling/patching lttng for Cobalt (which I really would not
> > want to do) or using a
> > clock plugin.
> > If the later is supposed to be minimal, then that would mean I would
> > have to get the
> > interpolation factors cobalt uses (without bringing in libcobalt).
> >
> > Btw. the Xenomai and Linux monotonic clocks arent synchronised at all
> > AFAIK, so timestamps will
> > be different to the rest of Linux.
> > On my last plattform I did some tracing using internal stamp and
> > regulary wrote a
> > block with internal and external timestamps so those could be
> > converted "offline".
> > Anything similar with lttng or tools handling the traces?
>
> Can a Xenomai thread issue clock_gettime(CLOCK_MONOTONIC) ?

Yes it can, if the calls goes through the VDSO, then it mostly works.
And once in a while deadlocks the system if a Xenomai thread waits for a
spinlock that the Linux kernel owns and doesnt give back as said thread will
not let the Linux Kernel run (as described above).

>
> AFAIK we don't have tooling to do what you describe out of the box,
> but it could probably be implemented as a babeltrace 2 filter plugin.

There are alot ways to do that, I hoped for some standardized way.
regards, Norbert