[ltt-dev] tracepoint layer and marker layer

[Mathieu Desnoyers]

* Jan Blunck (jblunck at suse.de) wrote:
> On Wed, Sep 17, Jiaying Zhang wrote:
> 
> > Hi Folks,
> > 
> > I have a general question about LTTng kernel code. As I understand, LTTng
> > adds an extra tracepoint layer between the traced subsystems and the
> > marker layer. The extra code added by the tracepoint layer seems to
> > introduce
> > noticeable overhead according to my measurements. I wonder whether it will
> > be more efficient and simpler if we let subsystems call the marker layer
> > directly?
> 
> I guess that was how it worked before the tracepoints. Although I don't
> fully understand why the redirection is necessary.
> 

The redirection has been asked by kernel developers for four reasons :

1 - Markers make the kernel code ugly by adding a format string
2 - Markers do not support all what is needed by in-kernel probes from
other tracers, which is mainly to export pointers to structures along
with the complete type so the structure content can be later looked into
by the probe. Markers can only export "%p" (void *). And type casting is
a bad idea.
3 - Markers are exported directly to userspace. Kernel developers don't
want to have to bother about a userspace API when they move around
kernel code. This is why it makes sense to expose the changes made by
modifying the kernel code to an in-kernel API only.
4 - Maintainability. Tracepoints offer a central repository in
include/trace/*.h, while markers are declared directly in the kernel C
files, which makes it very hard to maintain.

> > Any thought?
> 
> You could try to disable the lttng-instrumentation-*-tracepoints-probes.patch
> patches. These patches register tracepoint probes just to setup a marker. I
> guess this is where the overhead is coming from.
> 
> Anyway, do you have numbers that you can share with us? Mathieu still has the
> performance numbers with the marker patches in the tracepoints.patch. I guess
> he would be glad if he could update them.
> 

This performance degradation is not tied to the tracepoint mechanism
(it's more or less the same mechanism as the markers) but to the way I
enable all the tracepoints contained in a probe (see
kernel/kernel-trace.c for example) at module load time.

The way to fix this would be to come with a way to dynamically enable
the tracepoint which corresponds to a marker one by one.

I could just do something similar to /proc/ltt which exports the
tracepoint list and let userspace enable them, but as I said, this is
and is meant to stay an in-kernel API. Therefore, we have to think of a
way to let a given kernel probe figure out it's time to enable a given
tracepoint.

Here is what I could do : I could add an optional parameter to the
markers that would take the associated tracepoint name and the name of
the callback we are planning to associate with the tracepoint. I could
therefore use either a gcc builtin type verification or create a
static inline unused function in the macro to make sure the function
type matches the type exported by the tracepoint.

Then it's simply a matter of making the marker code call into the
tracepoint registration/unregistration functions whenever the marker is
enabled. Note that in this case, we can remove the if(unlikely()) branch
from the marker code because it's already done at the tracepoint level,
and thus becomes redundant.

Comments ?

Mathieu


> Regards,
> 	Jan
> 
> -- 
> Jan Blunck <jblunck at suse.de>
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68