[ltt-dev] tracepoint layer and marker layer

[Mathieu Desnoyers]

* Jiaying Zhang (jiayingz at google.com) wrote:
> Hi Mathieu,
> 
> Thanks for the detailed explanation! See my comments below.
> 
> On Thu, Sep 18, 2008 at 9:00 AM, Mathieu Desnoyers <
> compudj at krystal.dyndns.org> wrote:
> 
> > * Jan Blunck (jblunck at suse.de) wrote:
> > > On Wed, Sep 17, Jiaying Zhang wrote:
> > >
> > > > Hi Folks,
> > > >
> > > > I have a general question about LTTng kernel code. As I understand,
> > LTTng
> > > > adds an extra tracepoint layer between the traced subsystems and the
> > > > marker layer. The extra code added by the tracepoint layer seems to
> > > > introduce
> > > > noticeable overhead according to my measurements. I wonder whether it
> > will
> > > > be more efficient and simpler if we let subsystems call the marker
> > layer
> > > > directly?
> > >
> > > I guess that was how it worked before the tracepoints. Although I don't
> > > fully understand why the redirection is necessary.
> > >
> >
> > The redirection has been asked by kernel developers for four reasons :
> >
> > 1 - Markers make the kernel code ugly by adding a format string
> > 2 - Markers do not support all what is needed by in-kernel probes from
> > other tracers, which is mainly to export pointers to structures along
> > with the complete type so the structure content can be later looked into
> > by the probe. Markers can only export "%p" (void *). And type casting is
> > a bad idea.
> > 3 - Markers are exported directly to userspace. Kernel developers don't
> > want to have to bother about a userspace API when they move around
> > kernel code. This is why it makes sense to expose the changes made by
> > modifying the kernel code to an in-kernel API only.
> > 4 - Maintainability. Tracepoints offer a central repository in
> > include/trace/*.h, while markers are declared directly in the kernel C
> > files, which makes it very hard to maintain.
> 
> 
> For reasons 1 and 2, can we change the marker code instead? I know it
> usually takes longer time to push patches to upstream if they change the
> existing code. But I feel that may be the long term solution. For reasons
> 3 and 4, I wonder if we can use a lighter API layer on top of the marker
> code.
> One problem is that the current tracepoint code shares a lot of similarities
> 
> as the marker code, e.g., they both maintain an entry table and use their
> own data sections. We need to drop the tracing overhead to a small percent.
> I think it is easier to get there with fewer lines of code.
> 

No. Tracepoints are by no mean a replacement for the markers.
Tracepoints do not include the format string that is exported by markers
to userspace to keep track of event typing.

So we keep tracepoints as an in-kernel API, presented to various
tracers, including LTTng, and also markers, which are mostly used by
LTTng to format the data into the buffers.

A patch implementing what I proposed below will be available ASAP.

Mathieu

> 
> > > > Any thought?
> > >
> > > You could try to disable the
> > lttng-instrumentation-*-tracepoints-probes.patch
> > > patches. These patches register tracepoint probes just to setup a marker.
> > I
> > > guess this is where the overhead is coming from.
> > >
> > > Anyway, do you have numbers that you can share with us? Mathieu still has
> > the
> > > performance numbers with the marker patches in the tracepoints.patch. I
> > guess
> > > he would be glad if he could update them.
> > >
> >
> > This performance degradation is not tied to the tracepoint mechanism
> > (it's more or less the same mechanism as the markers) but to the way I
> > enable all the tracepoints contained in a probe (see
> > kernel/kernel-trace.c for example) at module load time.
> >
> > The way to fix this would be to come with a way to dynamically enable
> > the tracepoint which corresponds to a marker one by one.
> >
> > I could just do something similar to /proc/ltt which exports the
> > tracepoint list and let userspace enable them, but as I said, this is
> > and is meant to stay an in-kernel API. Therefore, we have to think of a
> > way to let a given kernel probe figure out it's time to enable a given
> > tracepoint.
> 
> 
> I was thinking to use this method. But while working on that, I feel it
> basically the same kind of code as marker connect/disconnect and am
> not sure why we want to do both.
> 
> 
> > Here is what I could do : I could add an optional parameter to the
> > markers that would take the associated tracepoint name and the name of
> > the callback we are planning to associate with the tracepoint. I could
> > therefore use either a gcc builtin type verification or create a
> > static inline unused function in the macro to make sure the function
> > type matches the type exported by the tracepoint.
> 
> 
> This sounds a better solution because we can enable/disable a tracepoint
> with one action to /proc/ltt. Users would like simple solutions.
> 
> Jiaying
> 
> 
> >
> > Then it's simply a matter of making the marker code call into the
> > tracepoint registration/unregistration functions whenever the marker is
> > enabled. Note that in this case, we can remove the if(unlikely()) branch
> > from the marker code because it's already done at the tracepoint level,
> > and thus becomes redundant.
> >
> > Comments ?
> >
> > Mathieu
> >
> >
> > > Regards,
> > >       Jan
> > >
> > > --
> > > Jan Blunck <jblunck at suse.de>
> > >
> >
> > --
> > Mathieu Desnoyers
> > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> >

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68