[ltt-dev] tracepoint layer and marker layer

[Jiaying Zhang]

Hi Mathieu,

Thanks for the detailed explanation! See my comments below.

On Thu, Sep 18, 2008 at 9:00 AM, Mathieu Desnoyers <
compudj at krystal.dyndns.org> wrote:

> * Jan Blunck (jblunck at suse.de) wrote:
> > On Wed, Sep 17, Jiaying Zhang wrote:
> >
> > > Hi Folks,
> > >
> > > I have a general question about LTTng kernel code. As I understand,
> LTTng
> > > adds an extra tracepoint layer between the traced subsystems and the
> > > marker layer. The extra code added by the tracepoint layer seems to
> > > introduce
> > > noticeable overhead according to my measurements. I wonder whether it
> will
> > > be more efficient and simpler if we let subsystems call the marker
> layer
> > > directly?
> >
> > I guess that was how it worked before the tracepoints. Although I don't
> > fully understand why the redirection is necessary.
> >
>
> The redirection has been asked by kernel developers for four reasons :
>
> 1 - Markers make the kernel code ugly by adding a format string
> 2 - Markers do not support all what is needed by in-kernel probes from
> other tracers, which is mainly to export pointers to structures along
> with the complete type so the structure content can be later looked into
> by the probe. Markers can only export "%p" (void *). And type casting is
> a bad idea.
> 3 - Markers are exported directly to userspace. Kernel developers don't
> want to have to bother about a userspace API when they move around
> kernel code. This is why it makes sense to expose the changes made by
> modifying the kernel code to an in-kernel API only.
> 4 - Maintainability. Tracepoints offer a central repository in
> include/trace/*.h, while markers are declared directly in the kernel C
> files, which makes it very hard to maintain.


For reasons 1 and 2, can we change the marker code instead? I know it
usually takes longer time to push patches to upstream if they change the
existing code. But I feel that may be the long term solution. For reasons
3 and 4, I wonder if we can use a lighter API layer on top of the marker
code.
One problem is that the current tracepoint code shares a lot of similarities

as the marker code, e.g., they both maintain an entry table and use their
own data sections. We need to drop the tracing overhead to a small percent.
I think it is easier to get there with fewer lines of code.


> > > Any thought?
> >
> > You could try to disable the
> lttng-instrumentation-*-tracepoints-probes.patch
> > patches. These patches register tracepoint probes just to setup a marker.
> I
> > guess this is where the overhead is coming from.
> >
> > Anyway, do you have numbers that you can share with us? Mathieu still has
> the
> > performance numbers with the marker patches in the tracepoints.patch. I
> guess
> > he would be glad if he could update them.
> >
>
> This performance degradation is not tied to the tracepoint mechanism
> (it's more or less the same mechanism as the markers) but to the way I
> enable all the tracepoints contained in a probe (see
> kernel/kernel-trace.c for example) at module load time.
>
> The way to fix this would be to come with a way to dynamically enable
> the tracepoint which corresponds to a marker one by one.
>
> I could just do something similar to /proc/ltt which exports the
> tracepoint list and let userspace enable them, but as I said, this is
> and is meant to stay an in-kernel API. Therefore, we have to think of a
> way to let a given kernel probe figure out it's time to enable a given
> tracepoint.


I was thinking to use this method. But while working on that, I feel it
basically the same kind of code as marker connect/disconnect and am
not sure why we want to do both.


> Here is what I could do : I could add an optional parameter to the
> markers that would take the associated tracepoint name and the name of
> the callback we are planning to associate with the tracepoint. I could
> therefore use either a gcc builtin type verification or create a
> static inline unused function in the macro to make sure the function
> type matches the type exported by the tracepoint.


This sounds a better solution because we can enable/disable a tracepoint
with one action to /proc/ltt. Users would like simple solutions.

Jiaying


>
> Then it's simply a matter of making the marker code call into the
> tracepoint registration/unregistration functions whenever the marker is
> enabled. Note that in this case, we can remove the if(unlikely()) branch
> from the marker code because it's already done at the tracepoint level,
> and thus becomes redundant.
>
> Comments ?
>
> Mathieu
>
>
> > Regards,
> >       Jan
> >
> > --
> > Jan Blunck <jblunck at suse.de>
> >
>
> --
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20080918/aa554f82/attachment-0003.htm>