[lttng-dev] [PATCH lttng-modules] Update kernel probes to more detailed match to kernel versions

[Gabbasov, Andrew]

Hi Mathieu,

> > * Gabbasov, Andrew (Andrew_Gabbasov at mentor.com) wrote:
> > > Hi Mathieu,
> > >
> > > > * Andrew Gabbasov (andrew_gabbasov at mentor.com) wrote:
> > > > > We have added some more ifdef's to kernel probes to more closely match
> > > > > the kernel tracepoints for different kernel versions. The changes cover
> > > > > kernel versions from 2.6.38 up to 3.7. We did not track the earlier
> > > > > versions, and they are filtered out in probes Makefile.
> > > > >
> > > > > ext3 probe was modified to use #include <../fs/ext3/ext3.h> instead of
> > > > > copying this include from kernel source to lttng-modules source.
> > > > > Using such includes (there will be more similar in other probes)
> > > > > imposes a restriction to use the full kernel source for building
> > > > > lttng-modules rather than having just kernel "include" directory,
> > > > > but it looks like full source is required anyway, at least with respect
> > > > > to Makefiles structure. May be it is worth mentioning somewhere
> > > > > in the documentation.
> > > > >
> > > > > The code was verified to compile with latest versions in all stable
> > > > > branches from 2.6.38.x to 3.6.x and 3.7-rc7.
> > > >
> > > > I get this build failure on 3.5 with this patch:
> > > >
> > > >   CC [M]  /home/compudj/work/lttng-modules/probes/lttng-probe-ext3.o
> > > > /home/compudj/work/lttng-modules/probes/lttng-probe-ext3.c:30:29: fatal
> > > > error: ../fs/ext3/ext3.h: No such file or directory
> > > > compilation terminated.
> > > >
> > > > can you look into it and submit a revised patch ?
> > >
> > > You must be using kernel headers for building lttng-modules
> > > (linux-headers package or something like that), right? This is what I
> > > was writing about: since ext3.h is not available in kernel's "include"
> > > directory, and in order to avoid duplicating this file in
> > > lttng-modules sources, we have to use that <../> based reference, and
> > > this requires having full kernel source for building. The same issue
> > > will later be with btrfs and ext4 probes: their necessary include
> > > files are also not available in kernel headers.
> > >
> > > Trying to duplicate these header files to lttng-modules source seems
> > > to be not a good idea, especially if we need to add many ifdef's to
> > > them to track different kernel versions. We can try to insert some
> > > checks to the Makefile and skip building those probes if we do not
> > > have full kernel sources (actually if we do not have necessary include
> > > files). Or to declare having full kernel source as a requirement for
> > > building lttng-modules. What do you think?
> >
> > The user experience should be that if lttng-modules is built against
> > linux-headers packages, the build should work fine, even if it means
> > some extra features are disabled.
> >
> > And we don't want local copies of kernel headers, because then we could
> > trigger runtime bugs if the headers don't match between the kernel
> > version and local version.
> 
> OK, then I'm inserting the checks to Makefile, that build such probes if the full
> kernel source is available and to skip them if not.
> 
> >
> > >
> > > > Also, although the README file states that we support only 3.6.38+, we
> > > > happen to also have support for kernels down to 2.6.32 with the patches
> > > > in the linux-patches/ directory applied to the appropriate kernel
> > > > versions. So it would be good that the instrumentation compiles for
> > > > those older kernels too. I promise we won't go further back than 2.6.32
> > > > though.
> > >
> > > We'll try to add the ifdef's for earlier version down to 2.6.32 to the
> > > probes, although the additional amount of conditions makes the code
> > > overcomplicated ;-)
> >
> > Indeed, but I never said supporting various kernels was easy ;)
> >
> > I don't care immensely if older kernels have less features: in some
> > cases, it can be just OK to disable an entire tracepoint probe for older
> > kernels. I just want the build to work fine for those.
> 
> Understood, updated probes (as well as some additional ones) will be
> sent soon.
> 
> > By the way, we're planning to go for a 2.1 release in a matter of days.
> > At that point, I'll have to decide if we keep the recent instrumentation
> > commits introduced after the start of the -rc cycle in this release (if
> > mature enough), or if we should branch before those for the 2.1 release
> > if they need more testing (and keep these instrumentation extensions for
> > 2.2). Any thoughts on this ?
> 
> I believe, it would be desirable to make more thorough testing of new
> instrumentations. On our side the testing was not too extensive, and was
> done for ARM platform only. Although the probes are mostly replicating
> the kernel's tracepoints, it would be reasonable to keep the commits
> in a branch for some time so that more people in the community could
> make some testing.
> 
> Also, a side note: the "lock" probe seems to be not working. It causes
> kernel hang when trying to perform 'lttng start' at least on arm platform).
> We suspect that it may be caused by some kind of locks recursion or
> deadlocks (since tracing engine use locks too and they may be tried
> to trace again). There is no solution or even more detailed analysis
> so far, so I'll disable building of "lock" probe in the next update.

I'm sending a separate email with the second version of the patch,
containing instrumentations, updated for kernel versions 2.6.32+.

The probes/Makefile skips building of ext3 probe if appropriate header files
in the kernel source are not available.

Also, building of "lock" probe is blocked/commented out in the Makefile,
since it's not working. I'm leaving the probe itself in the sources for the case
if somebody will be able to fix it. ;-)

Thanks.

Best regards,
Andrew