[lttng-dev] loading modules with modprobe
Kienan Stewart
kstewart at efficios.com
Thu Jul 18 10:58:20 EDT 2024
A couple of addendums inline below
On 7/18/24 10:47 AM, Kienan Stewart via lttng-dev wrote:
> Hi François,
>
> The dmesg excerpts are spaced by a number of minutes each, are they from
> different attempts to build and load the modules?
>
> On 7/17/24 8:44 PM, François Belias wrote:
>> Hello Kienan,
>>
>> Thanks to your advice, I managed to load some modules. However, I
>> encountered errors like these when running `dmesg | grep lttng`:
>>
>> ```
>> [ 218.043672] module lttng_kretprobes: .gnu.linkonce.this_module
>> section size must match the kernel's built struct module size at run time
>
> This sounds like the version of the module is built against the wrong
> kernel headers or source.
Also, are you using lttng-modules master or did you swap back to
stable-2.13?
If you're still on master and have the most recent changes
lttng_kretprobes isn't required anymore. You will also need lttng-tools
from master with the corresponding changes so that it doesn't try to
load lttng_kretprobes.
>
>> [ 650.560883] lttng_kretprobes: module verification failed: signature
>> and/or required key missing - tainting kernel
>
> Is your system running with secure boot enabled? If so, you either need
> to disable secure boot or sign the built modules with a MOK key that is
> correctly enrolled.
>
> See the following documentation:
>
> - https://lttng.org/docs/v2.13/#doc-linux-kernel-sig
> - https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html
> - https://github.com/dell/dkms
> -
> https://gitlab.com/linux-kernel/stable/-/blob/master/kernel/module/main.c#L2064
>
> Whatever you did for this test is pretty closed to working. The previous
> errors with the section size check happen earlier in the module load
> process. This check seems to be one of the last, c.f.
> https://elixir.bootlin.com/linux/v6.10/source/kernel/module/main.c#L2908
>
>> [ 1468.329988] module lttng_ring_buffer_client_discard:
>> .gnu.linkonce.this_module section size must match the kernel's built
>> struct module size at run time
>> ```
>>
>> I'm unsure of their meaning. After loading my module, how can I be
>> certain that my tracepoints are active? I read online that people use
>> `sudo lttng list -k` for this purpose, but when I tried it, I got the
>> error: "Error: Unable to list kernel events: Kernel tracer not
>> available". Is there another way?
>
> If you're still getting "kernel tracer not available" then either
> required kernel modules haven't been loaded or you started lttng-session
> with `--no-kernel` option. `lttng-sessiond -vvv`, dmesg, and/or
> journalctl should have information on which modules failed to load.
>
I know you're doing `sudo lttng list -k`, but the basic checks still apply:
- is the lttng-sessiond running as the root user
- (when running without sudo) does the tracing group exist, and is your
user a member of that group
> kienan
>
>>
>> Thank you for your assistance.
>>
>> Le mer. 17 juill. 2024, à 12 h 08, Kienan Stewart
>> <kstewart at efficios.com <mailto:kstewart at efficios.com>> a écrit :
>>
>> Hi François,
>>
>> On 7/17/24 11:47 AM, François Belias wrote:
>> > Hello, Kienan
>> >
>> >
>> > *Right. Which kernel are you actually running at this time? You
>> can check
>> > using: `uname -a`*
>> > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
>> PREEMPT_DYNAMIC
>> > Mon Jun 3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
>> >
>> > *Since you've got a few different images and headers installed,
>> it is
>> > possible to check that same info the linux-source package:
>> `dpkg -l |
>> > grep linux-source`.*
>> > ii linux-source 5.15.0.116.116
>> > all Linux kernel source
>> with
>> > Ubuntu patches
>> > ii linux-source-5.15.0 5.15.0-116.126
>> > all Linux kernel source for
>> version
>> > 5.15.0 with Ubuntu patches
>> > ii linux-source-6.5.0 6.5.0-44.44~22.04.1
>> > all Linux kernel source for
>> version
>>
>> Your currently running `6.5.0-41-generic #41~22.04.2-Ubuntu`
>>
>> You have source installed for `6.5.0-44.44~22.04.1`. From your
>> previous
>> e-mail, there is also a linux image installed:
>>
>> ```
>> linux-image-6.5.0-44-generic 6.5.0-44.44~22.04.1
>> ```
>>
>> Assuming you are using grub, could you verify that one of the options
>> for booting in `/boot/grub/grub.cfg` is linux 6.5.0-44.44. Look
>> for the
>> lines similar to the following inside a menu entry block:
>>
>> (Note my example is 5.15.0-116, but you will be lookng for
>> 6.5.0-44 or
>> 6.5.0-44.44)
>>
>> ```
>> linux /boot/vmlinuz-5.15.0-116-generic root=....
>> initrd /boot/initrd.img-5.15.0-116-generic
>> ```
>>
>> If you have a grub entry for linux 6.5.0-44.44, could you reboot
>> to run
>> on the linux 6.5.0-44.44.
>>
>> After rebooting verify with `uname -a` that your kernel version is
>> 6.5.0-44.44.
>>
>> Make sure you have unpacked the linux sources:
>>
>> ```
>> # Approximating the version numbers, since I don't have 6.5.0
>> installed
>> cd /usr/src/linux-source-6.5.0
>> bunzip2 linux-source-6.5.0.tar.bz2
>> tar -xf linux-source-6.5.0.tar
>> ```
>>
>> Then retry the build with
>> `KERNELDIR=/usr/src/linux-source-6.5.0/linux-source-6.5.0/`
>>
>> I haven't personally tried the full steps on an Ubuntu
>> installation so
>> you may still need to tweak a couple of things, but I think this
>> should
>> get you on the right path.
>>
>> Really the important thing is to make sure that the kernel source
>> code
>> matches the kernel you are running.
>>
>> > 6.5.0 with Ubuntu patches
>> > *Could you run either `apt-cache policy` or `apt policy` and
>> `apt-mark
>> > showhold` in order to know which sources you have configured.*
>> > For this part, /*apt-mark showhold*/ gave me nothing I don't know
>> what
>> > is means.
>>
>> `apt-mark showhold` will show any packages that are on hold - it is a
>> way of marking packages that shouldn't be updated.
>>
>> As you have the linux-source for 6.5.0-44.44~22.04.1, I don't
>> think it
>> will be necessary to have the output of `apt-cache policy`.
>>
>> thanks,
>> kienan
>>
>> >
>> > I am looking at the links, but nothing seems to match my kernel
>> version.
>> >
>> > Thanks for your assistance.
>> >
>> > Le mer. 17 juill. 2024, à 11 h 46, François Belias
>> > <beliasossim at gmail.com <mailto:beliasossim at gmail.com>
>> <mailto:beliasossim at gmail.com <mailto:beliasossim at gmail.com>>> a
>> écrit :
>> >
>> > Hello, Kienan
>> >
>> >
>> > *Right. Which kernel are you actually running at this time?
>> You can
>> > check
>> > using: `uname -a`*
>> > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP
>> > PREEMPT_DYNAMIC Mon Jun 3 11:32:55 UTC 2 x86_64 x86_64
>> x86_64 GNU/Linux
>> >
>> > *Since you've got a few different images and headers
>> installed, it is
>> > possible to check that same info the linux-source package:
>> `dpkg -l |
>> > grep linux-source`.*
>> > ii linux-source 5.15.0.116.116
>> > all Linux kernel
>> source
>> > with Ubuntu patches
>> > ii linux-source-5.15.0 5.15.0-116.126
>> > all Linux kernel
>> source for
>> > version 5.15.0 with Ubuntu patches
>> > ii linux-source-6.5.0
>> 6.5.0-44.44~22.04.1
>> > all Linux kernel source
>> for
>> > version 6.5.0 with Ubuntu patches
>> > *Could you run either `apt-cache policy` or `apt policy` and
>> `apt-mark
>> > showhold` in order to know which sources you have configured.*
>> > For this part, /*apt-mark showhold*/ gave me nothing I
>> don't know
>> > what is means.
>> >
>> > I am looking at the links, but nothing seems to match my
>> kernel
>> > version.
>> >
>> > Thanks for your assistance.
>> >
>> > Le mer. 17 juill. 2024, à 10 h 55, Kienan Stewart
>> > <kstewart at efficios.com <mailto:kstewart at efficios.com>
>> <mailto:kstewart at efficios.com <mailto:kstewart at efficios.com>>> a
>> écrit :
>> >
>> > Hi François,
>> >
>> > On 7/16/24 4:34 PM, François Belias wrote:
>> > > Hello Kienan,
>> > >
>> > > When I proceed with the following steps:
>> > >
>> > > ```
>> > > # Warning: this removes all untracked content from the
>> > lttng-modules
>> > > directory
>> > > cd /path/to/lttng-modules-source
>> > > git clean -dxf
>> > >
>> > > make
>> > > sudo make modules_install
>> > > sudo depmod -a
>> > > ```
>> > >
>> > > I encounter a warning in
>> > > `/home/philippe/lttng-modules/src/probes/Kbuild:79`
>> stating
>> > "File
>> > > ./arch/x86/kvm/lapic.h not found. Probe 'kvm'
>> x86-specific is
>> > disabled.
>> > > Use full kernel source tree to enable it." Although
>> > everything compiles
>> > > fine and I'm able to load modules, when I execute
>> `sudo lttng
>> > list -k`,
>> > > I don't see the tracepoints `kvm_entry` and
>> `kvm_exit`, which
>> > I believe
>> > > should be available. Could this be due to not using my
>> full
>> > kernel tree
>> > > during compilation?
>> >
>> >
>> > Exactly. Those probes depend on having the full kernel
>> source as
>> > they
>> > require private headers which aren't shipped as part of
>> the
>> > `linux-headers` package.
>> >
>> > >
>> > > Here are the answers to your questions:
>> > >
>> > > - *Where did `/usr/src/linux-source-6.5.0` come from?*
>> > > I initially attempted `sudo apt-get update && sudo
>> apt-get
>> > install
>> > > linux-source`, but it gave me this answer
>> */linux-source is
>> > already the
>> > > newest version (5.15.0.116.116)/*`, which doesn't
>> match my
>> > current
>> > > kernel version. So, I downloaded it directly from
>> kernel.org <http://kernel.org>
>> > <http://kernel.org <http://kernel.org>>
>> > > <http://kernel.org <http://kernel.org>
>> <http://kernel.org <http://kernel.org>>> using this command:
>> > `wget
>> > >
>> > https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
>> >
>> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
>> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>
>> > >
>> >
>> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`
>> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>
>> >
>> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz
>> <https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz>>>.
>> > >
>> >
>> > Right. Which kernel are you actually running at this
>> time? You
>> > can check
>> > using: `uname -a`
>> >
>> > In any case, Linux upstream 6.5 is not the same as
>> Ubuntus's 6.5.
>> >
>> > > - *Which Linux distribution and version are you using?*
>> > > Running `lsb_release -a` gives me:
>> > > ```
>> > > No LSB modules are available.
>> > > Distributor ID: Ubuntu
>> > > Description: Ubuntu 22.04.4 LTS
>> > > Release: 22.04
>> > > Codename: jammy
>> > > ```
>> > >
>> > > - *Assuming your distribution has kernel packages,
>> which do
>> > you have
>> > > installed, and do you have the corresponding kernel
>> headers
>> > package
>> > > installed?*
>> > >
>> > > Running `dpkg -l | grep linux-image` shows these
>> installed
>> > packages:
>> > > ```
>> > > ii linux-image-5.15.0-116-generic
>> 5.15.0-116.126
>> > > amd64 Signed kernel image generic
>> > > ii linux-image-6.5.0-28-generic
>> > 6.5.0-28.29~22.04.1
>> > > amd64 Signed kernel image generic
>> > > ii linux-image-6.5.0-41-generic
>> > 6.5.0-41.41~22.04.2
>> > > amd64 Signed kernel image generic
>> > > ii linux-image-6.5.0-44-generic
>> > 6.5.0-44.44~22.04.1
>> > > amd64 Signed kernel image generic
>> > > ii linux-image-generic 5.15.0.116.116
>> > > amd64 Generic Linux kernel image
>> > > ii linux-image-generic-hwe-22.04
>> > 6.5.0.44.44~22.04.1
>> > > amd64 Generic Linux kernel image
>> > > ```
>> > >
>> > > And `dpkg -l | grep linux-headers` shows these
>> installed
>> > header packages:
>> > > ```
>> > > ii linux-headers-5.15.0-116
>> > 5.15.0-116.126 all
>> > > Header files related to Linux kernel version
>> 5.15.0
>> > > ii linux-headers-5.15.0-116-generic
>> 5.15.0-116.126
>> > > amd64 Linux kernel headers for version 5.15.0 on
>> 64 bit
>> > x86 SMP
>> > > ii linux-headers-6.5.0-28-generic
>> > 6.5.0-28.29~22.04.1
>> > > amd64 Linux kernel headers for version 6.5.0 on
>> 64 bit
>> > x86 SMP
>> > > ii linux-headers-6.5.0-41-generic
>> > 6.5.0-41.41~22.04.2
>> > > amd64 Linux kernel headers for version 6.5.0 on
>> 64 bit
>> > x86 SMP
>> > > ii linux-headers-6.5.0-44-generic
>> > 6.5.0-44.44~22.04.1
>> > > amd64 Linux kernel headers for version 6.5.0 on
>> 64 bit
>> > x86 SMP
>> > > ii linux-headers-generic 5.15.0.116.116
>> > > amd64 Generic Linux kernel headers
>> > > ii linux-headers-generic-hwe-22.04
>> > 6.5.0.44.44~22.04.1
>> > > amd64 Generic Linux kernel headers
>> > > ```
>> > >
>> >
>> > Since you've got a few different images and headers
>> installed,
>> > it is
>> > possible to check that same info the linux-source
>> package: `dpkg
>> > -l |
>> > grep linux-source`.
>> >
>> > Maybe one thing tripping this process up is that you
>> don't have
>> > an apt
>> > source that provides linux-source that matches the
>> kernel you
>> > are using.
>> > Could you run either `apt-cache policy` or `apt policy`
>> and
>> > `apt-mark
>> > showhold` in order to know which sources you have
>> configured.
>> >
>> > There are also alternative sources for the full source
>> tree:
>> >
>> > - You could download the specific package from
>> > packages.ubuntu.com <http://packages.ubuntu.com>
>> <http://packages.ubuntu.com <http://packages.ubuntu.com>>. See
>> > https://packages.ubuntu.com <https://packages.ubuntu.com>
>> <https://packages.ubuntu.com <https://packages.ubuntu.com>> and
>> > use `apt install ./downloaded_package.deb`
>> >
>> > - You could clone the kernel source for Ubuntu and
>> checkout the
>> > appropriate tag. See
>> >
>>
>> https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags> <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags <https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags>>
>> >
>> > thanks,
>> > kienan
>> >
>>
> _______________________________________________
> lttng-dev mailing list
> lttng-dev at lists.lttng.org
> https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
More information about the lttng-dev
mailing list