<div dir="ltr">Hello Kienan, <div><br></div><div> my issue has been fixed. thanks for your help</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le jeu. 18 juill. 2024, à 10 h 58, Kienan Stewart <<a href="mailto:kstewart@efficios.com">kstewart@efficios.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">A couple of addendums inline below<br>
<br>
On 7/18/24 10:47 AM, Kienan Stewart via lttng-dev wrote:<br>
> Hi François,<br>
> <br>
> The dmesg excerpts are spaced by a number of minutes each, are they from <br>
> different attempts to build and load the modules?<br>
> <br>
> On 7/17/24 8:44 PM, François Belias wrote:<br>
>> Hello Kienan,<br>
>><br>
>> Thanks to your advice, I managed to load some modules. However, I <br>
>> encountered errors like these when running `dmesg | grep lttng`:<br>
>><br>
>> ```<br>
>> [ 218.043672] module lttng_kretprobes: .gnu.linkonce.this_module <br>
>> section size must match the kernel's built struct module size at run time<br>
> <br>
> This sounds like the version of the module is built against the wrong <br>
> kernel headers or source.<br>
<br>
Also, are you using lttng-modules master or did you swap back to <br>
stable-2.13?<br>
<br>
If you're still on master and have the most recent changes <br>
lttng_kretprobes isn't required anymore. You will also need lttng-tools <br>
from master with the corresponding changes so that it doesn't try to <br>
load lttng_kretprobes.<br>
<br>
> <br>
>> [ 650.560883] lttng_kretprobes: module verification failed: signature <br>
>> and/or required key missing - tainting kernel<br>
> <br>
> Is your system running with secure boot enabled? If so, you either need <br>
> to disable secure boot or sign the built modules with a MOK key that is <br>
> correctly enrolled.<br>
> <br>
> See the following documentation:<br>
> <br>
> - <a href="https://lttng.org/docs/v2.13/#doc-linux-kernel-sig" rel="noreferrer" target="_blank">https://lttng.org/docs/v2.13/#doc-linux-kernel-sig</a><br>
> - <a href="https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html" rel="noreferrer" target="_blank">https://www.kernel.org/doc/html/latest/admin-guide/module-signing.html</a><br>
> - <a href="https://github.com/dell/dkms" rel="noreferrer" target="_blank">https://github.com/dell/dkms</a><br>
> - <br>
> <a href="https://gitlab.com/linux-kernel/stable/-/blob/master/kernel/module/main.c#L2064" rel="noreferrer" target="_blank">https://gitlab.com/linux-kernel/stable/-/blob/master/kernel/module/main.c#L2064</a><br>
> <br>
> Whatever you did for this test is pretty closed to working. The previous <br>
> errors with the section size check happen earlier in the module load <br>
> process. This check seems to be one of the last, c.f. <br>
> <a href="https://elixir.bootlin.com/linux/v6.10/source/kernel/module/main.c#L2908" rel="noreferrer" target="_blank">https://elixir.bootlin.com/linux/v6.10/source/kernel/module/main.c#L2908</a><br>
> <br>
>> [ 1468.329988] module lttng_ring_buffer_client_discard: <br>
>> .gnu.linkonce.this_module section size must match the kernel's built <br>
>> struct module size at run time<br>
>> ```<br>
>><br>
>> I'm unsure of their meaning. After loading my module, how can I be <br>
>> certain that my tracepoints are active? I read online that people use <br>
>> `sudo lttng list -k` for this purpose, but when I tried it, I got the <br>
>> error: "Error: Unable to list kernel events: Kernel tracer not <br>
>> available". Is there another way?<br>
> <br>
> If you're still getting "kernel tracer not available" then either <br>
> required kernel modules haven't been loaded or you started lttng-session <br>
> with `--no-kernel` option. `lttng-sessiond -vvv`, dmesg, and/or <br>
> journalctl should have information on which modules failed to load.<br>
> <br>
<br>
I know you're doing `sudo lttng list -k`, but the basic checks still apply:<br>
<br>
- is the lttng-sessiond running as the root user<br>
- (when running without sudo) does the tracing group exist, and is your <br>
user a member of that group<br>
<br>
> kienan<br>
> <br>
>><br>
>> Thank you for your assistance.<br>
>><br>
>> Le mer. 17 juill. 2024, à 12 h 08, Kienan Stewart <br>
>> <<a href="mailto:kstewart@efficios.com" target="_blank">kstewart@efficios.com</a> <mailto:<a href="mailto:kstewart@efficios.com" target="_blank">kstewart@efficios.com</a>>> a écrit :<br>
>><br>
>> Hi François,<br>
>><br>
>> On 7/17/24 11:47 AM, François Belias wrote:<br>
>> > Hello, Kienan<br>
>> ><br>
>> ><br>
>> > *Right. Which kernel are you actually running at this time? You<br>
>> can check<br>
>> > using: `uname -a`*<br>
>> > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP<br>
>> PREEMPT_DYNAMIC<br>
>> > Mon Jun 3 11:32:55 UTC 2 x86_64 x86_64 x86_64 GNU/Linux<br>
>> ><br>
>> > *Since you've got a few different images and headers installed, <br>
>> it is<br>
>> > possible to check that same info the linux-source package: <br>
>> `dpkg -l |<br>
>> > grep linux-source`.*<br>
>> > ii linux-source 5.15.0.116.116<br>
>> > all Linux kernel source <br>
>> with<br>
>> > Ubuntu patches<br>
>> > ii linux-source-5.15.0 5.15.0-116.126<br>
>> > all Linux kernel source for <br>
>> version<br>
>> > 5.15.0 with Ubuntu patches<br>
>> > ii linux-source-6.5.0 6.5.0-44.44~22.04.1<br>
>> > all Linux kernel source for <br>
>> version<br>
>><br>
>> Your currently running `6.5.0-41-generic #41~22.04.2-Ubuntu`<br>
>><br>
>> You have source installed for `6.5.0-44.44~22.04.1`. From your <br>
>> previous<br>
>> e-mail, there is also a linux image installed:<br>
>><br>
>> ```<br>
>> linux-image-6.5.0-44-generic 6.5.0-44.44~22.04.1<br>
>> ```<br>
>><br>
>> Assuming you are using grub, could you verify that one of the options<br>
>> for booting in `/boot/grub/grub.cfg` is linux 6.5.0-44.44. Look <br>
>> for the<br>
>> lines similar to the following inside a menu entry block:<br>
>><br>
>> (Note my example is 5.15.0-116, but you will be lookng for <br>
>> 6.5.0-44 or<br>
>> 6.5.0-44.44)<br>
>><br>
>> ```<br>
>> linux /boot/vmlinuz-5.15.0-116-generic root=....<br>
>> initrd /boot/initrd.img-5.15.0-116-generic<br>
>> ```<br>
>><br>
>> If you have a grub entry for linux 6.5.0-44.44, could you reboot <br>
>> to run<br>
>> on the linux 6.5.0-44.44.<br>
>><br>
>> After rebooting verify with `uname -a` that your kernel version is<br>
>> 6.5.0-44.44.<br>
>><br>
>> Make sure you have unpacked the linux sources:<br>
>><br>
>> ```<br>
>> # Approximating the version numbers, since I don't have 6.5.0 <br>
>> installed<br>
>> cd /usr/src/linux-source-6.5.0<br>
>> bunzip2 linux-source-6.5.0.tar.bz2<br>
>> tar -xf linux-source-6.5.0.tar<br>
>> ```<br>
>><br>
>> Then retry the build with<br>
>> `KERNELDIR=/usr/src/linux-source-6.5.0/linux-source-6.5.0/`<br>
>><br>
>> I haven't personally tried the full steps on an Ubuntu <br>
>> installation so<br>
>> you may still need to tweak a couple of things, but I think this <br>
>> should<br>
>> get you on the right path.<br>
>><br>
>> Really the important thing is to make sure that the kernel source <br>
>> code<br>
>> matches the kernel you are running.<br>
>><br>
>> > 6.5.0 with Ubuntu patches<br>
>> > *Could you run either `apt-cache policy` or `apt policy` and<br>
>> `apt-mark<br>
>> > showhold` in order to know which sources you have configured.*<br>
>> > For this part, /*apt-mark showhold*/ gave me nothing I don't know<br>
>> what<br>
>> > is means.<br>
>><br>
>> `apt-mark showhold` will show any packages that are on hold - it is a<br>
>> way of marking packages that shouldn't be updated.<br>
>><br>
>> As you have the linux-source for 6.5.0-44.44~22.04.1, I don't <br>
>> think it<br>
>> will be necessary to have the output of `apt-cache policy`.<br>
>><br>
>> thanks,<br>
>> kienan<br>
>><br>
>> ><br>
>> > I am looking at the links, but nothing seems to match my kernel<br>
>> version.<br>
>> ><br>
>> > Thanks for your assistance.<br>
>> ><br>
>> > Le mer. 17 juill. 2024, à 11 h 46, François Belias<br>
>> > <<a href="mailto:beliasossim@gmail.com" target="_blank">beliasossim@gmail.com</a> <mailto:<a href="mailto:beliasossim@gmail.com" target="_blank">beliasossim@gmail.com</a>><br>
>> <mailto:<a href="mailto:beliasossim@gmail.com" target="_blank">beliasossim@gmail.com</a> <mailto:<a href="mailto:beliasossim@gmail.com" target="_blank">beliasossim@gmail.com</a>>>> a <br>
>> écrit :<br>
>> ><br>
>> > Hello, Kienan<br>
>> ><br>
>> ><br>
>> > *Right. Which kernel are you actually running at this time?<br>
>> You can<br>
>> > check<br>
>> > using: `uname -a`*<br>
>> > Linux philippe 6.5.0-41-generic #41~22.04.2-Ubuntu SMP<br>
>> > PREEMPT_DYNAMIC Mon Jun 3 11:32:55 UTC 2 x86_64 x86_64<br>
>> x86_64 GNU/Linux<br>
>> ><br>
>> > *Since you've got a few different images and headers<br>
>> installed, it is<br>
>> > possible to check that same info the linux-source package:<br>
>> `dpkg -l |<br>
>> > grep linux-source`.*<br>
>> > ii linux-source 5.15.0.116.116<br>
>> > all Linux kernel<br>
>> source<br>
>> > with Ubuntu patches<br>
>> > ii linux-source-5.15.0 5.15.0-116.126<br>
>> > all Linux kernel <br>
>> source for<br>
>> > version 5.15.0 with Ubuntu patches<br>
>> > ii linux-source-6.5.0 <br>
>> 6.5.0-44.44~22.04.1<br>
>> > all Linux kernel source <br>
>> for<br>
>> > version 6.5.0 with Ubuntu patches<br>
>> > *Could you run either `apt-cache policy` or `apt policy` and<br>
>> `apt-mark<br>
>> > showhold` in order to know which sources you have configured.*<br>
>> > For this part, /*apt-mark showhold*/ gave me nothing I <br>
>> don't know<br>
>> > what is means.<br>
>> ><br>
>> > I am looking at the links, but nothing seems to match my <br>
>> kernel<br>
>> > version.<br>
>> ><br>
>> > Thanks for your assistance.<br>
>> ><br>
>> > Le mer. 17 juill. 2024, à 10 h 55, Kienan Stewart<br>
>> > <<a href="mailto:kstewart@efficios.com" target="_blank">kstewart@efficios.com</a> <mailto:<a href="mailto:kstewart@efficios.com" target="_blank">kstewart@efficios.com</a>><br>
>> <mailto:<a href="mailto:kstewart@efficios.com" target="_blank">kstewart@efficios.com</a> <mailto:<a href="mailto:kstewart@efficios.com" target="_blank">kstewart@efficios.com</a>>>> a <br>
>> écrit :<br>
>> ><br>
>> > Hi François,<br>
>> ><br>
>> > On 7/16/24 4:34 PM, François Belias wrote:<br>
>> > > Hello Kienan,<br>
>> > ><br>
>> > > When I proceed with the following steps:<br>
>> > ><br>
>> > > ```<br>
>> > > # Warning: this removes all untracked content from the<br>
>> > lttng-modules<br>
>> > > directory<br>
>> > > cd /path/to/lttng-modules-source<br>
>> > > git clean -dxf<br>
>> > ><br>
>> > > make<br>
>> > > sudo make modules_install<br>
>> > > sudo depmod -a<br>
>> > > ```<br>
>> > ><br>
>> > > I encounter a warning in<br>
>> > > `/home/philippe/lttng-modules/src/probes/Kbuild:79`<br>
>> stating<br>
>> > "File<br>
>> > > ./arch/x86/kvm/lapic.h not found. Probe 'kvm'<br>
>> x86-specific is<br>
>> > disabled.<br>
>> > > Use full kernel source tree to enable it." Although<br>
>> > everything compiles<br>
>> > > fine and I'm able to load modules, when I execute<br>
>> `sudo lttng<br>
>> > list -k`,<br>
>> > > I don't see the tracepoints `kvm_entry` and<br>
>> `kvm_exit`, which<br>
>> > I believe<br>
>> > > should be available. Could this be due to not using my<br>
>> full<br>
>> > kernel tree<br>
>> > > during compilation?<br>
>> ><br>
>> ><br>
>> > Exactly. Those probes depend on having the full kernel<br>
>> source as<br>
>> > they<br>
>> > require private headers which aren't shipped as part of <br>
>> the<br>
>> > `linux-headers` package.<br>
>> ><br>
>> > ><br>
>> > > Here are the answers to your questions:<br>
>> > ><br>
>> > > - *Where did `/usr/src/linux-source-6.5.0` come from?*<br>
>> > > I initially attempted `sudo apt-get update && sudo<br>
>> apt-get<br>
>> > install<br>
>> > > linux-source`, but it gave me this answer<br>
>> */linux-source is<br>
>> > already the<br>
>> > > newest version (5.15.0.116.116)/*`, which doesn't <br>
>> match my<br>
>> > current<br>
>> > > kernel version. So, I downloaded it directly from<br>
>> <a href="http://kernel.org" rel="noreferrer" target="_blank">kernel.org</a> <<a href="http://kernel.org" rel="noreferrer" target="_blank">http://kernel.org</a>><br>
>> > <<a href="http://kernel.org" rel="noreferrer" target="_blank">http://kernel.org</a> <<a href="http://kernel.org" rel="noreferrer" target="_blank">http://kernel.org</a>>><br>
>> > > <<a href="http://kernel.org" rel="noreferrer" target="_blank">http://kernel.org</a> <<a href="http://kernel.org" rel="noreferrer" target="_blank">http://kernel.org</a>><br>
>> <<a href="http://kernel.org" rel="noreferrer" target="_blank">http://kernel.org</a> <<a href="http://kernel.org" rel="noreferrer" target="_blank">http://kernel.org</a>>>> using this command:<br>
>> > `wget<br>
>> > ><br>
>> > <a href="https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz" rel="noreferrer" target="_blank">https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`</a><br>
>> <<a href="https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz" rel="noreferrer" target="_blank">https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz</a>><br>
>> > <br>
>> <<a href="https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz" rel="noreferrer" target="_blank">https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz</a><br>
>> <<a href="https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz" rel="noreferrer" target="_blank">https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz</a>>><br>
>> > ><br>
>> > <br>
>> <<a href="https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz" rel="noreferrer" target="_blank">https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz`</a><br>
>> <<a href="https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz" rel="noreferrer" target="_blank">https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz</a>><br>
>> > <br>
>> <<a href="https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz" rel="noreferrer" target="_blank">https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz</a><br>
>> <<a href="https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz" rel="noreferrer" target="_blank">https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.tar.xz</a>>>>.<br>
>> > ><br>
>> ><br>
>> > Right. Which kernel are you actually running at this<br>
>> time? You<br>
>> > can check<br>
>> > using: `uname -a`<br>
>> ><br>
>> > In any case, Linux upstream 6.5 is not the same as<br>
>> Ubuntus's 6.5.<br>
>> ><br>
>> > > - *Which Linux distribution and version are you using?*<br>
>> > > Running `lsb_release -a` gives me:<br>
>> > > ```<br>
>> > > No LSB modules are available.<br>
>> > > Distributor ID: Ubuntu<br>
>> > > Description: Ubuntu 22.04.4 LTS<br>
>> > > Release: 22.04<br>
>> > > Codename: jammy<br>
>> > > ```<br>
>> > ><br>
>> > > - *Assuming your distribution has kernel packages,<br>
>> which do<br>
>> > you have<br>
>> > > installed, and do you have the corresponding kernel<br>
>> headers<br>
>> > package<br>
>> > > installed?*<br>
>> > ><br>
>> > > Running `dpkg -l | grep linux-image` shows these<br>
>> installed<br>
>> > packages:<br>
>> > > ```<br>
>> > > ii linux-image-5.15.0-116-generic <br>
>> 5.15.0-116.126<br>
>> > > amd64 Signed kernel image generic<br>
>> > > ii linux-image-6.5.0-28-generic<br>
>> > 6.5.0-28.29~22.04.1<br>
>> > > amd64 Signed kernel image generic<br>
>> > > ii linux-image-6.5.0-41-generic<br>
>> > 6.5.0-41.41~22.04.2<br>
>> > > amd64 Signed kernel image generic<br>
>> > > ii linux-image-6.5.0-44-generic<br>
>> > 6.5.0-44.44~22.04.1<br>
>> > > amd64 Signed kernel image generic<br>
>> > > ii linux-image-generic 5.15.0.116.116<br>
>> > > amd64 Generic Linux kernel image<br>
>> > > ii linux-image-generic-hwe-22.04<br>
>> > 6.5.0.44.44~22.04.1<br>
>> > > amd64 Generic Linux kernel image<br>
>> > > ```<br>
>> > ><br>
>> > > And `dpkg -l | grep linux-headers` shows these<br>
>> installed<br>
>> > header packages:<br>
>> > > ```<br>
>> > > ii linux-headers-5.15.0-116<br>
>> > 5.15.0-116.126 all<br>
>> > > Header files related to Linux kernel version <br>
>> 5.15.0<br>
>> > > ii linux-headers-5.15.0-116-generic <br>
>> 5.15.0-116.126<br>
>> > > amd64 Linux kernel headers for version 5.15.0 on<br>
>> 64 bit<br>
>> > x86 SMP<br>
>> > > ii linux-headers-6.5.0-28-generic<br>
>> > 6.5.0-28.29~22.04.1<br>
>> > > amd64 Linux kernel headers for version 6.5.0 on <br>
>> 64 bit<br>
>> > x86 SMP<br>
>> > > ii linux-headers-6.5.0-41-generic<br>
>> > 6.5.0-41.41~22.04.2<br>
>> > > amd64 Linux kernel headers for version 6.5.0 on <br>
>> 64 bit<br>
>> > x86 SMP<br>
>> > > ii linux-headers-6.5.0-44-generic<br>
>> > 6.5.0-44.44~22.04.1<br>
>> > > amd64 Linux kernel headers for version 6.5.0 on <br>
>> 64 bit<br>
>> > x86 SMP<br>
>> > > ii linux-headers-generic 5.15.0.116.116<br>
>> > > amd64 Generic Linux kernel headers<br>
>> > > ii linux-headers-generic-hwe-22.04<br>
>> > 6.5.0.44.44~22.04.1<br>
>> > > amd64 Generic Linux kernel headers<br>
>> > > ```<br>
>> > ><br>
>> ><br>
>> > Since you've got a few different images and headers<br>
>> installed,<br>
>> > it is<br>
>> > possible to check that same info the linux-source<br>
>> package: `dpkg<br>
>> > -l |<br>
>> > grep linux-source`.<br>
>> ><br>
>> > Maybe one thing tripping this process up is that you<br>
>> don't have<br>
>> > an apt<br>
>> > source that provides linux-source that matches the <br>
>> kernel you<br>
>> > are using.<br>
>> > Could you run either `apt-cache policy` or `apt policy` <br>
>> and<br>
>> > `apt-mark<br>
>> > showhold` in order to know which sources you have <br>
>> configured.<br>
>> ><br>
>> > There are also alternative sources for the full source <br>
>> tree:<br>
>> ><br>
>> > - You could download the specific package from<br>
>> > <a href="http://packages.ubuntu.com" rel="noreferrer" target="_blank">packages.ubuntu.com</a> <<a href="http://packages.ubuntu.com" rel="noreferrer" target="_blank">http://packages.ubuntu.com</a>><br>
>> <<a href="http://packages.ubuntu.com" rel="noreferrer" target="_blank">http://packages.ubuntu.com</a> <<a href="http://packages.ubuntu.com" rel="noreferrer" target="_blank">http://packages.ubuntu.com</a>>>. See<br>
>> > <a href="https://packages.ubuntu.com" rel="noreferrer" target="_blank">https://packages.ubuntu.com</a> <<a href="https://packages.ubuntu.com" rel="noreferrer" target="_blank">https://packages.ubuntu.com</a>><br>
>> <<a href="https://packages.ubuntu.com" rel="noreferrer" target="_blank">https://packages.ubuntu.com</a> <<a href="https://packages.ubuntu.com" rel="noreferrer" target="_blank">https://packages.ubuntu.com</a>>> and<br>
>> > use `apt install ./downloaded_package.deb`<br>
>> ><br>
>> > - You could clone the kernel source for Ubuntu and<br>
>> checkout the<br>
>> > appropriate tag. See<br>
>> ><br>
>> <br>
>> <a href="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags" rel="noreferrer" target="_blank">https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags</a> <<a href="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags" rel="noreferrer" target="_blank">https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags</a>> <<a href="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags" rel="noreferrer" target="_blank">https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags</a> <<a href="https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags" rel="noreferrer" target="_blank">https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/refs/tags</a>>><br>
>> ><br>
>> > thanks,<br>
>> > kienan<br>
>> ><br>
>><br>
> _______________________________________________<br>
> lttng-dev mailing list<br>
> <a href="mailto:lttng-dev@lists.lttng.org" target="_blank">lttng-dev@lists.lttng.org</a><br>
> <a href="https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev" rel="noreferrer" target="_blank">https://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev</a><br>
</blockquote></div>