Kernel tracing from inside a Docker container

Thu Nov 7 11:00:50 EST 2024

Hi,

I guess I found it! The kernel module files have to be present under
/lib/modules inside the container, even when they are already loaded
on the host side.
After putting them there, everything works fine and I finally can
record kernel events.

Thank you so much for your support.

Regards
Dirk

On Thu, Nov 7, 2024 at 3:58 PM Dirk Eibach <dirk.eibach at googlemail.com> wrote:
>
> Hi Mathieu,
>
> > 1) You modprobe all LTTng modules from your top level namespace (host
> >     system) before starting lttng-sessiond within the container, or
>
> since I already did a lttng session on the host/top level namespacer
> the modules are already loaded:
>
> root at 9348eb19e6f4:/# lsmod
> Module                  Size  Used by
> lttng_probe_x86_exceptions    16384  0
> lttng_probe_x86_irq_vectors    16384  0
> lttng_probe_workqueue    20480  0
> lttng_probe_vmscan     32768  0
> lttng_probe_udp        16384  0
> lttng_probe_timer      28672  0
> lttng_probe_sunrpc     20480  0
> lttng_probe_statedump    45056  0
> lttng_probe_sock       16384  0
> lttng_probe_skb        16384  0
> lttng_probe_signal     16384  0
> lttng_probe_scsi       20480  0
> lttng_probe_sched      40960  0
> lttng_probe_rcu        16384  0
> lttng_probe_printk     16384  0
> lttng_probe_power      20480  0
> lttng_probe_net        28672  0
> lttng_probe_napi       16384  0
> lttng_probe_module     20480  0
> lttng_probe_kvm        32768  0
> lttng_probe_kmem       28672  0
> lttng_probe_jbd2       28672  0
> lttng_probe_irq        16384  0
> lttng_probe_i2c        20480  0
> lttng_probe_ext4      155648  0
> lttng_probe_compaction    16384  0
> lttng_probe_btrfs      65536  0
> lttng_probe_block      36864  0
> lttng_counter_client_percpu_32_modular    16384  0
> lttng_counter_client_percpu_64_modular    16384  1
> lttng_counter          16384  2
> lttng_counter_client_percpu_64_modular,lttng_counter_client_percpu_32_modular
> lttng_ring_buffer_event_notifier_client    16384  2
> lttng_ring_buffer_metadata_mmap_client    16384  0
> lttng_ring_buffer_client_mmap_overwrite    20480  0
> lttng_ring_buffer_client_mmap_discard    20480  0
> lttng_ring_buffer_metadata_client    16384  0
> lttng_ring_buffer_client_overwrite    20480  0
> lttng_ring_buffer_client_discard    20480  0
> lttng_tracer         2666496  40
> lttng_probe_udp,lttng_probe_scsi,lttng_probe_sched,lttng_probe_compaction,lttng_probe_net,lttng_probe_vmscan,lttng_probe_x86_irq_vectors,lttng_probe_power,lttng_probe_rcu,lttng_probe_module,lttng_ring_buffer_client_mmap_overwrite,lttng_probe_statedump,lttng_ring_buffer_client_discard,lttng_probe_printk,lttng_probe_x86_exceptions,lttng_probe_ext4,lttng_probe_sock,lttng_counter_client_percpu_64_modular,lttng_probe_irq,lttng_ring_buffer_client_mmap_discard,lttng_probe_kvm,lttng_probe_timer,lttng_probe_btrfs,lttng_ring_buffer_event_notifier_client,lttng_counter_client_percpu_32_modular,lttng_probe_workqueue,lttng_probe_jbd2,lttng_probe_i2c,lttng_probe_signal,lttng_probe_skb,lttng_probe_block,lttng_probe_napi,lttng_ring_buffer_metadata_client,lttng_probe_kmem,lttng_ring_buffer_metadata_mmap_client,lttng_ring_buffer_client_overwrite,lttng_probe_sunrpc
> lttng_statedump        49152  1 lttng_tracer
> lttng_wrapper          16384  7
> lttng_statedump,lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_probe_kmem,lttng_ring_buffer_client_overwrite
> lttng_clock            16384  5
> lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_ring_buffer_client_overwrite
> lttng_lib_ring_buffer    61440  8
> lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_ring_buffer_event_notifier_client,lttng_ring_buffer_metadata_client,lttng_ring_buffer_metadata_mmap_client,lttng_ring_buffer_client_overwrite
>
> Thanks
> Dirk
>
> On Thu, Nov 7, 2024 at 3:29 PM Mathieu Desnoyers
> <mathieu.desnoyers at efficios.com> wrote:
> >
> > On 2024-11-07 03:03, Dirk Eibach via lttng-dev wrote:
> > > Hi,
> > >
> > > sorry, Outlook is messing up my plaintext mails, so I am switching to
> > > my gmail account.
> > >
> > >> lttng-modules exposes its control ABI through the virtual file /proc/lttng.
> > >>
> > >> Does you container have access to that file ?
> > >
> > > The container is running in privileged mode.
> > > root at 9348eb19e6f4:/# ls -l /proc/lttng
> > > -rw------- 1 root root 0 Nov  6 16:05 /proc/lttng
> > >
> > >> Note that if you give access to that file within the container, then the container
> > >> can observe detailed information about kernel internals (including related to
> > >> other containers).
> > >
> > > This is a development-only system, so the security implications are acceptable.
> > >
> > > I have observed one more thing. When I start the session daemon with
> > > the -vvv option I get:
> > > root at 9348eb19e6f4:/# lttng list --kernel
> > > DBG1 - 07:31:23.811796947 [Client management]: Wait for client
> > > response (in thread_manage_clients() at client.c:2576)
> > > DBG1 - 07:31:23.811867247 [Client management]: Receiving data from
> > > client ... (in thread_manage_clients() at client.c:2604)
> > > DBG1 - 07:31:23.811945247 [Client management]: Processing client
> > > command 'LTTNG_LIST_TRACEPOINTS' (14) (in process_client_msg() at
> > > client.c:1016)
> > > Error: Failed to load kmod library resources
> > > Warning: No kernel tracer available
> >
> > I suspect that you have two choices here:
> >
> > 1) You modprobe all LTTng modules from your top level namespace (host
> >     system) before starting lttng-sessiond within the container, or
> >
> > 2) You ensure that the container has the ability to load kernel
> >     modules, and you make sure the lttng-modules .ko are available
> >     in the container's under /lib/modules/$(uname -r)/ directory.
> >     (and run depmod -a), then launch lttng-sessiond as root.
> >
> > Thanks,
> >
> > Mathieu
> >
> > > DBG1 - 07:31:23.812076447 [Client management]: Missing llm header,
> > > creating one. (in process_client_msg() at client.c:2381)
> > > DBG1 - 07:31:23.812085547 [Client management]: Sending response (size:
> > > 24, retcode: Kernel tracer not available (33)) (in
> > > thread_manage_clients() at client.c:2683)
> > > DBG1 - 07:31:23.812156547 [Client management]: Accepting client
> > > command ... (in thread_manage_clients() at client.c:2534)
> > > Error: Unable to list kernel events: Kernel tracer not available
> > >
> > > Regards
> > > Dirk
> >
> > --
> > Mathieu Desnoyers
> > EfficiOS Inc.
> > https://www.efficios.com
> >