Kernel tracing from inside a Docker container

[Dirk Eibach]

Hi Mathieu,

> 1) You modprobe all LTTng modules from your top level namespace (host
>     system) before starting lttng-sessiond within the container, or

since I already did a lttng session on the host/top level namespacer
the modules are already loaded:

root at 9348eb19e6f4:/# lsmod
Module                  Size  Used by
lttng_probe_x86_exceptions    16384  0
lttng_probe_x86_irq_vectors    16384  0
lttng_probe_workqueue    20480  0
lttng_probe_vmscan     32768  0
lttng_probe_udp        16384  0
lttng_probe_timer      28672  0
lttng_probe_sunrpc     20480  0
lttng_probe_statedump    45056  0
lttng_probe_sock       16384  0
lttng_probe_skb        16384  0
lttng_probe_signal     16384  0
lttng_probe_scsi       20480  0
lttng_probe_sched      40960  0
lttng_probe_rcu        16384  0
lttng_probe_printk     16384  0
lttng_probe_power      20480  0
lttng_probe_net        28672  0
lttng_probe_napi       16384  0
lttng_probe_module     20480  0
lttng_probe_kvm        32768  0
lttng_probe_kmem       28672  0
lttng_probe_jbd2       28672  0
lttng_probe_irq        16384  0
lttng_probe_i2c        20480  0
lttng_probe_ext4      155648  0
lttng_probe_compaction    16384  0
lttng_probe_btrfs      65536  0
lttng_probe_block      36864  0
lttng_counter_client_percpu_32_modular    16384  0
lttng_counter_client_percpu_64_modular    16384  1
lttng_counter          16384  2
lttng_counter_client_percpu_64_modular,lttng_counter_client_percpu_32_modular
lttng_ring_buffer_event_notifier_client    16384  2
lttng_ring_buffer_metadata_mmap_client    16384  0
lttng_ring_buffer_client_mmap_overwrite    20480  0
lttng_ring_buffer_client_mmap_discard    20480  0
lttng_ring_buffer_metadata_client    16384  0
lttng_ring_buffer_client_overwrite    20480  0
lttng_ring_buffer_client_discard    20480  0
lttng_tracer         2666496  40
lttng_probe_udp,lttng_probe_scsi,lttng_probe_sched,lttng_probe_compaction,lttng_probe_net,lttng_probe_vmscan,lttng_probe_x86_irq_vectors,lttng_probe_power,lttng_probe_rcu,lttng_probe_module,lttng_ring_buffer_client_mmap_overwrite,lttng_probe_statedump,lttng_ring_buffer_client_discard,lttng_probe_printk,lttng_probe_x86_exceptions,lttng_probe_ext4,lttng_probe_sock,lttng_counter_client_percpu_64_modular,lttng_probe_irq,lttng_ring_buffer_client_mmap_discard,lttng_probe_kvm,lttng_probe_timer,lttng_probe_btrfs,lttng_ring_buffer_event_notifier_client,lttng_counter_client_percpu_32_modular,lttng_probe_workqueue,lttng_probe_jbd2,lttng_probe_i2c,lttng_probe_signal,lttng_probe_skb,lttng_probe_block,lttng_probe_napi,lttng_ring_buffer_metadata_client,lttng_probe_kmem,lttng_ring_buffer_metadata_mmap_client,lttng_ring_buffer_client_overwrite,lttng_probe_sunrpc
lttng_statedump        49152  1 lttng_tracer
lttng_wrapper          16384  7
lttng_statedump,lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_probe_kmem,lttng_ring_buffer_client_overwrite
lttng_clock            16384  5
lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_ring_buffer_client_overwrite
lttng_lib_ring_buffer    61440  8
lttng_ring_buffer_client_mmap_overwrite,lttng_ring_buffer_client_discard,lttng_tracer,lttng_ring_buffer_client_mmap_discard,lttng_ring_buffer_event_notifier_client,lttng_ring_buffer_metadata_client,lttng_ring_buffer_metadata_mmap_client,lttng_ring_buffer_client_overwrite

Thanks
Dirk

On Thu, Nov 7, 2024 at 3:29 PM Mathieu Desnoyers
<mathieu.desnoyers at efficios.com> wrote:
>
> On 2024-11-07 03:03, Dirk Eibach via lttng-dev wrote:
> > Hi,
> >
> > sorry, Outlook is messing up my plaintext mails, so I am switching to
> > my gmail account.
> >
> >> lttng-modules exposes its control ABI through the virtual file /proc/lttng.
> >>
> >> Does you container have access to that file ?
> >
> > The container is running in privileged mode.
> > root at 9348eb19e6f4:/# ls -l /proc/lttng
> > -rw------- 1 root root 0 Nov  6 16:05 /proc/lttng
> >
> >> Note that if you give access to that file within the container, then the container
> >> can observe detailed information about kernel internals (including related to
> >> other containers).
> >
> > This is a development-only system, so the security implications are acceptable.
> >
> > I have observed one more thing. When I start the session daemon with
> > the -vvv option I get:
> > root at 9348eb19e6f4:/# lttng list --kernel
> > DBG1 - 07:31:23.811796947 [Client management]: Wait for client
> > response (in thread_manage_clients() at client.c:2576)
> > DBG1 - 07:31:23.811867247 [Client management]: Receiving data from
> > client ... (in thread_manage_clients() at client.c:2604)
> > DBG1 - 07:31:23.811945247 [Client management]: Processing client
> > command 'LTTNG_LIST_TRACEPOINTS' (14) (in process_client_msg() at
> > client.c:1016)
> > Error: Failed to load kmod library resources
> > Warning: No kernel tracer available
>
> I suspect that you have two choices here:
>
> 1) You modprobe all LTTng modules from your top level namespace (host
>     system) before starting lttng-sessiond within the container, or
>
> 2) You ensure that the container has the ability to load kernel
>     modules, and you make sure the lttng-modules .ko are available
>     in the container's under /lib/modules/$(uname -r)/ directory.
>     (and run depmod -a), then launch lttng-sessiond as root.
>
> Thanks,
>
> Mathieu
>
> > DBG1 - 07:31:23.812076447 [Client management]: Missing llm header,
> > creating one. (in process_client_msg() at client.c:2381)
> > DBG1 - 07:31:23.812085547 [Client management]: Sending response (size:
> > 24, retcode: Kernel tracer not available (33)) (in
> > thread_manage_clients() at client.c:2683)
> > DBG1 - 07:31:23.812156547 [Client management]: Accepting client
> > command ... (in thread_manage_clients() at client.c:2534)
> > Error: Unable to list kernel events: Kernel tracer not available
> >
> > Regards
> > Dirk
>
> --
> Mathieu Desnoyers
> EfficiOS Inc.
> https://www.efficios.com
>