[lttng-dev] lttng-tracer missing

[Neil Bryan]

Hello Forum,

I am having a few problems enabling LTTng kernel tracing. I am using Yocto on an Alter SOC FPGA platform.
I have added the following to local.conf to bring in the required packages:
<SNIP>
EXTRA_IMAGE_FEATURES = "debug-tweaks tools-debug tools-profile"
PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'

INHIBIT_PACKAGE_STRIP = "1"

KERNEL_CONFIG_TRACEPOINTS="y"
KERNEL_CONFIG_HAVE_ARCH_TRACEHOOK="y"
KERNEL_CONFIG_TRACE_IRQFLAGS="y"
KERNEL_CONFIG_STACKTRACE="y"
KERNEL_CONFIG_FRAME_POINTER="y"
KERNEL_CONFIG_NOP_TRACER="y"
KERNEL_CONFIG_TRACER_MAX_TRACE="y"
KERNEL_CONFIG_EVENT_TRACING="y"
KERNEL_CONFIG_EVENT_POWER_TRACING_DEPRECATED="y"
KERNEL_CONFIG_CONTEXT_SWITCH_TRACER="y"
KERNEL_CONFIG_TRACING="y"
KERNEL_CONFIG_GENERIC_TRACER="y"
KERNEL_CONFIG_FTRACE="y"
KERNEL_CONFIG_FUNCTION_TRACER="y"
KERNEL_CONFIG_FUNCTION_GRAPH_TRACER="y"
KERNEL_CONFIG_IRQSOFF_TRACER="y"
KERNEL_CONFIG_PREEMPT_TRACER="y"
KERNEL_CONFIG_SCHED_TRACER="y"
KERNEL_CONFIG_FTRACE_SYSCALLS="y"
KERNEL_CONFIG_BRANCH_PROFILE_NONE="y"
KERNEL_CONFIG_STACK_TRACER="y"
KERNEL_CONFIG_BLK_DEV_IO_TRACE="y"
KERNEL_CONFIG_KPROBES="y"
KERNEL_CONFIG_KPROBE_EVENT="y"
KERNEL_CONFIG_DYNAMIC_FTRACE="y"
KERNEL_CONFIG_FUNCTION_PROFILER="y"
KERNEL_CONFIG_FTRACE_MCOUNT_RECORD="y"
KERNEL_CONFIG_FTRACE_SELFTEST="y"
KERNEL_CONFIG_FTRACE_STARTUP_TEST="y"
KERNEL_CONFIG_EVENT_TRACE_TEST_SYSCALLS="y"
KERNEL_CONFIG_RING_BUFFER_BENCHMARK="y"
<SNIP>

Following the bitbake build I see these kernel modules in the build output directory:

./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/lib/lib-ring-buffer.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-overwrite.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-irq.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-lttng.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-block.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-sched.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-types.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-metadata-mmap-client.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-relay.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-metadata-client.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-mmap-discard.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-discard.ko
./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-mmap-overwrite.ko

When I try to create an lttng session on the target I see the following (not surprisingly as the lttng-tracer module is not present above):

root at socfpga_cyclone5:/lib/modules/3.7.0# lttng-sessiond -vvv
DEBUG3: Creating LTTng run directory: /var/run/lttng [in create_lttng_rundir() at main.c:4315]
DEBUG2: Kernel consumer err path: /var/run/lttng/kconsumerd/error [in main() at main.c:4543]
DEBUG2: Kernel consumer cmd path: /var/run/lttng/kconsumerd/command [in main() at main.c:4545]
DEBUG1: Client socket path /var/run/lttng/client-lttng-sessiond [in main() at main.c:4592]
DEBUG1: Application socket path /var/run/lttng/apps-lttng-sessiond [in main() at main.c:4593]
DEBUG1: LTTng run directory path: /var/run/lttng [in main() at main.c:4594]
DEBUG2: UST consumer 32 bits err path: /var/run/lttng/ustconsumerd32/error [in main() at main.c:4603]
DEBUG2: UST consumer 32 bits cmd path: /var/run/lttng/ustconsumerd32/command [in main() at main.c:4605]
DEBUG2: UST consumer 64 bits err path: /var/run/lttng/ustconsumerd64/error [in main() at main.c:4614]
DEBUG2: UST consumer 64 bits cmd path: /var/run/lttng/ustconsumerd64/command [in main() at main.c:4616]
DEBUG3: Created hashtable size 4 at 0x37080 of type 1 [in lttng_ht_new() at hashtable.c:96]
DEBUG3: Created hashtable size 4 at 0x37168 of type 1 [in lttng_ht_new() at hashtable.c:96]
DEBUG2: Creating consumer directory: /var/run/lttng/kconsumerd [in set_consumer_sockets() at main.c:4357]
FATAL: Module lttng-tracer not found.
Error: Unable to load module lttng-tracer
DEBUG1: Failed to open /proc/lttng [in init_kernel_tracer() at main.c:1871]
Error: Unable to remove module lttng-tracer
Warning: No kernel tracer available
DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd64 [in set_consumer_sockets() at main.c:4357]
DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd32 [in set_consumer_sockets() at main.c:4357]
DEBUG1: Signal handler set for SIGTERM, SIGPIPE and SIGINT [in set_signal_handler() at main.c:4449]
DEBUG1: All permissions are set [in set_permissions() at main.c:4250]
DEBUG1: epoll set max size is 89794 [in compat_epoll_set_max_size() at compat-epoll.c:224]
DEBUG1: [thread] Manage client started [in thread_manage_clients() at main.c:3794]
DEBUG1: [thread] Dispatch UST command started [in thread_dispatch_ust_registration() at main.c:1324]
DEBUG1: Futex n to 1 prepare done [in futex_nto1_prepare() at futex.c:73]
DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:3826]
DEBUG1: Woken up but nothing in the UST command queue [in thread_dispatch_ust_registration() at main.c:1334]
DEBUG1: Thread manage kernel started [in thread_manage_kernel() at main.c:876]
DEBUG1: [thread] Manage application started [in thread_manage_apps() at main.c:1179]
DEBUG1: [thread] Manage application registration started [in thread_registration_apps() at main.c:1392]
DEBUG1: Notifying applications of session daemon state: 1 [in notify_ust_apps() at main.c:687]
DEBUG1: Updating kernel poll set [in update_kernel_poll() at main.c:748]
DEBUG1: Apps thread polling on 2 fds [in thread_manage_apps() at main.c:1200]
DEBUG1: Thread kernel polling on 2 fds [in thread_manage_kernel() at main.c:905]
DEBUG1: Got the wait shm fd 18 [in get_wait_shm() at shm.c:117]
DEBUG1: Futex wait update active 1 [in futex_wait_update() at futex.c:62]
DEBUG1: Accepting application registration [in thread_registration_apps() at main.c:1423]

DEBUG1: SIGINT caught [in sighandler() at main.c:4404]
DEBUG1: Terminating all threads [in stop_threads() at main.c:412]
DEBUG1: Futex n to 1 wake done [in futex_nto1_wake() at futex.c:100]
DEBUG1: Kernel thread dying [in thread_manage_kernel() at main.c:967]
DEBUG1: Application communication apps thread cleanup complete [in thread_manage_apps() at main.c:1308]
DEBUG1: Futex n to 1 wait done [in futex_nto1_wait() at futex.c:87]
DEBUG1: Notifying applications of session daemon state: 0 [in notify_ust_apps() at main.c:687]
DEBUG1: Dispatch thread dying [in thread_dispatch_ust_registration() at main.c:1374]
DEBUG1: Got the wait shm fd 15 [in get_wait_shm() at shm.c:117]
DEBUG1: Client thread dying [in thread_manage_clients() at main.c:3953]
DEBUG1: Futex wait update active 0 [in futex_wait_update() at futex.c:62]
DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:534]
DEBUG1: UST Registration thread cleanup complete [in thread_registration_apps() at main.c:1551]
DEBUG1: Cleaning up [in cleanup() at main.c:432]
DEBUG1: Removing /var/run/lttng directory [in cleanup() at main.c:434]
DEBUG1: Cleaning up all sessions [in cleanup() at main.c:447]
DEBUG1: Closing all UST sockets [in cleanup() at main.c:462]
DEBUG2: UST app cleaning registered apps hash table [in ust_app_clean_list() at ust-app.c:1509]
DEBUG2: Closing kernel fd [in cleanup() at main.c:468]
DEBUG1: Unloading kernel modules [in cleanup() at main.c:475]
DEBUG1: Modprobe removal successful lttng-probe-timer [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-statedump [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-signal [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-sched [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-kvm [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-irq [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-block [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-types [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-probe-lttng [in modprobe_remove_lttng_data() at modprobe.c:118]
Error: Unable to remove module lttng-ring-buffer-metadata-mmap-client
Error: Unable to remove module lttng-ring-buffer-client-mmap-overwrite
Error: Unable to remove module lttng-ring-buffer-client-mmap-discard
Error: Unable to remove module lttng-ring-buffer-metadata-client
Error: Unable to remove module lttng-ring-buffer-client-overwrite
Error: Unable to remove module lttng-ring-buffer-client-discard
Error: Unable to remove module lttng-lib-ring-buffer
DEBUG1: Modprobe removal successful lttng-kretprobes [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-kprobes [in modprobe_remove_lttng_data() at modprobe.c:118]
DEBUG1: Modprobe removal successful lttng-ftrace [in modprobe_remove_lttng_data() at modprobe.c:118]
Error: Unable to remove module lttng-tracer
DEBUG1: *** assert failed :-) *** ==> Matthew, BEET driven development works! [in cleanup()

I am running an early(ish) version of the kernel (3.7) but this is because the Altera Yocto source package is quite an old vintage.

Any clues as to what am I doing wrong?

Thanks for any help anyone can offer.

Best regards,

Neil.



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lttng.org/pipermail/lttng-dev/attachments/20150217/f196a2de/attachment-0001.html>