<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:2.0cm 42.5pt 2.0cm 3.0cm;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello Forum,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am having a few problems enabling LTTng kernel tracing. I am using Yocto on an Alter SOC FPGA platform.<o:p></o:p></p>
<p class="MsoNormal">I have added the following to local.conf to bring in the required packages:<o:p></o:p></p>
<p class="MsoNormal"><SNIP><o:p></o:p></p>
<p class="MsoNormal">EXTRA_IMAGE_FEATURES = "debug-tweaks tools-debug tools-profile"<o:p></o:p></p>
<p class="MsoNormal">PACKAGE_DEBUG_SPLIT_STYLE = 'debug-file-directory'<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">INHIBIT_PACKAGE_STRIP = "1"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_TRACEPOINTS="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_HAVE_ARCH_TRACEHOOK="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_TRACE_IRQFLAGS="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_STACKTRACE="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_FRAME_POINTER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_NOP_TRACER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_TRACER_MAX_TRACE="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_EVENT_TRACING="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_EVENT_POWER_TRACING_DEPRECATED="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_CONTEXT_SWITCH_TRACER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_TRACING="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_GENERIC_TRACER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_FTRACE="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_FUNCTION_TRACER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_FUNCTION_GRAPH_TRACER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_IRQSOFF_TRACER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_PREEMPT_TRACER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_SCHED_TRACER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_FTRACE_SYSCALLS="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_BRANCH_PROFILE_NONE="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_STACK_TRACER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_BLK_DEV_IO_TRACE="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_KPROBES="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_KPROBE_EVENT="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_DYNAMIC_FTRACE="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_FUNCTION_PROFILER="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_FTRACE_MCOUNT_RECORD="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_FTRACE_SELFTEST="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_FTRACE_STARTUP_TEST="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_EVENT_TRACE_TEST_SYSCALLS="y"<o:p></o:p></p>
<p class="MsoNormal">KERNEL_CONFIG_RING_BUFFER_BENCHMARK="y"<o:p></o:p></p>
<p class="MsoNormal"><SNIP><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Following the bitbake build I see these kernel modules in the build output directory:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/lib/lib-ring-buffer.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-overwrite.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-irq.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-lttng.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-block.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-probe-sched.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/probes/lttng-types.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-metadata-mmap-client.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-relay.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-metadata-client.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-mmap-discard.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-discard.ko<o:p></o:p></p>
<p class="MsoNormal">./altera-image-1.0-r0/rootfs/lib/modules/3.7.0/kernel/lttng-modules/ltt-ring-buffer-client-mmap-overwrite.ko<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When I try to create an lttng session on the target I see the following (not surprisingly as the lttng-tracer module is not present above):<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">root@socfpga_cyclone5:/lib/modules/3.7.0# lttng-sessiond -vvv<o:p></o:p></p>
<p class="MsoNormal">DEBUG3: Creating LTTng run directory: /var/run/lttng [in create_lttng_rundir() at main.c:4315]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: Kernel consumer err path: /var/run/lttng/kconsumerd/error [in main() at main.c:4543]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: Kernel consumer cmd path: /var/run/lttng/kconsumerd/command [in main() at main.c:4545]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Client socket path /var/run/lttng/client-lttng-sessiond [in main() at main.c:4592]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Application socket path /var/run/lttng/apps-lttng-sessiond [in main() at main.c:4593]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: LTTng run directory path: /var/run/lttng [in main() at main.c:4594]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: UST consumer 32 bits err path: /var/run/lttng/ustconsumerd32/error [in main() at main.c:4603]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: UST consumer 32 bits cmd path: /var/run/lttng/ustconsumerd32/command [in main() at main.c:4605]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: UST consumer 64 bits err path: /var/run/lttng/ustconsumerd64/error [in main() at main.c:4614]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: UST consumer 64 bits cmd path: /var/run/lttng/ustconsumerd64/command [in main() at main.c:4616]<o:p></o:p></p>
<p class="MsoNormal">DEBUG3: Created hashtable size 4 at 0x37080 of type 1 [in lttng_ht_new() at hashtable.c:96]<o:p></o:p></p>
<p class="MsoNormal">DEBUG3: Created hashtable size 4 at 0x37168 of type 1 [in lttng_ht_new() at hashtable.c:96]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: Creating consumer directory: /var/run/lttng/kconsumerd [in set_consumer_sockets() at main.c:4357]<o:p></o:p></p>
<p class="MsoNormal">FATAL: Module lttng-tracer not found.<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to load module lttng-tracer<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Failed to open /proc/lttng [in init_kernel_tracer() at main.c:1871]<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to remove module lttng-tracer<o:p></o:p></p>
<p class="MsoNormal">Warning: No kernel tracer available<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd64 [in set_consumer_sockets() at main.c:4357]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: Creating consumer directory: /var/run/lttng/ustconsumerd32 [in set_consumer_sockets() at main.c:4357]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Signal handler set for SIGTERM, SIGPIPE and SIGINT [in set_signal_handler() at main.c:4449]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: All permissions are set [in set_permissions() at main.c:4250]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: epoll set max size is 89794 [in compat_epoll_set_max_size() at compat-epoll.c:224]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: [thread] Manage client started [in thread_manage_clients() at main.c:3794]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: [thread] Dispatch UST command started [in thread_dispatch_ust_registration() at main.c:1324]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Futex n to 1 prepare done [in futex_nto1_prepare() at futex.c:73]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Accepting client command ... [in thread_manage_clients() at main.c:3826]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Woken up but nothing in the UST command queue [in thread_dispatch_ust_registration() at main.c:1334]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Thread manage kernel started [in thread_manage_kernel() at main.c:876]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: [thread] Manage application started [in thread_manage_apps() at main.c:1179]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: [thread] Manage application registration started [in thread_registration_apps() at main.c:1392]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Notifying applications of session daemon state: 1 [in notify_ust_apps() at main.c:687]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Updating kernel poll set [in update_kernel_poll() at main.c:748]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Apps thread polling on 2 fds [in thread_manage_apps() at main.c:1200]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Thread kernel polling on 2 fds [in thread_manage_kernel() at main.c:905]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Got the wait shm fd 18 [in get_wait_shm() at shm.c:117]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Futex wait update active 1 [in futex_wait_update() at futex.c:62]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Accepting application registration [in thread_registration_apps() at main.c:1423]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">DEBUG1: SIGINT caught [in sighandler() at main.c:4404]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Terminating all threads [in stop_threads() at main.c:412]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Futex n to 1 wake done [in futex_nto1_wake() at futex.c:100]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Kernel thread dying [in thread_manage_kernel() at main.c:967]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Application communication apps thread cleanup complete [in thread_manage_apps() at main.c:1308]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Futex n to 1 wait done [in futex_nto1_wait() at futex.c:87]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Notifying applications of session daemon state: 0 [in notify_ust_apps() at main.c:687]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Dispatch thread dying [in thread_dispatch_ust_registration() at main.c:1374]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Got the wait shm fd 15 [in get_wait_shm() at shm.c:117]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Client thread dying [in thread_manage_clients() at main.c:3953]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Futex wait update active 0 [in futex_wait_update() at futex.c:62]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Clean command context structure [in clean_command_ctx() at main.c:534]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: UST Registration thread cleanup complete [in thread_registration_apps() at main.c:1551]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Cleaning up [in cleanup() at main.c:432]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Removing /var/run/lttng directory [in cleanup() at main.c:434]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Cleaning up all sessions [in cleanup() at main.c:447]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Closing all UST sockets [in cleanup() at main.c:462]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: UST app cleaning registered apps hash table [in ust_app_clean_list() at ust-app.c:1509]<o:p></o:p></p>
<p class="MsoNormal">DEBUG2: Closing kernel fd [in cleanup() at main.c:468]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Unloading kernel modules [in cleanup() at main.c:475]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-probe-timer [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-probe-statedump [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-probe-signal [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-probe-sched [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-probe-kvm [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-probe-irq [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-probe-block [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-types [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-probe-lttng [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to remove module lttng-ring-buffer-metadata-mmap-client<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to remove module lttng-ring-buffer-client-mmap-overwrite<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to remove module lttng-ring-buffer-client-mmap-discard<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to remove module lttng-ring-buffer-metadata-client<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to remove module lttng-ring-buffer-client-overwrite<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to remove module lttng-ring-buffer-client-discard<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to remove module lttng-lib-ring-buffer<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-kretprobes [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-kprobes [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: Modprobe removal successful lttng-ftrace [in modprobe_remove_lttng_data() at modprobe.c:118]<o:p></o:p></p>
<p class="MsoNormal">Error: Unable to remove module lttng-tracer<o:p></o:p></p>
<p class="MsoNormal">DEBUG1: *** assert failed :-) *** ==> Matthew, BEET driven development works! [in cleanup()<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am running an early(ish) version of the kernel (3.7) but this is because the Altera Yocto source package is quite an old vintage.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any clues as to what am I doing wrong?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks for any help anyone can offer.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Best regards,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Neil.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>