[ltt-dev] LTTng help

Wed Apr 6 11:35:47 EDT 2011

Hi,
I'm a LTTng newbie and would appreciate some hand holding to get me 
started the right way...
I'm having trouble getting traces.  I'm trying to debug a bottleneck in 
a multi-threaded app and have been
struggling the last few days to get things started.
I initially got a trace but all threads were named the same so I started 
to instrument the code
with thread branding calls.   I must have messed something up in the 
process since the lttv-gui now crashes  (bottom of this email)
when loading the trace dir.  When I tried some of the sample programs 
from the ltt-usertrace-0.12  I did not get
any useful trace info...  Again, I'm new to this so please excuse 
ignorant comments.
So basically I'm looking for help in identifying anything that I might 
have missed in the setup and any possible
clues as to why the trace info is not getting generated.  Thanks in advance.
-Mike

Here's the kernel menuconfig:

I did not do anything with genevent (was I supposed to?).
Are there any other patches for user space tracing ?

I'm cross compiling, targetting the Linux kernel  2.6.16.9 (using a 
target specific toolchain)
I downloaded the following components:

patch-2.6.16-lttng-0.5.69.tar.bz2.
ltt-control-0.10-12062006
ltt-usertrace-0.12
userspace-rcu-0.5.4
ust-0.12
LinuxTraceToolkitViewer-0.8.47-12062006

Is this all or am I missing anything?

I took the contents of ltt-control-installed and copied them onto the 
target into respective dirs.

For the sake of completeness, the contents are the below.

ltt-control-installed/
ltt-control-installed/usr/
ltt-control-installed/usr/bin/
ltt-control-installed/usr/bin/usttrace
ltt-control-installed/usr/bin/ustctl
ltt-control-installed/usr/bin/ust-consumerd
ltt-control-installed/usr/bin/lttctl
ltt-control-installed/usr/bin/lttd
ltt-control-installed/usr/share/
ltt-control-installed/usr/share/man/
ltt-control-installed/usr/share/man/man1/
ltt-control-installed/usr/share/man/man1/ust-consumerd.1
ltt-control-installed/usr/share/man/man1/usttrace.1
ltt-control-installed/usr/share/man/man1/ustctl.1
ltt-control-installed/usr/share/info/
ltt-control-installed/usr/share/info/ust.info
ltt-control-installed/usr/share/ltt-control/
ltt-control-installed/usr/share/ltt-control/facilities/
ltt-control-installed/usr/share/ltt-control/facilities/stack_arch_i386.xml
ltt-control-installed/usr/share/ltt-control/facilities/fs_data.xml
ltt-control-installed/usr/share/ltt-control/facilities/user_generic.xml
ltt-control-installed/usr/share/ltt-control/facilities/locking.xml
ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_ppc.xml
ltt-control-installed/usr/share/ltt-control/facilities/network_ip_interface.xml
ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_powerpc.xml
ltt-control-installed/usr/share/ltt-control/facilities/socket.xml
ltt-control-installed/usr/share/ltt-control/facilities/memory.xml
ltt-control-installed/usr/share/ltt-control/facilities/process.xml
ltt-control-installed/usr/share/ltt-control/facilities/statedump.xml
ltt-control-installed/usr/share/ltt-control/facilities/ipc.xml
ltt-control-installed/usr/share/ltt-control/facilities/network.xml
ltt-control-installed/usr/share/ltt-control/facilities/kernel.xml
ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_c2.xml
ltt-control-installed/usr/share/ltt-control/facilities/core.xml
ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_arm.xml
ltt-control-installed/usr/share/ltt-control/facilities/fs.xml
ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_i386.xml
ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_mips.xml
ltt-control-installed/usr/share/ltt-control/facilities/timer.xml
ltt-control-installed/usr/include/
ltt-control-installed/usr/include/urcu-defer-static.h
ltt-control-installed/usr/include/urcu-pointer.h
ltt-control-installed/usr/include/ltt/
ltt-control-installed/usr/include/ltt/ltt-usertrace-fast.h
ltt-control-installed/usr/include/ltt/ltt-usertrace.h
ltt-control-installed/usr/include/ltt/atomic-ppc.h
ltt-control-installed/usr/include/ltt/ltt-facility-id-user_generic.h
ltt-control-installed/usr/include/ltt/ppc_asm-ppc.h
ltt-control-installed/usr/include/ltt/system-ppc64.h
ltt-control-installed/usr/include/ltt/ltt-facility-user_generic.h
ltt-control-installed/usr/include/ltt/system-ppc.h
ltt-control-installed/usr/include/ltt/timex-ppc.h
ltt-control-installed/usr/include/ltt/atomic-ppc64.h
ltt-control-installed/usr/include/ltt/ltt-facility-custom-user_generic.h
ltt-control-installed/usr/include/ltt/ltt-usertrace-ppc.h
ltt-control-installed/usr/include/urcu-bp-static.h
ltt-control-installed/usr/include/ust/
ltt-control-installed/usr/include/ust/marker.h
ltt-control-installed/usr/include/ust/type-serializer.h
ltt-control-installed/usr/include/ust/immediate.h
ltt-control-installed/usr/include/ust/kcompat/
ltt-control-installed/usr/include/ust/kcompat/jhash.h
ltt-control-installed/usr/include/ust/kcompat/types.h
ltt-control-installed/usr/include/ust/kcompat/stringify.h
ltt-control-installed/usr/include/ust/kcompat/kcompat.h
ltt-control-installed/usr/include/ust/kcompat/compiler.h
ltt-control-installed/usr/include/ust/kcompat/simple.h
ltt-control-installed/usr/include/ust/define_trace.h
ltt-control-installed/usr/include/ust/probe.h
ltt-control-installed/usr/include/ust/clock.h
ltt-control-installed/usr/include/ust/ust_trace.h
ltt-control-installed/usr/include/ust/processor.h
ltt-control-installed/usr/include/ust/core.h
ltt-control-installed/usr/include/ust/ust.h
ltt-control-installed/usr/include/ust/tracectl.h
ltt-control-installed/usr/include/ust/tracepoint.h
ltt-control-installed/usr/include/ust/ustctl.h
ltt-control-installed/usr/include/ust/ustconsumer.h
ltt-control-installed/usr/include/urcu-bp.h
ltt-control-installed/usr/include/urcu-pointer-static.h
ltt-control-installed/usr/include/urcu-defer.h
ltt-control-installed/usr/include/urcu-qsbr.h
ltt-control-installed/usr/include/urcu/
ltt-control-installed/usr/include/urcu/urcu-futex.h
ltt-control-installed/usr/include/urcu/rculist.h
ltt-control-installed/usr/include/urcu/rculfqueue.h
ltt-control-installed/usr/include/urcu/uatomic_arch.h
ltt-control-installed/usr/include/urcu/hlist.h
ltt-control-installed/usr/include/urcu/wfstack-static.h
ltt-control-installed/usr/include/urcu/urcu_ref.h
ltt-control-installed/usr/include/urcu/compiler.h
ltt-control-installed/usr/include/urcu/rcuhlist.h
ltt-control-installed/usr/include/urcu/rculfstack-static.h
ltt-control-installed/usr/include/urcu/wfstack.h
ltt-control-installed/usr/include/urcu/arch_generic.h
ltt-control-installed/usr/include/urcu/arch.h
ltt-control-installed/usr/include/urcu/config.h
ltt-control-installed/usr/include/urcu/system.h
ltt-control-installed/usr/include/urcu/wfqueue-static.h
ltt-control-installed/usr/include/urcu/wfqueue.h
ltt-control-installed/usr/include/urcu/uatomic_generic.h
ltt-control-installed/usr/include/urcu/list.h
ltt-control-installed/usr/include/urcu/rculfstack.h
ltt-control-installed/usr/include/urcu/rculfqueue-static.h
ltt-control-installed/usr/include/urcu-static.h
ltt-control-installed/usr/include/urcu.h
ltt-control-installed/usr/include/liblttctl/
ltt-control-installed/usr/include/liblttctl/lttctl.h
ltt-control-installed/usr/include/ltt-facility-loader-user_generic.h
ltt-control-installed/usr/include/urcu-qsbr-static.h
ltt-control-installed/usr/lib/
ltt-control-installed/usr/lib/libustfork.so
ltt-control-installed/usr/lib/libust.la
ltt-control-installed/usr/lib/libust.so.0
ltt-control-installed/usr/lib/libustctl.so.0
ltt-control-installed/usr/lib/libltt-usertrace-fast.so.0
ltt-control-installed/usr/lib/libtap.so
ltt-control-installed/usr/lib/librculfqueue.a
ltt-control-installed/usr/lib/liburcu-bp.a
ltt-control-installed/usr/lib/libustinstr-malloc.so.0.0.0
ltt-control-installed/usr/lib/libustfork.a
ltt-control-installed/usr/lib/liburcu-signal.a
ltt-control-installed/usr/lib/libustinstr-malloc.la
ltt-control-installed/usr/lib/libwfqueue.la
ltt-control-installed/usr/lib/liblttctl.a
ltt-control-installed/usr/lib/libustconsumer.so
ltt-control-installed/usr/lib/libust.so
ltt-control-installed/usr/lib/libltt-instrument-functions.a
ltt-control-installed/usr/lib/libustfork.so.0.0.0
ltt-control-installed/usr/lib/libtap.la
ltt-control-installed/usr/lib/liburcu-bp.la
ltt-control-installed/usr/lib/liblttctl.so.0.0.0
ltt-control-installed/usr/lib/librculfqueue.la
ltt-control-installed/usr/lib/liblttctl.so.0
ltt-control-installed/usr/lib/libustctl.la
ltt-control-installed/usr/lib/libtap.a
ltt-control-installed/usr/lib/libustctl.so
ltt-control-installed/usr/lib/libtap.so.0
ltt-control-installed/usr/lib/libltt-instrument-functions.so
ltt-control-installed/usr/lib/libwfstack.a
ltt-control-installed/usr/lib/libustconsumer.a
ltt-control-installed/usr/lib/librculfstack.a
ltt-control-installed/usr/lib/libustfork.so.0
ltt-control-installed/usr/lib/libltt-usertrace-fast.so
ltt-control-installed/usr/lib/libltt-usertrace-fast.a
ltt-control-installed/usr/lib/libustctl.so.0.0.0
ltt-control-installed/usr/lib/libustconsumer.la
ltt-control-installed/usr/lib/liblttctl.la
ltt-control-installed/usr/lib/libustfork.la
ltt-control-installed/usr/lib/libwfqueue.a
ltt-control-installed/usr/lib/liburcu-defer.la
ltt-control-installed/usr/lib/libust-initializer.o
ltt-control-installed/usr/lib/liburcu-qsbr.a
ltt-control-installed/usr/lib/liburcu-mb.a
ltt-control-installed/usr/lib/liburcu-defer.a
ltt-control-installed/usr/lib/libustinstr-malloc.so.0
ltt-control-installed/usr/lib/libtap.so.0.0.0
ltt-control-installed/usr/lib/liburcu-qsbr.la
ltt-control-installed/usr/lib/librculfstack.la
ltt-control-installed/usr/lib/libustinstr-malloc.a
ltt-control-installed/usr/lib/libustctl.a
ltt-control-installed/usr/lib/libustinstr-malloc.so
ltt-control-installed/usr/lib/liblttctl.so
ltt-control-installed/usr/lib/liburcu-mb.la
ltt-control-installed/usr/lib/liburcu.a
ltt-control-installed/usr/lib/liburcu-signal.la
ltt-control-installed/usr/lib/libust.so.0.0.0
ltt-control-installed/usr/lib/liburcu.la
ltt-control-installed/usr/lib/libltt-instrument-functions.so.0
ltt-control-installed/usr/lib/libustconsumer.so.0
ltt-control-installed/usr/lib/libwfstack.la
ltt-control-installed/usr/lib/libust.a
ltt-control-installed/usr/lib/libustconsumer.so.0.0.0
ltt-control-installed/usr/src/
ltt-control-installed/usr/src/sample-thread-brand.c
ltt-control-installed/usr/src/sample-instrument-fct.c
ltt-control-installed/usr/src/ltt-usertrace-fast.c
ltt-control-installed/usr/src/sample-printf.c
ltt-control-installed/usr/src/sample-highspeed.c
ltt-control-installed/usr/src/ltt-instrument-functions.c
ltt-control-installed/usr/src/sample-thread-slow.c
ltt-control-installed/usr/src/sample-thread-fast.c
ltt-control-installed/usr/src/sample.c
ltt-control-installed/usr/src/ltt-facility-loader-user_generic.c
ltt-control-installed/usr/test/
ltt-control-installed/usr/test/sample-instrument-fct
ltt-control-installed/usr/test/sample-thread-fast
ltt-control-installed/usr/test/sample-thread-slow
ltt-control-installed/usr/test/sample-thread-brand
ltt-control-installed/usr/test/sample-highspeed
ltt-control-installed/usr/test/sample-printf

First of all, here's how I started things off:
(bash)
mkdir /mnt/debugfs
echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab
mount /mnt/debugfs
mkdir /mnt/relayfs
mount -t relayfs relayfs /mnt/relayfs
export LTT_FACILITIES=/usr/share/ltt-control/facilities
export LTT_DAEMON=/usr/bin/lttd

To start the tracer:
lttctl -n trace4  -c -d -t /var/log/trace4 -l /mnt/relayfs/ltt
bash-2.04# lttctl -n trace4 -c -d -t /var/log/trace4 -l /mnt/relayfs/ltt/
Linux Trace Toolkit Trace Control

Controlling trace : trace4

Linux Trace Toolkit Trace Daemon

Reading from relayfs directory : /mnt/relayfs/ltt//trace4
Writing to trace directory : /var/log/trace4

signal 17 received
Creating supplementary trace files
Appending facility file stack_arch_i386.xml
Appending facility file locking.xml
Appending facility file process.xml
Appending facility file fs_data.xml
Appending facility file kernel_arch_mips.xml
Appending facility file kernel_arch_ppc.xml
Appending facility file timer.xml
Appending facility file ipc.xml
Appending facility file memory.xml
Appending facility file fs.xml
Appending facility file kernel_arch_c2.xml
Appending facility file kernel.xml
Appending facility file network.xml
Appending facility file user_generic.xml
Appending facility file kernel_arch_i386.xml
Appending facility file core.xml
Appending facility file socket.xml
Appending facility file kernel_arch_powerpc.xml
Appending facility file kernel_arch_arm.xml
Appending facility file network_ip_interface.xml
Appending facility file statedump.xml

<invoke my app>

To stop tracing:
bash-2.04# lttctl -n trace4 -R
Linux Trace Toolkit Trace Control

Controlling trace : trace4

here's lttv-gui crashing when opening the trace

[FedoraBox traces]$ lttv-gui
The input data file /work/traces/trace4/control/facilities_0 does not 
contain a trace
The input data file /work/traces/trace4/control/interrupts_0 does not 
contain a trace
The input data file /work/traces/trace4/control/network_0 does not 
contain a trace
The input data file /work/traces/trace4/control/processes_0 does not 
contain a trace
The input data file /work/traces/trace4/control/modules_0 does not 
contain a trace

** ERROR **: Trace /work/traces/trace4 has no facility tracefile
aborting...
/usr/local/bin/lttv-gui: line 10: 18614 Aborted                 (core 
dumped) $LTTV_CMD.real -m guievents -m guifilter -m guicontrolflow -m 
guistatistics -m guitracecontrol $*
[FedoraBox traces]$

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20110406/5162ca4e/attachment-0003.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: clip_image002.jpg
Type: image/jpeg
Size: 27216 bytes
Desc: not available
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20110406/5162ca4e/attachment-0003.jpg>