[ltt-dev] LTTng help

Wed Apr 6 14:22:18 EDT 2011

OK, it looks like I messed up the kernel installation on the target. 
After rebuilding the kernel and installing correctly  I finally got the 
trace with branded threads.
-Mike


On 4/6/2011 11:35 AM, Mike Walczak wrote:
> Hi,
> I'm a LTTng newbie and would appreciate some hand holding to get me 
> started the right way...
> I'm having trouble getting traces.  I'm trying to debug a bottleneck 
> in a multi-threaded app and have been
> struggling the last few days to get things started.
> I initially got a trace but all threads were named the same so I 
> started to instrument the code
> with thread branding calls.   I must have messed something up in the 
> process since the lttv-gui now crashes  (bottom of this email)
> when loading the trace dir.  When I tried some of the sample programs 
> from the ltt-usertrace-0.12  I did not get
> any useful trace info...  Again, I'm new to this so please excuse 
> ignorant comments.
> So basically I'm looking for help in identifying anything that I might 
> have missed in the setup and any possible
> clues as to why the trace info is not getting generated.  Thanks in 
> advance.
> -Mike
>
> Here's the kernel menuconfig:
>
>
>
>
> I did not do anything with genevent (was I supposed to?).
> Are there any other patches for user space tracing ?
>
>
>
>
> I'm cross compiling, targetting the Linux kernel  2.6.16.9 (using a 
> target specific toolchain)
> I downloaded the following components:
>
> patch-2.6.16-lttng-0.5.69.tar.bz2.
> ltt-control-0.10-12062006
> ltt-usertrace-0.12
> userspace-rcu-0.5.4
> ust-0.12
> LinuxTraceToolkitViewer-0.8.47-12062006
>
> Is this all or am I missing anything?
>
>
> I took the contents of ltt-control-installed and copied them onto the 
> target into respective dirs.
>
> For the sake of completeness, the contents are the below.
>
>
>
> ltt-control-installed/
> ltt-control-installed/usr/
> ltt-control-installed/usr/bin/
> ltt-control-installed/usr/bin/usttrace
> ltt-control-installed/usr/bin/ustctl
> ltt-control-installed/usr/bin/ust-consumerd
> ltt-control-installed/usr/bin/lttctl
> ltt-control-installed/usr/bin/lttd
> ltt-control-installed/usr/share/
> ltt-control-installed/usr/share/man/
> ltt-control-installed/usr/share/man/man1/
> ltt-control-installed/usr/share/man/man1/ust-consumerd.1
> ltt-control-installed/usr/share/man/man1/usttrace.1
> ltt-control-installed/usr/share/man/man1/ustctl.1
> ltt-control-installed/usr/share/info/
> ltt-control-installed/usr/share/info/ust.info
> ltt-control-installed/usr/share/ltt-control/
> ltt-control-installed/usr/share/ltt-control/facilities/
> ltt-control-installed/usr/share/ltt-control/facilities/stack_arch_i386.xml
> ltt-control-installed/usr/share/ltt-control/facilities/fs_data.xml
> ltt-control-installed/usr/share/ltt-control/facilities/user_generic.xml
> ltt-control-installed/usr/share/ltt-control/facilities/locking.xml
> ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_ppc.xml
> ltt-control-installed/usr/share/ltt-control/facilities/network_ip_interface.xml
> ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_powerpc.xml
> ltt-control-installed/usr/share/ltt-control/facilities/socket.xml
> ltt-control-installed/usr/share/ltt-control/facilities/memory.xml
> ltt-control-installed/usr/share/ltt-control/facilities/process.xml
> ltt-control-installed/usr/share/ltt-control/facilities/statedump.xml
> ltt-control-installed/usr/share/ltt-control/facilities/ipc.xml
> ltt-control-installed/usr/share/ltt-control/facilities/network.xml
> ltt-control-installed/usr/share/ltt-control/facilities/kernel.xml
> ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_c2.xml
> ltt-control-installed/usr/share/ltt-control/facilities/core.xml
> ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_arm.xml
> ltt-control-installed/usr/share/ltt-control/facilities/fs.xml
> ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_i386.xml
> ltt-control-installed/usr/share/ltt-control/facilities/kernel_arch_mips.xml
> ltt-control-installed/usr/share/ltt-control/facilities/timer.xml
> ltt-control-installed/usr/include/
> ltt-control-installed/usr/include/urcu-defer-static.h
> ltt-control-installed/usr/include/urcu-pointer.h
> ltt-control-installed/usr/include/ltt/
> ltt-control-installed/usr/include/ltt/ltt-usertrace-fast.h
> ltt-control-installed/usr/include/ltt/ltt-usertrace.h
> ltt-control-installed/usr/include/ltt/atomic-ppc.h
> ltt-control-installed/usr/include/ltt/ltt-facility-id-user_generic.h
> ltt-control-installed/usr/include/ltt/ppc_asm-ppc.h
> ltt-control-installed/usr/include/ltt/system-ppc64.h
> ltt-control-installed/usr/include/ltt/ltt-facility-user_generic.h
> ltt-control-installed/usr/include/ltt/system-ppc.h
> ltt-control-installed/usr/include/ltt/timex-ppc.h
> ltt-control-installed/usr/include/ltt/atomic-ppc64.h
> ltt-control-installed/usr/include/ltt/ltt-facility-custom-user_generic.h
> ltt-control-installed/usr/include/ltt/ltt-usertrace-ppc.h
> ltt-control-installed/usr/include/urcu-bp-static.h
> ltt-control-installed/usr/include/ust/
> ltt-control-installed/usr/include/ust/marker.h
> ltt-control-installed/usr/include/ust/type-serializer.h
> ltt-control-installed/usr/include/ust/immediate.h
> ltt-control-installed/usr/include/ust/kcompat/
> ltt-control-installed/usr/include/ust/kcompat/jhash.h
> ltt-control-installed/usr/include/ust/kcompat/types.h
> ltt-control-installed/usr/include/ust/kcompat/stringify.h
> ltt-control-installed/usr/include/ust/kcompat/kcompat.h
> ltt-control-installed/usr/include/ust/kcompat/compiler.h
> ltt-control-installed/usr/include/ust/kcompat/simple.h
> ltt-control-installed/usr/include/ust/define_trace.h
> ltt-control-installed/usr/include/ust/probe.h
> ltt-control-installed/usr/include/ust/clock.h
> ltt-control-installed/usr/include/ust/ust_trace.h
> ltt-control-installed/usr/include/ust/processor.h
> ltt-control-installed/usr/include/ust/core.h
> ltt-control-installed/usr/include/ust/ust.h
> ltt-control-installed/usr/include/ust/tracectl.h
> ltt-control-installed/usr/include/ust/tracepoint.h
> ltt-control-installed/usr/include/ust/ustctl.h
> ltt-control-installed/usr/include/ust/ustconsumer.h
> ltt-control-installed/usr/include/urcu-bp.h
> ltt-control-installed/usr/include/urcu-pointer-static.h
> ltt-control-installed/usr/include/urcu-defer.h
> ltt-control-installed/usr/include/urcu-qsbr.h
> ltt-control-installed/usr/include/urcu/
> ltt-control-installed/usr/include/urcu/urcu-futex.h
> ltt-control-installed/usr/include/urcu/rculist.h
> ltt-control-installed/usr/include/urcu/rculfqueue.h
> ltt-control-installed/usr/include/urcu/uatomic_arch.h
> ltt-control-installed/usr/include/urcu/hlist.h
> ltt-control-installed/usr/include/urcu/wfstack-static.h
> ltt-control-installed/usr/include/urcu/urcu_ref.h
> ltt-control-installed/usr/include/urcu/compiler.h
> ltt-control-installed/usr/include/urcu/rcuhlist.h
> ltt-control-installed/usr/include/urcu/rculfstack-static.h
> ltt-control-installed/usr/include/urcu/wfstack.h
> ltt-control-installed/usr/include/urcu/arch_generic.h
> ltt-control-installed/usr/include/urcu/arch.h
> ltt-control-installed/usr/include/urcu/config.h
> ltt-control-installed/usr/include/urcu/system.h
> ltt-control-installed/usr/include/urcu/wfqueue-static.h
> ltt-control-installed/usr/include/urcu/wfqueue.h
> ltt-control-installed/usr/include/urcu/uatomic_generic.h
> ltt-control-installed/usr/include/urcu/list.h
> ltt-control-installed/usr/include/urcu/rculfstack.h
> ltt-control-installed/usr/include/urcu/rculfqueue-static.h
> ltt-control-installed/usr/include/urcu-static.h
> ltt-control-installed/usr/include/urcu.h
> ltt-control-installed/usr/include/liblttctl/
> ltt-control-installed/usr/include/liblttctl/lttctl.h
> ltt-control-installed/usr/include/ltt-facility-loader-user_generic.h
> ltt-control-installed/usr/include/urcu-qsbr-static.h
> ltt-control-installed/usr/lib/
> ltt-control-installed/usr/lib/libustfork.so
> ltt-control-installed/usr/lib/libust.la
> ltt-control-installed/usr/lib/libust.so.0
> ltt-control-installed/usr/lib/libustctl.so.0
> ltt-control-installed/usr/lib/libltt-usertrace-fast.so.0
> ltt-control-installed/usr/lib/libtap.so
> ltt-control-installed/usr/lib/librculfqueue.a
> ltt-control-installed/usr/lib/liburcu-bp.a
> ltt-control-installed/usr/lib/libustinstr-malloc.so.0.0.0
> ltt-control-installed/usr/lib/libustfork.a
> ltt-control-installed/usr/lib/liburcu-signal.a
> ltt-control-installed/usr/lib/libustinstr-malloc.la
> ltt-control-installed/usr/lib/libwfqueue.la
> ltt-control-installed/usr/lib/liblttctl.a
> ltt-control-installed/usr/lib/libustconsumer.so
> ltt-control-installed/usr/lib/libust.so
> ltt-control-installed/usr/lib/libltt-instrument-functions.a
> ltt-control-installed/usr/lib/libustfork.so.0.0.0
> ltt-control-installed/usr/lib/libtap.la
> ltt-control-installed/usr/lib/liburcu-bp.la
> ltt-control-installed/usr/lib/liblttctl.so.0.0.0
> ltt-control-installed/usr/lib/librculfqueue.la
> ltt-control-installed/usr/lib/liblttctl.so.0
> ltt-control-installed/usr/lib/libustctl.la
> ltt-control-installed/usr/lib/libtap.a
> ltt-control-installed/usr/lib/libustctl.so
> ltt-control-installed/usr/lib/libtap.so.0
> ltt-control-installed/usr/lib/libltt-instrument-functions.so
> ltt-control-installed/usr/lib/libwfstack.a
> ltt-control-installed/usr/lib/libustconsumer.a
> ltt-control-installed/usr/lib/librculfstack.a
> ltt-control-installed/usr/lib/libustfork.so.0
> ltt-control-installed/usr/lib/libltt-usertrace-fast.so
> ltt-control-installed/usr/lib/libltt-usertrace-fast.a
> ltt-control-installed/usr/lib/libustctl.so.0.0.0
> ltt-control-installed/usr/lib/libustconsumer.la
> ltt-control-installed/usr/lib/liblttctl.la
> ltt-control-installed/usr/lib/libustfork.la
> ltt-control-installed/usr/lib/libwfqueue.a
> ltt-control-installed/usr/lib/liburcu-defer.la
> ltt-control-installed/usr/lib/libust-initializer.o
> ltt-control-installed/usr/lib/liburcu-qsbr.a
> ltt-control-installed/usr/lib/liburcu-mb.a
> ltt-control-installed/usr/lib/liburcu-defer.a
> ltt-control-installed/usr/lib/libustinstr-malloc.so.0
> ltt-control-installed/usr/lib/libtap.so.0.0.0
> ltt-control-installed/usr/lib/liburcu-qsbr.la
> ltt-control-installed/usr/lib/librculfstack.la
> ltt-control-installed/usr/lib/libustinstr-malloc.a
> ltt-control-installed/usr/lib/libustctl.a
> ltt-control-installed/usr/lib/libustinstr-malloc.so
> ltt-control-installed/usr/lib/liblttctl.so
> ltt-control-installed/usr/lib/liburcu-mb.la
> ltt-control-installed/usr/lib/liburcu.a
> ltt-control-installed/usr/lib/liburcu-signal.la
> ltt-control-installed/usr/lib/libust.so.0.0.0
> ltt-control-installed/usr/lib/liburcu.la
> ltt-control-installed/usr/lib/libltt-instrument-functions.so.0
> ltt-control-installed/usr/lib/libustconsumer.so.0
> ltt-control-installed/usr/lib/libwfstack.la
> ltt-control-installed/usr/lib/libust.a
> ltt-control-installed/usr/lib/libustconsumer.so.0.0.0
> ltt-control-installed/usr/src/
> ltt-control-installed/usr/src/sample-thread-brand.c
> ltt-control-installed/usr/src/sample-instrument-fct.c
> ltt-control-installed/usr/src/ltt-usertrace-fast.c
> ltt-control-installed/usr/src/sample-printf.c
> ltt-control-installed/usr/src/sample-highspeed.c
> ltt-control-installed/usr/src/ltt-instrument-functions.c
> ltt-control-installed/usr/src/sample-thread-slow.c
> ltt-control-installed/usr/src/sample-thread-fast.c
> ltt-control-installed/usr/src/sample.c
> ltt-control-installed/usr/src/ltt-facility-loader-user_generic.c
> ltt-control-installed/usr/test/
> ltt-control-installed/usr/test/sample-instrument-fct
> ltt-control-installed/usr/test/sample-thread-fast
> ltt-control-installed/usr/test/sample-thread-slow
> ltt-control-installed/usr/test/sample-thread-brand
> ltt-control-installed/usr/test/sample-highspeed
> ltt-control-installed/usr/test/sample-printf
>
>
>
>
>
> First of all, here's how I started things off:
> (bash)
> mkdir /mnt/debugfs
> echo "debugfs /mnt/debugfs debugfs rw 0 0" >> /etc/fstab
> mount /mnt/debugfs
> mkdir /mnt/relayfs
> mount -t relayfs relayfs /mnt/relayfs
> export LTT_FACILITIES=/usr/share/ltt-control/facilities
> export LTT_DAEMON=/usr/bin/lttd
>
> To start the tracer:
> lttctl -n trace4  -c -d -t /var/log/trace4 -l /mnt/relayfs/ltt
> bash-2.04# lttctl -n trace4 -c -d -t /var/log/trace4 -l /mnt/relayfs/ltt/
> Linux Trace Toolkit Trace Control
>
> Controlling trace : trace4
>
> Linux Trace Toolkit Trace Daemon
>
> Reading from relayfs directory : /mnt/relayfs/ltt//trace4
> Writing to trace directory : /var/log/trace4
>
> signal 17 received
> Creating supplementary trace files
> Appending facility file stack_arch_i386.xml
> Appending facility file locking.xml
> Appending facility file process.xml
> Appending facility file fs_data.xml
> Appending facility file kernel_arch_mips.xml
> Appending facility file kernel_arch_ppc.xml
> Appending facility file timer.xml
> Appending facility file ipc.xml
> Appending facility file memory.xml
> Appending facility file fs.xml
> Appending facility file kernel_arch_c2.xml
> Appending facility file kernel.xml
> Appending facility file network.xml
> Appending facility file user_generic.xml
> Appending facility file kernel_arch_i386.xml
> Appending facility file core.xml
> Appending facility file socket.xml
> Appending facility file kernel_arch_powerpc.xml
> Appending facility file kernel_arch_arm.xml
> Appending facility file network_ip_interface.xml
> Appending facility file statedump.xml
>
> <invoke my app>
>
>
> To stop tracing:
> bash-2.04# lttctl -n trace4 -R
> Linux Trace Toolkit Trace Control
>
> Controlling trace : trace4
>
>
>
>
> here's lttv-gui crashing when opening the trace
>
> [FedoraBox traces]$ lttv-gui
> The input data file /work/traces/trace4/control/facilities_0 does not 
> contain a trace
> The input data file /work/traces/trace4/control/interrupts_0 does not 
> contain a trace
> The input data file /work/traces/trace4/control/network_0 does not 
> contain a trace
> The input data file /work/traces/trace4/control/processes_0 does not 
> contain a trace
> The input data file /work/traces/trace4/control/modules_0 does not 
> contain a trace
>
> ** ERROR **: Trace /work/traces/trace4 has no facility tracefile
> aborting...
> /usr/local/bin/lttv-gui: line 10: 18614 Aborted                 (core 
> dumped) $LTTV_CMD.real -m guievents -m guifilter -m guicontrolflow -m 
> guistatistics -m guitracecontrol $*
> [FedoraBox traces]$
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20110406/8af8863b/attachment-0003.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 27216 bytes
Desc: not available
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20110406/8af8863b/attachment-0003.jpeg>