[ltt-dev] Linux Trace Toolkit

[Goetz, George]

Hello,

We are trying to use the Linux Trace Toolkit (LTT) for the first time to perform some performance modeling on an application we are developing.

We are using the real time TimeSys Linux kernel. The vendor recommends the LTT tool but does not offer direct support for it. We are using Version 2.6.22.1-rt4 to patch the TimeSys kernel (Recommended by the vendor). This patch constrains us to
0.8.83 version of the LTT Viewer and 0.42 for LTT Control. We have successfully built all the kernel and tools.  Several files had to be patched manually. We linked in all the modules statically so did not do any of the modprobe commands. We have been able to generate some trace data but have not

It appears as if everything started up ok as indicated in the following excerpt from dmesg :

io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered
io scheduler cfq registered (default)
LTT : ltt-facilities init
LTT : Facility core registered with id 0
LTT : Facility fs registered with id 225
LTT : Facility kernel_arch registered with id 230
LTT : Facility kernel registered with id 211
LTT : Facility list registered with id 145
LTT : Facility mm registered with id 59
LTT : Facility net registered with id 179
LTT : Facility locking registered with id 178
LTT : ltt-relay init
ltt-control init
LTT : ltt-facility-statedump init
Boot video device is 0000:00:02.0
pci_hotplug: PCI Hot Plug PCI Core version: 0.5
ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
ACPI: Processor [CPU1] (supports 2 throttling states)
ACPI Exception (processor_core-0782): AE_NOT_FOUND, Processor Device is not present [20070126]
Real Time Clock Driver v1.12ac

When using the following command :

-bash-2.05b# lttctl -n TryAgain -c -d -l /mnt/debugfs/ltt -t /usr/trace

The output to the terminal looks like everything is starting up correctly:

Linux Trace Toolkit Trace Control 0.42-16072007

Controlling trace : TryAgain

Linux Trace Toolkit Trace Daemon 0.42-16072007

Reading from debugfs directory : /mnt/debugfs/ltt/TryAgain
Writing to trace directory : /usr/trace

Creating supplementary trace files
Appending facility file core.xml
Appending facility file fs.xml
Appending facility file kernel.xml
Appending facility file kernel_arch_arm.xml
Appending facility file kernel_arch_c2.xml
Appending facility file kernel_arch_i386.xml
Appending facility file kernel_arch_mips.xml
Appending facility file kernel_arch_powerpc.xml
Appending facility file kernel_arch_ppc.xml
Appending facility file kernel_arch_x86_64.xml
Appending facility file locking.xml
Appending facility file mm.xml
Appending facility file net.xml
Appending facility file stack_arch_i386.xml
Appending facility file stack_arch_x86_64.xml
Appending facility file list.xml
Appending facility file user_generic.xml
Appending facility file xen.xml
Appending facility file compact.xml

The directory /mnt/debugfs/ initially has directories kprobes, ltt, and usbmon.  The usbmon has files 0s, 0t, and 0u but all are empty. The directory kprobes has files enabled and list but both are empty. The ltt directory is empty. After the command the directory TryAgain is created under directory ltt. TryAgain is created with file cpu_0 and directory 'control' under it. All the files are empty.

The messages in dmesg again appear to indicate things worked

ACPI: Power Button (FF) [PWRF]
input: Power Button (CM) as /class/input/input1
ACPI: Power Button (CM) [PWRB]
device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel at redhat.com
eth0: link up, 100Mbps, full-duplex
NET: Registered protocol family 10
lo: Disabled Privacy Extensions
eth0: no IPv6 routers present
ltt-control ltt_control_input
ltt_control : trace TryAgain
Creating trace TryAgain
ltt-control ltt_control_input
ltt_control : trace TryAgain
Start tracing TryAgain
Dumping facility core
Dumping facility mm
Dumping facility list
Dumping facility locking
Dumping facility net
Dumping facility kernel
Dumping facility fs
Dumping facility kernel_arch
ltt_statedump_start
do_ltt_statedump
do_ltt_statedump end

When I stop the trace with

-bash-2.05b# lttctl -n TryAgain -q

The following output is produced

Linux Trace Toolkit Trace Control 0.42-16072007

Controlling trace : TryAgain

The following messages were added to dmesg

ltt-control ltt_control_input
ltt_control : trace TryAgain
Stop tracing TryAgain

The /usr/trace directory contains cpu_0 and two directories - control and eventdefs. The file cpu_0 contains the majority of the data.  The eventdefs contains XML descriptions of events. These are created when the command is entered. There are 5 files under the control directory - facilities_0, interrupts_0, modules_0, network_0, and processes_0. The only file which contains data is processes_0. The others are empty.

We are using a separate computer to analyze the data so I copy the entire trace directory to the remote computer. The Viewer does not recognize the trace data. If I invoke the viewer with the command

lttv -m batchAnalysis -t trace

It states the obvious - the newtwork_0, interrupts_0, modules_0, and faciltities_0 do not contain a trace. It also gives an error

**ERROR** Trace /mnt/trace1 has no facility tracefile

And aborts.

I have several basic questions:


 1.  Should all the files under control have data when invoked with the above command line?
 2.  Is the facilities_0 file the one the viewer is complaining about in the above error message? If not what does the error message mean? How do I generate the missing information?
 3.  Is there some way to look at or analyze the data in processes_0 and cpu_0 using the Viewer if some of the files are empty?
 4.  Is there a web site with documentation beyond that on LTTng explaining the Viewer?

We have also tried to run LTT using the non real-time version of Linux (2.6.20 from TimeSys) with the same results. In this case the patch (2.6.20-lttng-0.9.0.tar.bz2) applied cleanly and no manual editing was required. The Viewer (0.8.79-02032007.tar.gz) was also rebuilt.

Thanks in advance for any help and information.

George Goetz

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20100305/6357e1e5/attachment-0003.htm>