[ltt-dev] Linux Trace Toolkit

[Mathieu Desnoyers]

* Goetz, George (george.goetz at lmco.com) wrote:
> Hello,
> 
> We are trying to use the Linux Trace Toolkit (LTT) for the first time to perform some performance modeling on an application we are developing.
> 
> We are using the real time TimeSys Linux kernel. The vendor recommends the LTT tool but does not offer direct support for it. We are using Version 2.6.22.1-rt4 to patch the TimeSys kernel (Recommended by the vendor). This patch constrains us to
> 0.8.83 version of the LTT Viewer and 0.42 for LTT Control. We have successfully built all the kernel and tools.  Several files had to be patched manually. We linked in all the modules statically so did not do any of the modprobe commands. We have been able to generate some trace data but have not
> 
> It appears as if everything started up ok as indicated in the following excerpt from dmesg :
> 
> io scheduler noop registered
> io scheduler anticipatory registered
> io scheduler deadline registered
> io scheduler cfq registered (default)
> LTT : ltt-facilities init
> LTT : Facility core registered with id 0
> LTT : Facility fs registered with id 225
> LTT : Facility kernel_arch registered with id 230
> LTT : Facility kernel registered with id 211
> LTT : Facility list registered with id 145
> LTT : Facility mm registered with id 59
> LTT : Facility net registered with id 179
> LTT : Facility locking registered with id 178
> LTT : ltt-relay init
> ltt-control init
> LTT : ltt-facility-statedump init
> Boot video device is 0000:00:02.0
> pci_hotplug: PCI Hot Plug PCI Core version: 0.5
> ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3])
> ACPI: Processor [CPU1] (supports 2 throttling states)
> ACPI Exception (processor_core-0782): AE_NOT_FOUND, Processor Device is not present [20070126]
> Real Time Clock Driver v1.12ac
> 
> When using the following command :
> 
> -bash-2.05b# lttctl -n TryAgain -c -d -l /mnt/debugfs/ltt -t /usr/trace
> 
> The output to the terminal looks like everything is starting up correctly:
> 
> Linux Trace Toolkit Trace Control 0.42-16072007
> 
> Controlling trace : TryAgain
> 
> Linux Trace Toolkit Trace Daemon 0.42-16072007
> 
> Reading from debugfs directory : /mnt/debugfs/ltt/TryAgain
> Writing to trace directory : /usr/trace
> 
> Creating supplementary trace files
> Appending facility file core.xml
> Appending facility file fs.xml
> Appending facility file kernel.xml
> Appending facility file kernel_arch_arm.xml
> Appending facility file kernel_arch_c2.xml
> Appending facility file kernel_arch_i386.xml
> Appending facility file kernel_arch_mips.xml
> Appending facility file kernel_arch_powerpc.xml
> Appending facility file kernel_arch_ppc.xml
> Appending facility file kernel_arch_x86_64.xml
> Appending facility file locking.xml
> Appending facility file mm.xml
> Appending facility file net.xml
> Appending facility file stack_arch_i386.xml
> Appending facility file stack_arch_x86_64.xml
> Appending facility file list.xml
> Appending facility file user_generic.xml
> Appending facility file xen.xml
> Appending facility file compact.xml
> 
> The directory /mnt/debugfs/ initially has directories kprobes, ltt, and usbmon.  The usbmon has files 0s, 0t, and 0u but all are empty. The directory kprobes has files enabled and list but both are empty. The ltt directory is empty. After the command the directory TryAgain is created under directory ltt. TryAgain is created with file cpu_0 and directory 'control' under it. All the files are empty.
> 
> The messages in dmesg again appear to indicate things worked
> 
> ACPI: Power Button (FF) [PWRF]
> input: Power Button (CM) as /class/input/input1
> ACPI: Power Button (CM) [PWRB]
> device-mapper: ioctl: 4.11.0-ioctl (2006-10-12) initialised: dm-devel at redhat.com
> eth0: link up, 100Mbps, full-duplex
> NET: Registered protocol family 10
> lo: Disabled Privacy Extensions
> eth0: no IPv6 routers present
> ltt-control ltt_control_input
> ltt_control : trace TryAgain
> Creating trace TryAgain
> ltt-control ltt_control_input
> ltt_control : trace TryAgain
> Start tracing TryAgain
> Dumping facility core
> Dumping facility mm
> Dumping facility list
> Dumping facility locking
> Dumping facility net
> Dumping facility kernel
> Dumping facility fs
> Dumping facility kernel_arch
> ltt_statedump_start
> do_ltt_statedump
> do_ltt_statedump end
> 
> When I stop the trace with
> 
> -bash-2.05b# lttctl -n TryAgain -q
> 
> The following output is produced
> 
> Linux Trace Toolkit Trace Control 0.42-16072007
> 
> Controlling trace : TryAgain
> 
> The following messages were added to dmesg
> 
> ltt-control ltt_control_input
> ltt_control : trace TryAgain
> Stop tracing TryAgain
> 
> The /usr/trace directory contains cpu_0 and two directories - control
> and eventdefs. The file cpu_0 contains the majority of the data.  The
> eventdefs contains XML descriptions of events. These are created when
> the command is entered. There are 5 files under the control directory
> - facilities_0, interrupts_0, modules_0, network_0, and processes_0.
> The only file which contains data is processes_0. The others are
> empty.

Clearly, there is a problem here. facilities_0 should contain the event
metadata. Please look into lttctl --help to see if -q was the
appropriate way to stop tracing _and_ end the trace session. My first
guess is that you are just stopping the trace. Make sure that the
/mnt/debugfs/ltt directory is empty after you stop tracing and end the
trace session. It looks like there is still data sitting in your
buffers in memory.

> 
> We are using a separate computer to analyze the data so I copy the
> entire trace directory to the remote computer. The Viewer does not
> recognize the trace data. If I invoke the viewer with the command
> 
> lttv -m batchAnalysis -t trace
> 
> It states the obvious - the newtwork_0, interrupts_0, modules_0, and faciltities_0 do not contain a trace. It also gives an error
> 
> **ERROR** Trace /mnt/trace1 has no facility tracefile
> 
> And aborts.
> 
> I have several basic questions:
> 
> 
>  1.  Should all the files under control have data when invoked with
>  the above command line?

Yes, at least a buffer header.

>  2.  Is the facilities_0 file the one the viewer is complaining about
>  in the above error message? If not what does the error message mean?
>  How do I generate the missing information?

See response above.

>  3.  Is there some way to look at or analyze the data in processes_0
>  and cpu_0 using the Viewer if some of the files are empty?

No. facilities_0 contains the metainfo, which is necessary to map
between event IDs and the event xml description.

>  4.  Is there a web site with documentation beyond that on LTTng
>  explaining the Viewer?

Everything we have is available through lttng.org. But it may not match
your old version though.

> 
> We have also tried to run LTT using the non real-time version of Linux
> (2.6.20 from TimeSys) with the same results. In this case the patch
> (2.6.20-lttng-0.9.0.tar.bz2) applied cleanly and no manual editing was
> required. The Viewer (0.8.79-02032007.tar.gz) was also rebuilt.
> 
> Thanks in advance for any help and information.

No problem. I hope this will be helpful to you.

Thanks for the detailed problem report.

Mathieu

> 
> George Goetz
> 

> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev


-- 
Mathieu Desnoyers
Operating System Efficiency Consultant
EfficiOS Inc.
http://www.efficios.com