[ltt-dev] lttctl -D option hangs while trying to destroy trace

[fazlin ahamed]

One more query in the meantime i try with 2.6.31..

This machine where i use 2.6.29 is a kind of embedded machine where in space
is very limited. Hence my lttctl and lttd are on a NFS mounted directory and
also i store the trace data in a NFS mounted directory. Does this affect the
performance or will this may be the reason for hang??

Thanks,
Fazlin

On Fri, Oct 30, 2009 at 10:52 PM, Mathieu Desnoyers <
compudj at krystal.dyndns.org> wrote:

> * fazlin ahamed (fazlincse at gmail.com) wrote:
> > Ok i ll try out with 2.6.31 and let you know the results.. But the prob
> is i
> > would have to make it working for 2.6.29 anyway as it is the kernel tat
> we
> > ll be using (we have made few modifications in 2.6.29 to suit our
> needs)...
> >
>
> If you try with 2.6.31 and it works, then bissecting from 2.6.29 to
> 2.6.31 will help you finding the fix.
>
> Mathieu
>
> > I debugged the problem more and here are my updates:
> >
> > Once "lttctl -D -w /tmp/tdir/trace3 trace3" cmd hangs, i did ctrl+C to
> kill
> > the process.
> >
> > After tat ps -aef shows two lttd process running. From the code i
> understood
> > tat one was started wen i started trace and other was started wen i
> issued
> > "lttctl -D" command.
> >
> > And if i kill both these lttd process using "kill" command, i see lot of
> > "unread channel" messages in dmesg. I have pasted few below:
> >
> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 0)
> > LTT : metadata : commit count : 262144, subbuf size 262144
> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 1)
> > LTT : metadata : commit count : 262144, subbuf size 262144
> > LTT : Tracing not active for trace trace1
> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 0)
> > LTT : metadata : commit count : 262144, subbuf size 262144
> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 1)
> > LTT : metadata : commit count : 262144, subbuf size 262144
> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 0)
> > LTT : metadata : commit count : 262144, subbuf size 262144
> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 1)
> > LTT : metadata : commit count : 262144, subbuf size 262144
> >
> > My guess from the above msg was there are some trace data in buffers tat
> > couldnt be read to userspace since i killed "lttctl" and "lttd". Am i
> > guessing rite?
> >
> > I also tried to run "lttd" by myself and i got the following error:
> >
> > [root@(none) /]# /tmp/ltt/ltt-control-0.70-18082009/lttd/lttd -t
> > /tmp/tdir/trace2 -a -c /tmp/debugfs/ltt -v -d -n
> > Linux Trace Toolkit Trace Daemon 0.70-18082009
> >
> > Reading from debugfs directory : /tmp/debugfs/ltt
> > Writing to trace directory : /tmp/tdir/trace2
> >
> > Creating trace subdirectory /tmp/tdir/trace2
> > Adding inotify for channel /tmp/debugfs/ltt/
> > Added inotify for channel /tmp/debugfs/ltt/, wd 4294967295
> > Channel file : /tmp/debugfs/ltt/write_event
> > Opening file.
> > Appending to file /tmp/tdir/trace2/write_event as requested
> > Channel file : /tmp/debugfs/ltt/kprobes
> > Entering channel subdirectory...
> > Creating trace subdirectory /tmp/tdir/trace2/kprobes
> > Adding inotify for channel /tmp/debugfs/ltt/kprobes/
> > Added inotify for channel /tmp/debugfs/ltt/kprobes/, wd 4294967295
> > Channel file : /tmp/debugfs/ltt/kprobes/list
> > Opening file.
> > Appending to file /tmp/tdir/trace2/kprobes/list as requested
> > Channel file : /tmp/debugfs/ltt/kprobes/disable
> > Opening file.
> > ...
> > ...
> > ...
> > Creating trace subdirectory /tmp/tdir/trace2/control
> > Adding inotify for channel /tmp/debugfs/ltt/control/
> > Added inotify for channel /tmp/debugfs/ltt/control/, wd 4294967295
> > Error in getting the number of subbuffers: Inappropriate ioctl for device
> >
> > I am not sure whether these are related or valid. But my guess is
> > destroy_trace is sleeping/waiting for someone else (maybe lttd) and it is
> > not able to recover.
> >
> > Is there any fix for this problem already in latest ltt-control or lttng
> > kernel patches??
> >
> > Thanks in advance,
> > Fazlin
> >
> > On Fri, Oct 30, 2009 at 7:36 PM, Mathieu Desnoyers <
> > compudj at krystal.dyndns.org> wrote:
> >
> > > * fazlin ahamed (fazlincse at gmail.com) wrote:
> > > > Hi all,
> > > >
> > > > I am new to using lttng and previously i was using lttng in 2.6.18
> > > without
> > > > any problems. Now when i switched to 2.6.29, i downloaded the
> > > corresponding
> > > > patches (patch-2.6.29-lttng-0.121) and ltt-control tool
> > > > (ltt-control-0.70-18082009)..
> > > >
> > > > I patched the kernel and tried to start/stop tracing using following
> > > > commands:
> > > >
> > > > lttctl -C -w /tmp/tdir/trace3 -o channel.all.overwrite=1 trace3
> > > >
> > > > This command ran successfully. And wen i executed
> > > >
> > > > lttctl -D -w /tmp/tdir/trace3 trace3
> > > >
> > > > The cmd nevers returns... when i debugged further i found tat "write"
> > > call
> > > > to write trace3 to debugfs destroy_trace file hangs.
> > > >
> > > > Can anyone throw some light on this problem... forgive me if it is
> silly
> > > > question..
> > > >
> > >
> > > You might want to try with a newer LTTng version (for kernel 2.6.31)
> and
> > > see if you can reproduce the problem.
> > >
> > > Thanks,
> > >
> > > Mathieu
> > >
> > >
> > > > thanks in advance,
> > > > Fazlin
> > >
> > > > _______________________________________________
> > > > ltt-dev mailing list
> > > > ltt-dev at lists.casi.polymtl.ca
> > > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> > >
> > >
> > > --
> > > Mathieu Desnoyers
> > > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE
> 9A68
> > >
>
> --
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.casi.polymtl.ca/pipermail/lttng-dev/attachments/20091031/a2ad8ae4/attachment-0003.htm>