[ltt-dev] lttctl -D option hangs while trying to destroy trace

[Mathieu Desnoyers]

Doing a diff of your two configuration files seems like a good start to
find out where the problem lays.

Mathieu

* fazlin ahamed (fazlincse at gmail.com) wrote:
> I got un update...
> 
> The machine tat i was facing problem had a stripped down version of 2.6.29
> kernel with very few kernel configuration options enabled.. (only those
> which are really required)..
> 
> I got another machine running 2.6.29 but with most of the kernel
> configuration option enabled. In this i applied the same patch and lttctl
> starts and destroys trace without any problem.
> 
> This makes me think whether i have missed any kernel configuration option
> that are required for LTT. I have supplied same value to all kernel config
> options supplied by LTT. But i guess there is some related kernel
> configuration option that i am missing.
> 
> Anyone has any idea what might be the missing link??? Or can someone pls let
> me know what are the basic options tat have to be enabled for LTT..
> 
> Any help is appreciated.
> 
> Thanks,
> Fazlin
> 
> On Sat, Oct 31, 2009 at 2:46 AM, fazlin ahamed <fazlincse at gmail.com> wrote:
> 
> > One more query in the meantime i try with 2.6.31..
> >
> > This machine where i use 2.6.29 is a kind of embedded machine where in
> > space is very limited. Hence my lttctl and lttd are on a NFS mounted
> > directory and also i store the trace data in a NFS mounted directory. Does
> > this affect the performance or will this may be the reason for hang??
> >
> > Thanks,
> > Fazlin
> >
> >
> > On Fri, Oct 30, 2009 at 10:52 PM, Mathieu Desnoyers <
> > compudj at krystal.dyndns.org> wrote:
> >
> >> * fazlin ahamed (fazlincse at gmail.com) wrote:
> >> > Ok i ll try out with 2.6.31 and let you know the results.. But the prob
> >> is i
> >> > would have to make it working for 2.6.29 anyway as it is the kernel tat
> >> we
> >> > ll be using (we have made few modifications in 2.6.29 to suit our
> >> needs)...
> >> >
> >>
> >> If you try with 2.6.31 and it works, then bissecting from 2.6.29 to
> >> 2.6.31 will help you finding the fix.
> >>
> >> Mathieu
> >>
> >> > I debugged the problem more and here are my updates:
> >> >
> >> > Once "lttctl -D -w /tmp/tdir/trace3 trace3" cmd hangs, i did ctrl+C to
> >> kill
> >> > the process.
> >> >
> >> > After tat ps -aef shows two lttd process running. From the code i
> >> understood
> >> > tat one was started wen i started trace and other was started wen i
> >> issued
> >> > "lttctl -D" command.
> >> >
> >> > And if i kill both these lttd process using "kill" command, i see lot of
> >> > "unread channel" messages in dmesg. I have pasted few below:
> >> >
> >> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 0)
> >> > LTT : metadata : commit count : 262144, subbuf size 262144
> >> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 1)
> >> > LTT : metadata : commit count : 262144, subbuf size 262144
> >> > LTT : Tracing not active for trace trace1
> >> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 0)
> >> > LTT : metadata : commit count : 262144, subbuf size 262144
> >> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 1)
> >> > LTT : metadata : commit count : 262144, subbuf size 262144
> >> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 0)
> >> > LTT : metadata : commit count : 262144, subbuf size 262144
> >> > LTT : unread channel metadata offset is 262144 and cons_off : 0 (cpu 1)
> >> > LTT : metadata : commit count : 262144, subbuf size 262144
> >> >
> >> > My guess from the above msg was there are some trace data in buffers tat
> >> > couldnt be read to userspace since i killed "lttctl" and "lttd". Am i
> >> > guessing rite?
> >> >
> >> > I also tried to run "lttd" by myself and i got the following error:
> >> >
> >> > [root@(none) /]# /tmp/ltt/ltt-control-0.70-18082009/lttd/lttd -t
> >> > /tmp/tdir/trace2 -a -c /tmp/debugfs/ltt -v -d -n
> >> > Linux Trace Toolkit Trace Daemon 0.70-18082009
> >> >
> >> > Reading from debugfs directory : /tmp/debugfs/ltt
> >> > Writing to trace directory : /tmp/tdir/trace2
> >> >
> >> > Creating trace subdirectory /tmp/tdir/trace2
> >> > Adding inotify for channel /tmp/debugfs/ltt/
> >> > Added inotify for channel /tmp/debugfs/ltt/, wd 4294967295
> >> > Channel file : /tmp/debugfs/ltt/write_event
> >> > Opening file.
> >> > Appending to file /tmp/tdir/trace2/write_event as requested
> >> > Channel file : /tmp/debugfs/ltt/kprobes
> >> > Entering channel subdirectory...
> >> > Creating trace subdirectory /tmp/tdir/trace2/kprobes
> >> > Adding inotify for channel /tmp/debugfs/ltt/kprobes/
> >> > Added inotify for channel /tmp/debugfs/ltt/kprobes/, wd 4294967295
> >> > Channel file : /tmp/debugfs/ltt/kprobes/list
> >> > Opening file.
> >> > Appending to file /tmp/tdir/trace2/kprobes/list as requested
> >> > Channel file : /tmp/debugfs/ltt/kprobes/disable
> >> > Opening file.
> >> > ...
> >> > ...
> >> > ...
> >> > Creating trace subdirectory /tmp/tdir/trace2/control
> >> > Adding inotify for channel /tmp/debugfs/ltt/control/
> >> > Added inotify for channel /tmp/debugfs/ltt/control/, wd 4294967295
> >> > Error in getting the number of subbuffers: Inappropriate ioctl for
> >> device
> >> >
> >> > I am not sure whether these are related or valid. But my guess is
> >> > destroy_trace is sleeping/waiting for someone else (maybe lttd) and it
> >> is
> >> > not able to recover.
> >> >
> >> > Is there any fix for this problem already in latest ltt-control or lttng
> >> > kernel patches??
> >> >
> >> > Thanks in advance,
> >> > Fazlin
> >> >
> >> > On Fri, Oct 30, 2009 at 7:36 PM, Mathieu Desnoyers <
> >> > compudj at krystal.dyndns.org> wrote:
> >> >
> >> > > * fazlin ahamed (fazlincse at gmail.com) wrote:
> >> > > > Hi all,
> >> > > >
> >> > > > I am new to using lttng and previously i was using lttng in 2.6.18
> >> > > without
> >> > > > any problems. Now when i switched to 2.6.29, i downloaded the
> >> > > corresponding
> >> > > > patches (patch-2.6.29-lttng-0.121) and ltt-control tool
> >> > > > (ltt-control-0.70-18082009)..
> >> > > >
> >> > > > I patched the kernel and tried to start/stop tracing using following
> >> > > > commands:
> >> > > >
> >> > > > lttctl -C -w /tmp/tdir/trace3 -o channel.all.overwrite=1 trace3
> >> > > >
> >> > > > This command ran successfully. And wen i executed
> >> > > >
> >> > > > lttctl -D -w /tmp/tdir/trace3 trace3
> >> > > >
> >> > > > The cmd nevers returns... when i debugged further i found tat
> >> "write"
> >> > > call
> >> > > > to write trace3 to debugfs destroy_trace file hangs.
> >> > > >
> >> > > > Can anyone throw some light on this problem... forgive me if it is
> >> silly
> >> > > > question..
> >> > > >
> >> > >
> >> > > You might want to try with a newer LTTng version (for kernel 2.6.31)
> >> and
> >> > > see if you can reproduce the problem.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > Mathieu
> >> > >
> >> > >
> >> > > > thanks in advance,
> >> > > > Fazlin
> >> > >
> >> > > > _______________________________________________
> >> > > > ltt-dev mailing list
> >> > > > ltt-dev at lists.casi.polymtl.ca
> >> > > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> >> > >
> >> > >
> >> > > --
> >> > > Mathieu Desnoyers
> >> > > OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE
> >> 9A68
> >> > >
> >>
> >> --
> >> Mathieu Desnoyers
> >> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE
> >> 9A68
> >>
> >
> >

> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev


-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68