[ltt-dev] net extended trace openin error with LTTV (without attachment)

Mathieu Desnoyers compudj at krystal.dyndns.org
Tue Feb 9 12:59:45 EST 2010


* Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote:
> * Benjamin Poirier (benjamin.poirier at polymtl.ca) wrote:
> > Mathieu Desnoyers wrote:
> > > * Benjamin Poirier (benjamin.poirier at polymtl.ca) wrote:
> > >   
> > >> paljak at mit.bme.hu wrote:
> > >>     
> > >>> -----Mathieu Desnoyers <compudj at krystal.dyndns.org> wrote: -----
> > >>>
> > >>>     To: paljak at mit.bme.hu
> > >>>     From: Mathieu Desnoyers <compudj at krystal.dyndns.org>
> > >>>     Date: 2010/02/08 16:36
> > >>>     cc: benjamin.poirier at polymtl.ca, ltt-dev at lists.casi.polymtl.ca
> > >>>     Subject: Re: [ltt-dev] net extended trace openin error with LTTV
> > >>>     (without attachment)
> > >>>
> > >>>     * paljak at mit.bme.hu (paljak at mit.bme.hu) wrote:
> > >>>     > Hi Ben,
> > >>>     >
> > >>>     > I have been working with LTTng lately and the more I know it, the
> > >>>     more I
> > >>>     > like it!
> > >>>     >
> > >>>     > I've also been able to open and examine traces, and it looks very
> > >>>     > interesting.
> > >>>     >
> > >>>     > I've also tried to create my own, but I've ran into an error,
> > >>>     could you
> > >>>     > take a look?
> > >>>     >
> > >>>     > Basically, I receive an error
> > >>>     > (ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed:
> > >>>     (info !=
> > >>>     > NULL)) every time i try to open a net_extended trace with LTTV.
> > >>>     >
> > >>>     > Here's what i've done:
> > >>>     >
> > >>>     > 1. Downloaded the patched kernel: git clone
> > >>>     >
> > >>>     git://git.kernel.org/pub/scm/linux/kernel/git/compudj/linux-2.6-lttng.git
> > >>>     > 2. Compiled the kernel with these options (.config):
> > >>>     > #
> > >>>     > # GCOV-based kernel profiling
> > >>>     > #
> > >>>     > # CONFIG_GCOV_KERNEL is not set
> > >>>     > CONFIG_SLOW_WORK=y
> > >>>     > # CONFIG_SLOW_WORK_DEBUG is not set
> > >>>     > CONFIG_LTT=y
> > >>>     > CONFIG_LTT_FILTER=m
> > >>>     > CONFIG_HAVE_LTT_DUMP_TABLES=y
> > >>>     > CONFIG_LTT_RELAY=y
> > >>>     > CONFIG_LTT_RELAY_LOCKLESS=y
> > >>>     > # CONFIG_LTT_RELAY_IRQOFF is not set
> > >>>     > # CONFIG_LTT_RELAY_LOCKED is not set
> > >>>     > CONFIG_LTT_SERIALIZE=y
> > >>>     > CONFIG_LTT_FAST_SERIALIZE=y
> > >>>     > CONFIG_LTT_TRACEPROBES=m
> > >>>     > CONFIG_LTT_TRACE_CONTROL=y
> > >>>     > CONFIG_LTT_TRACER=y
> > >>>     > CONFIG_LTT_ALIGNMENT=y
> > >>>     > CONFIG_LTT_CHECK_ARCH_EFFICIENT_UNALIGNED_ACCESS=y
> > >>>     > # CONFIG_LTT_DEBUG_EVENT_SIZE is not set
> > >>>     > CONFIG_LTT_USERSPACE_EVENT=m
> > >>>     > CONFIG_LTT_VMCORE=y
> > >>>     > CONFIG_LTT_STATEDUMP=m
> > >>>     > CONFIG_LTT_KPROBES=m
> > >>>     > # CONFIG_LTT_ASCII is not set
> > >>>     > 3. Install the kernel and boot it up
> > >>>     > 4. Load all the modules
> > >>>     > modprobe -v ltt-trace-control
> > >>>     > modprobe -v ltt-marker-control
> > >>>     > modprobe -v ltt-tracer
> > >>>     > modprobe -v ltt-serialize
> > >>>     > modprobe -v ltt-relay
> > >>>     > modprobe -v ipc-trace
> > >>>     > modprobe -v kernel-trace
> > >>>     > modprobe -v mm-trace
> > >>>     > modprobe -v net-trace
> > >>>     > modprobe -v fs-trace
> > >>>     > modprobe -v jbd2-trace
> > >>>     > modprobe -v ext4-trace
> > >>>     > modprobe -v syscall-trace
> > >>>     > modprobe -v trap-trace
> > >>>     > modprobe -v ltt-statedump
> > >>>     > #extended trace
> > >>>     > modprobe -v net-extended-trace
> > >>>     > 5. ltt-armall -n
> > >>>     > 6. lttctl -C -w /tmp/tr1 tr1
> > >>>     > 7. wget http://google.com
> > >>>     > 8. lttctl -D tr1
> > >>>     > 9. lttv -m textDump -t tr1
> > >>>     > **
> > >>>     > ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed:
> > >>>     (info !=
> > >>>     > NULL)
> > >>>     > /usr/local/bin/lttv: line 15:  2083 Aborted                 $0.real $*
> > >>>     > 10. lttv -m textDump -t tr1 --verbose --debug --edebug
> > >>>     > net.udpv4_rcv_extended: 1003.743082325 (/tmp/tr1/net_0), 1878, 1878,
> > >>>     > gnome-terminal, , 1, 0x0, SOFTIRQ { skb = 0xC4BBC240, saddr =
> > >>>     3576624693,
> > >>>     > daddr = 3232235780, unicast = 1, ulen = 110, source = 53, dest =
> > >>>     39087,
> > >>>     > data_start = a299884901000100 }
> > >>>     > Event header (tracefile /tmp/tr1/net_0 offset bb8):
> > >>>     >      bb8    40 C2 BB C4
> > >>>     > **
> > >>>     > ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed:
> > >>>     (info !=
> > >>>     > NULL)
> > >>>     > /usr/local/bin/lttv: line 15:  2104 Aborted                 $0.real $*
> > >>>     >
> > >>>     > The installed versions are:
> > >>>     > Linux Trace Toolkit Visualizer 0.12.29-02022010
> > >>>     > Linux Trace Toolkit Trace Control 0.79-01022010
> > >>>     >
> > >>>     > Here's the created trace for reference:
> > >>>     > http://www.mit.bme.hu/~paljak/trace.tgz
> > >>>     <http://www.mit.bme.hu/%7Epaljak/trace.tgz>
> > >>>     >
> > >>>     > If I don't use net extended, than it works just fine. And I've
> > >>>     also had
> > >>>     > success open your example trace.
> > >>>     >
> > >>>     > Do you have any idea why do I get this error and how could I
> > >>>     correct it?
> > >>>
> > >>>     That looks like a field type error in the net extended event. Can you
> > >>>     tell us what LTTng version you use ? (git branch should tell you)
> > >>>
> > >>>     Is your traced machine 32 or 64 bits ? And what about the machine
> > >>>     running lttv ?
> > >>>
> > >>>     Reviewing ltt/probes/net-extended-trace.c might help.
> > >>>
> > >>>     Thanks,
> > >>>
> > >>>     Mathieu
> > >>>
> > >>> I use lttng-0.188. And I run lttng and lttv on the same, 32 bit machine.
> > >>>
> > >>> I'm checking net-extended-trace.c but reason I could think of to cause
> > >>> this error...
> > >>>
> > >>>       
> > >> I can't reproduce the problem, but looking at the traces, I notice the alignment is different:
> > >>
> > >> your trace:
> > >> metadata.core_marker_id: 994.751622140 (/tmp/trace/tr1/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 4 }
> > >>
> > >> my trace:
> > >> metadata.core_marker_id: 193.282545152 (/tmp/tr1/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 0 }
> > >>
> > >> Could you apply the following patch over the git kernel, build and test? Thanks.
> > >>     
> > >
> > > Seem like my LTTng patch:
> > >
> > > lttng-trace-format-allow-large-alignment.patch
> > > "lttng trace format allow large alignment
> > >
> > > Do not limit alignment on architecture size anymore, because uint64_t
> > > types are
> > > aligned on 64-bit even on 32-bit archs. It's a waste of space for 32-bit
> > > architectures, but makes our format compatible with gcc."
> > >
> > > does not do everything it needs to do. Or maybe LTTV still do an
> > > alignment on min(arch size, data size) somewhere.
> > >
> > > A way to "just amke it work" is to disable alignment in LTTng. But it
> > > would be good to find the culprit.
> > >   
> > Indeed, I have CONFIG_LTT_ALIGNMENT=n in my kernel, lucky me! If I
> > activate it I can reproduce the problem and the patch I suggested
> > doesn't change it, so don't bother. Disable alignment in the meantime as
> > suggested by Mathieu.
> > 
> 
> I found the culprit:
> 
> include/linux/ltt-type-serializer.h:
> 
> /*
>  * Statically check that 0 < largest_align < sizeof(void *) to make sure it is
>  * dumb-proof. It will make sure 0 is changed into 1 and unsigned long long is
>  * changed into sizeof(void *) on 32-bit architectures.
>  */
> static inline void ltt_specialized_trace(const struct marker *mdata,
>                 void *probe_data,
>                 void *serialize_private, unsigned int data_size,
>                 unsigned int largest_align)
> {
>         largest_align = min_t(unsigned int, largest_align, sizeof(void *));
>         largest_align = max_t(unsigned int, largest_align, 1);
>         _ltt_specialized_trace(mdata, probe_data, serialize_private, data_size,
>                 largest_align);
> }
> 
> It caps the alignment to the architecture size. I'll remove the min_t()
> line and it should be allright for the next release. Can you test it
> meanwhile ?

LTTng 0.190 should fix this alignment problem. Feedback would be welcome.

Thanks,

Mathieu

> 
> Thanks,
> 
> Mathieu
> 
> 
> > -Ben
> > > Thanks,
> > >
> > > Mathieu
> > >
> > >
> > >   
> > >> diff --git a/ltt/probes/net-extended-trace.c b/ltt/probes/net-extended-trace.c
> > >> index 074bfec..88410f8 100644
> > >> --- a/ltt/probes/net-extended-trace.c
> > >> +++ b/ltt/probes/net-extended-trace.c
> > >> @@ -130,14 +130,14 @@ notrace void probe_udpv4_rcv_extended(struct sk_buff *skb)
> > >>  	 * bytes of UDP data if they are not in a fragment*/
> > >>  	data.f8 = 0;
> > >>  	if (skb_headlen(skb) >= sizeof(struct udphdr) + 8)
> > >> -		data.f8 = *(unsigned long long *)(skb->data + sizeof(*uh));
> > >> +		data.f8 = *(uint64_t *)(skb->data + sizeof(*uh));
> > >>  	else if (skb_headlen(skb) >= sizeof(struct udphdr))
> > >>  		memcpy(&data.f8, skb->data + sizeof(struct udphdr),
> > >>  			skb_headlen(skb) - sizeof(struct udphdr));
> > >>  
> > >>  	marker = &GET_MARKER(net, udpv4_rcv_extended);
> > >>  	ltt_specialized_trace(marker, marker->single.probe_private,
> > >> -		&data, serialize_sizeof(data), sizeof(unsigned long long));
> > >> +		&data, serialize_sizeof(data), sizeof(uint64_t));
> > >>  }
> > >>  
> > >>  MODULE_LICENSE("GPL and additional rights");
> > >>
> > >>     
> > >>> Thanks,
> > >>> Gergely
> > >>>
> > >>>
> > >>> ------------------------------------------------------------------------
> > >>>
> > >>> _______________________________________________
> > >>> ltt-dev mailing list
> > >>> ltt-dev at lists.casi.polymtl.ca
> > >>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> > >>>       
> > >> _______________________________________________
> > >> ltt-dev mailing list
> > >> ltt-dev at lists.casi.polymtl.ca
> > >> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> > >>
> > >>     
> > >
> > >   
> > 
> 
> -- 
> Mathieu Desnoyers
> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68
> 
> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> 

-- 
Mathieu Desnoyers
OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F  BA06 3F25 A8FE 3BAE 9A68




More information about the lttng-dev mailing list