[ltt-dev] net extended trace openin error with LTTV (without attachment)
Benjamin Poirier
benjamin.poirier at polymtl.ca
Tue Feb 9 13:16:22 EST 2010
Mathieu Desnoyers wrote:
> * Mathieu Desnoyers (compudj at krystal.dyndns.org) wrote:
>
>> * Benjamin Poirier (benjamin.poirier at polymtl.ca) wrote:
>>
>>> Mathieu Desnoyers wrote:
>>>
>>>> * Benjamin Poirier (benjamin.poirier at polymtl.ca) wrote:
>>>>
>>>>
>>>>> paljak at mit.bme.hu wrote:
>>>>>
>>>>>
>>>>>> -----Mathieu Desnoyers <compudj at krystal.dyndns.org> wrote: -----
>>>>>>
>>>>>> To: paljak at mit.bme.hu
>>>>>> From: Mathieu Desnoyers <compudj at krystal.dyndns.org>
>>>>>> Date: 2010/02/08 16:36
>>>>>> cc: benjamin.poirier at polymtl.ca, ltt-dev at lists.casi.polymtl.ca
>>>>>> Subject: Re: [ltt-dev] net extended trace openin error with LTTV
>>>>>> (without attachment)
>>>>>>
>>>>>> * paljak at mit.bme.hu (paljak at mit.bme.hu) wrote:
>>>>>> > Hi Ben,
>>>>>> >
>>>>>> > I have been working with LTTng lately and the more I know it, the
>>>>>> more I
>>>>>> > like it!
>>>>>> >
>>>>>> > I've also been able to open and examine traces, and it looks very
>>>>>> > interesting.
>>>>>> >
>>>>>> > I've also tried to create my own, but I've ran into an error,
>>>>>> could you
>>>>>> > take a look?
>>>>>> >
>>>>>> > Basically, I receive an error
>>>>>> > (ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed:
>>>>>> (info !=
>>>>>> > NULL)) every time i try to open a net_extended trace with LTTV.
>>>>>> >
>>>>>> > Here's what i've done:
>>>>>> >
>>>>>> > 1. Downloaded the patched kernel: git clone
>>>>>> >
>>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/compudj/linux-2.6-lttng.git
>>>>>> > 2. Compiled the kernel with these options (.config):
>>>>>> > #
>>>>>> > # GCOV-based kernel profiling
>>>>>> > #
>>>>>> > # CONFIG_GCOV_KERNEL is not set
>>>>>> > CONFIG_SLOW_WORK=y
>>>>>> > # CONFIG_SLOW_WORK_DEBUG is not set
>>>>>> > CONFIG_LTT=y
>>>>>> > CONFIG_LTT_FILTER=m
>>>>>> > CONFIG_HAVE_LTT_DUMP_TABLES=y
>>>>>> > CONFIG_LTT_RELAY=y
>>>>>> > CONFIG_LTT_RELAY_LOCKLESS=y
>>>>>> > # CONFIG_LTT_RELAY_IRQOFF is not set
>>>>>> > # CONFIG_LTT_RELAY_LOCKED is not set
>>>>>> > CONFIG_LTT_SERIALIZE=y
>>>>>> > CONFIG_LTT_FAST_SERIALIZE=y
>>>>>> > CONFIG_LTT_TRACEPROBES=m
>>>>>> > CONFIG_LTT_TRACE_CONTROL=y
>>>>>> > CONFIG_LTT_TRACER=y
>>>>>> > CONFIG_LTT_ALIGNMENT=y
>>>>>> > CONFIG_LTT_CHECK_ARCH_EFFICIENT_UNALIGNED_ACCESS=y
>>>>>> > # CONFIG_LTT_DEBUG_EVENT_SIZE is not set
>>>>>> > CONFIG_LTT_USERSPACE_EVENT=m
>>>>>> > CONFIG_LTT_VMCORE=y
>>>>>> > CONFIG_LTT_STATEDUMP=m
>>>>>> > CONFIG_LTT_KPROBES=m
>>>>>> > # CONFIG_LTT_ASCII is not set
>>>>>> > 3. Install the kernel and boot it up
>>>>>> > 4. Load all the modules
>>>>>> > modprobe -v ltt-trace-control
>>>>>> > modprobe -v ltt-marker-control
>>>>>> > modprobe -v ltt-tracer
>>>>>> > modprobe -v ltt-serialize
>>>>>> > modprobe -v ltt-relay
>>>>>> > modprobe -v ipc-trace
>>>>>> > modprobe -v kernel-trace
>>>>>> > modprobe -v mm-trace
>>>>>> > modprobe -v net-trace
>>>>>> > modprobe -v fs-trace
>>>>>> > modprobe -v jbd2-trace
>>>>>> > modprobe -v ext4-trace
>>>>>> > modprobe -v syscall-trace
>>>>>> > modprobe -v trap-trace
>>>>>> > modprobe -v ltt-statedump
>>>>>> > #extended trace
>>>>>> > modprobe -v net-extended-trace
>>>>>> > 5. ltt-armall -n
>>>>>> > 6. lttctl -C -w /tmp/tr1 tr1
>>>>>> > 7. wget http://google.com
>>>>>> > 8. lttctl -D tr1
>>>>>> > 9. lttv -m textDump -t tr1
>>>>>> > **
>>>>>> > ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed:
>>>>>> (info !=
>>>>>> > NULL)
>>>>>> > /usr/local/bin/lttv: line 15: 2083 Aborted $0.real $*
>>>>>> > 10. lttv -m textDump -t tr1 --verbose --debug --edebug
>>>>>> > net.udpv4_rcv_extended: 1003.743082325 (/tmp/tr1/net_0), 1878, 1878,
>>>>>> > gnome-terminal, , 1, 0x0, SOFTIRQ { skb = 0xC4BBC240, saddr =
>>>>>> 3576624693,
>>>>>> > daddr = 3232235780, unicast = 1, ulen = 110, source = 53, dest =
>>>>>> 39087,
>>>>>> > data_start = a299884901000100 }
>>>>>> > Event header (tracefile /tmp/tr1/net_0 offset bb8):
>>>>>> > bb8 40 C2 BB C4
>>>>>> > **
>>>>>> > ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed:
>>>>>> (info !=
>>>>>> > NULL)
>>>>>> > /usr/local/bin/lttv: line 15: 2104 Aborted $0.real $*
>>>>>> >
>>>>>> > The installed versions are:
>>>>>> > Linux Trace Toolkit Visualizer 0.12.29-02022010
>>>>>> > Linux Trace Toolkit Trace Control 0.79-01022010
>>>>>> >
>>>>>> > Here's the created trace for reference:
>>>>>> > http://www.mit.bme.hu/~paljak/trace.tgz
>>>>>> <http://www.mit.bme.hu/%7Epaljak/trace.tgz>
>>>>>> >
>>>>>> > If I don't use net extended, than it works just fine. And I've
>>>>>> also had
>>>>>> > success open your example trace.
>>>>>> >
>>>>>> > Do you have any idea why do I get this error and how could I
>>>>>> correct it?
>>>>>>
>>>>>> That looks like a field type error in the net extended event. Can you
>>>>>> tell us what LTTng version you use ? (git branch should tell you)
>>>>>>
>>>>>> Is your traced machine 32 or 64 bits ? And what about the machine
>>>>>> running lttv ?
>>>>>>
>>>>>> Reviewing ltt/probes/net-extended-trace.c might help.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Mathieu
>>>>>>
>>>>>> I use lttng-0.188. And I run lttng and lttv on the same, 32 bit machine.
>>>>>>
>>>>>> I'm checking net-extended-trace.c but reason I could think of to cause
>>>>>> this error...
>>>>>>
>>>>>>
>>>>>>
>>>>> I can't reproduce the problem, but looking at the traces, I notice the alignment is different:
>>>>>
>>>>> your trace:
>>>>> metadata.core_marker_id: 994.751622140 (/tmp/trace/tr1/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 4 }
>>>>>
>>>>> my trace:
>>>>> metadata.core_marker_id: 193.282545152 (/tmp/tr1/metadata_0), 0, 0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended", event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 0 }
>>>>>
>>>>> Could you apply the following patch over the git kernel, build and test? Thanks.
>>>>>
>>>>>
>>>> Seem like my LTTng patch:
>>>>
>>>> lttng-trace-format-allow-large-alignment.patch
>>>> "lttng trace format allow large alignment
>>>>
>>>> Do not limit alignment on architecture size anymore, because uint64_t
>>>> types are
>>>> aligned on 64-bit even on 32-bit archs. It's a waste of space for 32-bit
>>>> architectures, but makes our format compatible with gcc."
>>>>
>>>> does not do everything it needs to do. Or maybe LTTV still do an
>>>> alignment on min(arch size, data size) somewhere.
>>>>
>>>> A way to "just amke it work" is to disable alignment in LTTng. But it
>>>> would be good to find the culprit.
>>>>
>>>>
>>> Indeed, I have CONFIG_LTT_ALIGNMENT=n in my kernel, lucky me! If I
>>> activate it I can reproduce the problem and the patch I suggested
>>> doesn't change it, so don't bother. Disable alignment in the meantime as
>>> suggested by Mathieu.
>>>
>>>
>> I found the culprit:
>>
>> include/linux/ltt-type-serializer.h:
>>
>> /*
>> * Statically check that 0 < largest_align < sizeof(void *) to make sure it is
>> * dumb-proof. It will make sure 0 is changed into 1 and unsigned long long is
>> * changed into sizeof(void *) on 32-bit architectures.
>> */
>> static inline void ltt_specialized_trace(const struct marker *mdata,
>> void *probe_data,
>> void *serialize_private, unsigned int data_size,
>> unsigned int largest_align)
>> {
>> largest_align = min_t(unsigned int, largest_align, sizeof(void *));
>> largest_align = max_t(unsigned int, largest_align, 1);
>> _ltt_specialized_trace(mdata, probe_data, serialize_private, data_size,
>> largest_align);
>> }
>>
>> It caps the alignment to the architecture size. I'll remove the min_t()
>> line and it should be allright for the next release. Can you test it
>> meanwhile ?
>>
>
> LTTng 0.190 should fix this alignment problem. Feedback would be welcome.
>
> Thanks,
>
Nope, same problem I believe:
metadata.core_marker_id: 40.870000627 (/tmp/trace3/metadata_0), 0, 0, ,
, 0, 0x0, MODE_UNKNOWN { channel = "net", name = "udpv4_rcv_extended",
event_id = 2, int = 4, long = 4, pointer = 4, size_t = 4, alignment = 4 }
metadata.core_marker_format: 40.870002196 (/tmp/trace3/metadata_0), 0,
0, , , 0, 0x0, MODE_UNKNOWN { channel = "net", name =
"udpv4_rcv_extended", format = "skb 0x%lX saddr #n4u%lu daddr #n4u%lu
unicast #1u%u ulen #n2u%hu source #n2u%hu dest #n2u%hu data_start #8u%lx" }
[...]
net.udpv4_rcv_extended: 44.175529809 (/tmp/trace3/net_0), 0, 0, swapper,
, 0, 0x0, SOFTIRQ { skb = 0xDA225A80, saddr = 2228195330, daddr =
2228200116, unicast = 1, ulen = 188, source = 53, dest = 45569,
data_start = a2068db001000100 }
**
ERROR:tracefile.c:1582:ltt_update_event_size: assertion failed: (info !=
NULL)
> Mathieu
>
>
>> Thanks,
>>
>> Mathieu
>>
>>
>>
>>> -Ben
>>>
>>>> Thanks,
>>>>
>>>> Mathieu
>>>>
>>>>
>>>>
>>>>
>>>>> diff --git a/ltt/probes/net-extended-trace.c b/ltt/probes/net-extended-trace.c
>>>>> index 074bfec..88410f8 100644
>>>>> --- a/ltt/probes/net-extended-trace.c
>>>>> +++ b/ltt/probes/net-extended-trace.c
>>>>> @@ -130,14 +130,14 @@ notrace void probe_udpv4_rcv_extended(struct sk_buff *skb)
>>>>> * bytes of UDP data if they are not in a fragment*/
>>>>> data.f8 = 0;
>>>>> if (skb_headlen(skb) >= sizeof(struct udphdr) + 8)
>>>>> - data.f8 = *(unsigned long long *)(skb->data + sizeof(*uh));
>>>>> + data.f8 = *(uint64_t *)(skb->data + sizeof(*uh));
>>>>> else if (skb_headlen(skb) >= sizeof(struct udphdr))
>>>>> memcpy(&data.f8, skb->data + sizeof(struct udphdr),
>>>>> skb_headlen(skb) - sizeof(struct udphdr));
>>>>>
>>>>> marker = &GET_MARKER(net, udpv4_rcv_extended);
>>>>> ltt_specialized_trace(marker, marker->single.probe_private,
>>>>> - &data, serialize_sizeof(data), sizeof(unsigned long long));
>>>>> + &data, serialize_sizeof(data), sizeof(uint64_t));
>>>>> }
>>>>>
>>>>> MODULE_LICENSE("GPL and additional rights");
>>>>>
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>> Gergely
>>>>>>
>>>>>>
>>>>>> ------------------------------------------------------------------------
>>>>>>
>>>>>> _______________________________________________
>>>>>> ltt-dev mailing list
>>>>>> ltt-dev at lists.casi.polymtl.ca
>>>>>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> ltt-dev mailing list
>>>>> ltt-dev at lists.casi.polymtl.ca
>>>>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>> --
>> Mathieu Desnoyers
>> OpenPGP key fingerprint: 8CD5 52C3 8E3C 4140 715F BA06 3F25 A8FE 3BAE 9A68
>>
>> _______________________________________________
>> ltt-dev mailing list
>> ltt-dev at lists.casi.polymtl.ca
>> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
>>
>>
>
>
More information about the lttng-dev
mailing list