[ltt-dev] LTTng 2.0 on ARM
Mathieu Desnoyers
compudj at krystal.dyndns.org
Tue Sep 13 11:50:38 EDT 2011
* Avik Sil (avik.sil at linaro.org) wrote:
> On Tuesday 13 September 2011 12:35 AM, Mathieu Desnoyers wrote:
> > * Avik Sil (avik.sil at linaro.org) wrote:
> >>
> >>> Hi Avik,
> >>>
> >>> Can I get your kernel .config ? Also, adding a printk in lttng-modules
> >>> wrapper/vmalloc.h, just after vmalloc_sync_all_sym = (void *)
> >>> kallsyms_lookup_name("vmalloc_sync_all");
> >>>
> >>> printing the vmalloc_sync_all_sym pointer value would be clearly
> >>> helpful. I would think ARM does not implement vmalloc_sync_all, so the
> >>> dummy mm/vmalloc.c vmalloc_sync_all weak symbol should be used as a
> >>> valid empty function. Please grep for vmalloc_sync_all under your
> >>> arch/arm to see if your particular omap flavor is implementing a
> >>> vmalloc_sync_all.
> >>>
> >> Hi Mathieu,
> >>
> >> Please find attached my kernel .config. After adding printk just after
> >> vmalloc_sync_all_sym = (void *)kallsyms_lookup_name("vmalloc_sync_all");
> >> I got following output:
> >>
> >> [ 1139.173522] vmalloc_sync_all_sym: c00a1d14
> >> [ 1139.180877] Internal error: Oops - undefined instruction: 0 [#1]
> >> PREEMPT SMP
> >> [ 1139.191284] Modules linked in: lttng_ftrace(+)
> >> [ 1139.198974] CPU: 1 Not tainted (3.0.0-1004-linaro-omap #6)
> >> [ 1139.208099] PC is at vmalloc_sync_all+0x0/0x8
> >> [ 1139.215667] LR is at init_module+0x1c/0x28 [lttng_ftrace]
> >> [ 1139.224304] pc : [<c00a1d14>] lr : [<bf800211>] psr: 60000113
> >> [ 1139.224304] sp : ebb11f50 ip : 00000000 fp : 00000000
> >> [ 1139.242401] r10: 00000000 r9 : 00000000 r8 : c06a2f00
> >> [ 1139.250793] r7 : bf8001f5 r6 : 00000000 r5 : 00182008 r4 : c00a1d14
> >> [ 1139.260559] r3 : 271aed1f r2 : ebb11f44 r1 : bf8003a6 r0 : 00000034
> >> [ 1139.270233] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM
> >> Segment user
> >> [ 1139.280670] Control: 50c5387d Table: abdb004a DAC: 00000015
> >> [ 1139.289672] Process insmod (pid: 1368, stack limit = 0xebb102f8)
> >> [ 1139.298889] Stack: (0xebb11f50 to 0xebb12000)
> >> [ 1139.306335] 1f40: bf80044c
> >> c00085f7 00000000 00000001
> >> [ 1139.320587] 1f60: bf80044c bf80044c 00182008 bf80044c 00182008
> >> 00000000 000019de c000c364
> >> [ 1139.334899] 1f80: ebb10000 c00622bb 00182018 000019de 00182008
> >> 00000002 000019de beb099a0
> >> [ 1139.349273] 1fa0: 00000080 c000c161 00000002 000019de 00182018
> >> 000019de 00182008 402e6250
> >> [ 1139.363861] 1fc0: 00000002 000019de beb099a0 00000080 00000003
> >> 00004000 400b9000 00000000
> >> [ 1139.378540] 1fe0: 00011f80 beb09728 00008c01 40296ae0 00000110
> >> 00182018 a92aabaa aab0b0aa
> >> [ 1139.393310] [<c00a1d14>] (vmalloc_sync_all+0x0/0x8) from [<bf800211>]
> >> (init_module+0x1c/0x28 [lttng_ftrace])
> >> [ 1139.410095] [<bf800211>] (init_module+0x1c/0x28 [lttng_ftrace]) from
> >> [<c00085f7>] (do_one_initcall+0x6b/0xfc)
> >> [ 1139.427032] [<c00085f7>] (do_one_initcall+0x6b/0xfc) from
> >> [<c00622bb>] (sys_init_module+0x4b/0x11c)
> >> [ 1139.443176] [<c00622bb>] (sys_init_module+0x4b/0x11c) from
> >> [<c000c161>] (ret_fast_syscall+0x1/0x50)
> >> [ 1139.459472] Code: e8bdb007 bf008ff0 c0672290 c0be2e54 (f85db500)
> >> [ 1139.475524] ---[ end trace 7f9f02516cb400bf ]---
> >> Segmentation fault
> >>
> >> Also I found that vmalloc_sync_all is not implemented under arch/arm and
> >> it is using the mm/vmalloc.c vmalloc_sync_all weak symbol whose
> >> disassembly gives:
> >>
> >> bx lr
> >
> > Hrm, weird. It's possibly some linker issue around the weak symbol, or
> > the compiler does not generate the stub correctly. Can you look at the
> > full dissassembly of the weak vmalloc_sync_all ? I would like to know
> > the surrounding of 0xc00a1d14.
> >
> > e.g.
> > objdump -S --start-address=0xc00a1d14 vmlinux |head -n 30
>
> I got following output:
> # objdump -S --start-address=0xc00a1d14
> /usr/lib/debug/boot/vmlinux-3.0.0-1004-linaro-omap | head -n 30
>
> /usr/lib/debug/boot/vmlinux-3.0.0-1004-linaro-omap: file format
> elf32-littlearm
>
>
> Disassembly of section .text:
>
> c00a1d14 <vmalloc_sync_all>:
> c00a1d14: b500 push {lr}
> c00a1d16: f76a fa9d bl c000c254 <__gnu_mcount_nc>
> c00a1d1a: 4770 bx lr
>
> c00a1d1c <pcpu_free_vm_areas>:
> c00a1d1c: e92d 41f0 stmdb sp!, {r4, r5, r6, r7, r8, lr}
> c00a1d20: b500 push {lr}
> c00a1d22: f76a fa97 bl c000c254 <__gnu_mcount_nc>
> c00a1d26: 2500 movs r5, #0
> c00a1d28: 4604 mov r4, r0
> c00a1d2a: 460f mov r7, r1
> c00a1d2c: 4606 mov r6, r0
> c00a1d2e: e004 b.n c00a1d3a <pcpu_free_vm_areas+0x1e>
> c00a1d30: f856 0b04 ldr.w r0, [r6], #4
> c00a1d34: 3501 adds r5, #1
> c00a1d36: f7ff fc9b bl c00a1670 <free_vm_area>
> c00a1d3a: 42bd cmp r5, r7
> c00a1d3c: dbf8 blt.n c00a1d30 <pcpu_free_vm_areas+0x14>
> c00a1d3e: 4620 mov r0, r4
> c00a1d40: e8bd 41f0 ldmia.w sp!, {r4, r5, r6, r7, r8, lr}
> c00a1d44: f004 b888 b.w c00a5e58 <kfree>
>
> c00a1d48 <walk_pmd_range>:
>
> >
> > We have to keep in mind that this could also be a ftrace function tracer
> > bug, in which case the surrounding of 0xc00a1d14 from the vmlinux image
> > will not match the instructions in memory. Can you dump the hex content
> > around 0xc00a1d14 just before the vmalloc_sync_all gets called and
> > compare with the disassembly ?
>
> The hex content dump shows:
>
> [ 150.810119] c00a1d14: b500 f85d eb04 4770
>
> So it seems your guess is right, the content of vmlinux image
> surrounding 0xc00a1d14 is not matching the instruction in memory. By
> grepping objdump I found that instruction 'f85d eb04' translates to
> "ldr.w lr, [sp], #4".
I'm adding the ARM function tracer developers in CC. Maybe they can
enlighten us.
Best regards,
Mathieu
>
> >
> > Also, another test: what happens if you put a "return;" at the beginning
> > of the wrapper/vmalloc.h vmalloc_sync_all wrapper ?
> >
> By putting "return;" the vmalloc_sync_all wrapper behaves correctly
> without any oops.
>
> Regards,
> Avik
>
> > Thanks,
> >
> > Mathieu
> >
>
--
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com
More information about the lttng-dev
mailing list