[ltt-dev] LTTng 2.0 on ARM

Tue Sep 13 07:16:35 EDT 2011

On Tuesday 13 September 2011 12:35 AM, Mathieu Desnoyers wrote:
> * Avik Sil (avik.sil at linaro.org) wrote:
>>
>>> Hi Avik,
>>>
>>> Can I get your kernel .config ? Also, adding a printk in lttng-modules
>>> wrapper/vmalloc.h, just after vmalloc_sync_all_sym = (void *)
>>> kallsyms_lookup_name("vmalloc_sync_all");
>>>
>>> printing the vmalloc_sync_all_sym pointer value would be clearly
>>> helpful. I would think ARM does not implement vmalloc_sync_all, so the
>>> dummy mm/vmalloc.c vmalloc_sync_all weak symbol should be used as a
>>> valid empty function. Please grep for vmalloc_sync_all under your
>>> arch/arm to see if your particular omap flavor is implementing a
>>> vmalloc_sync_all.
>>>
>> Hi Mathieu,
>>
>> Please find attached my kernel .config. After adding printk just after
>> vmalloc_sync_all_sym = (void *)kallsyms_lookup_name("vmalloc_sync_all");
>> I got following output:
>>
>> [ 1139.173522] vmalloc_sync_all_sym: c00a1d14
>> [ 1139.180877] Internal error: Oops - undefined instruction: 0 [#1]
>> PREEMPT SMP
>> [ 1139.191284] Modules linked in: lttng_ftrace(+)
>> [ 1139.198974] CPU: 1    Not tainted  (3.0.0-1004-linaro-omap #6)
>> [ 1139.208099] PC is at vmalloc_sync_all+0x0/0x8
>> [ 1139.215667] LR is at init_module+0x1c/0x28 [lttng_ftrace]
>> [ 1139.224304] pc : [<c00a1d14>]    lr : [<bf800211>]    psr: 60000113
>> [ 1139.224304] sp : ebb11f50  ip : 00000000  fp : 00000000
>> [ 1139.242401] r10: 00000000  r9 : 00000000  r8 : c06a2f00
>> [ 1139.250793] r7 : bf8001f5  r6 : 00000000  r5 : 00182008  r4 : c00a1d14
>> [ 1139.260559] r3 : 271aed1f  r2 : ebb11f44  r1 : bf8003a6  r0 : 00000034
>> [ 1139.270233] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM
>> Segment user
>> [ 1139.280670] Control: 50c5387d  Table: abdb004a  DAC: 00000015
>> [ 1139.289672] Process insmod (pid: 1368, stack limit = 0xebb102f8)
>> [ 1139.298889] Stack: (0xebb11f50 to 0xebb12000)
>> [ 1139.306335] 1f40:                                     bf80044c
>> c00085f7 00000000 00000001
>> [ 1139.320587] 1f60: bf80044c bf80044c 00182008 bf80044c 00182008
>> 00000000 000019de c000c364
>> [ 1139.334899] 1f80: ebb10000 c00622bb 00182018 000019de 00182008
>> 00000002 000019de beb099a0
>> [ 1139.349273] 1fa0: 00000080 c000c161 00000002 000019de 00182018
>> 000019de 00182008 402e6250
>> [ 1139.363861] 1fc0: 00000002 000019de beb099a0 00000080 00000003
>> 00004000 400b9000 00000000
>> [ 1139.378540] 1fe0: 00011f80 beb09728 00008c01 40296ae0 00000110
>> 00182018 a92aabaa aab0b0aa
>> [ 1139.393310] [<c00a1d14>] (vmalloc_sync_all+0x0/0x8) from [<bf800211>]
>> (init_module+0x1c/0x28 [lttng_ftrace])
>> [ 1139.410095] [<bf800211>] (init_module+0x1c/0x28 [lttng_ftrace]) from
>> [<c00085f7>] (do_one_initcall+0x6b/0xfc)
>> [ 1139.427032] [<c00085f7>] (do_one_initcall+0x6b/0xfc) from
>> [<c00622bb>] (sys_init_module+0x4b/0x11c)
>> [ 1139.443176] [<c00622bb>] (sys_init_module+0x4b/0x11c) from
>> [<c000c161>] (ret_fast_syscall+0x1/0x50)
>> [ 1139.459472] Code: e8bdb007 bf008ff0 c0672290 c0be2e54 (f85db500)
>> [ 1139.475524] ---[ end trace 7f9f02516cb400bf ]---
>> Segmentation fault
>>
>> Also I found that vmalloc_sync_all is not implemented under arch/arm and
>> it is using the mm/vmalloc.c vmalloc_sync_all weak symbol whose
>> disassembly gives:
>>
>> bx	lr
> 
> Hrm, weird. It's possibly some linker issue around the weak symbol, or
> the compiler does not generate the stub correctly. Can you look at the
> full dissassembly of the weak vmalloc_sync_all ? I would like to know
> the surrounding of 0xc00a1d14.
> 
> e.g.
> objdump -S --start-address=0xc00a1d14 vmlinux |head -n 30

I got following output:
# objdump -S --start-address=0xc00a1d14
/usr/lib/debug/boot/vmlinux-3.0.0-1004-linaro-omap | head -n 30

/usr/lib/debug/boot/vmlinux-3.0.0-1004-linaro-omap:     file format
elf32-littlearm

Disassembly of section .text:

c00a1d14 <vmalloc_sync_all>:
c00a1d14:	b500      	push	{lr}
c00a1d16:	f76a fa9d 	bl	c000c254 <__gnu_mcount_nc>
c00a1d1a:	4770      	bx	lr

c00a1d1c <pcpu_free_vm_areas>:
c00a1d1c:	e92d 41f0 	stmdb	sp!, {r4, r5, r6, r7, r8, lr}
c00a1d20:	b500      	push	{lr}
c00a1d22:	f76a fa97 	bl	c000c254 <__gnu_mcount_nc>
c00a1d26:	2500      	movs	r5, #0
c00a1d28:	4604      	mov	r4, r0
c00a1d2a:	460f      	mov	r7, r1
c00a1d2c:	4606      	mov	r6, r0
c00a1d2e:	e004      	b.n	c00a1d3a <pcpu_free_vm_areas+0x1e>
c00a1d30:	f856 0b04 	ldr.w	r0, [r6], #4
c00a1d34:	3501      	adds	r5, #1
c00a1d36:	f7ff fc9b 	bl	c00a1670 <free_vm_area>
c00a1d3a:	42bd      	cmp	r5, r7
c00a1d3c:	dbf8      	blt.n	c00a1d30 <pcpu_free_vm_areas+0x14>
c00a1d3e:	4620      	mov	r0, r4
c00a1d40:	e8bd 41f0 	ldmia.w	sp!, {r4, r5, r6, r7, r8, lr}
c00a1d44:	f004 b888 	b.w	c00a5e58 <kfree>

c00a1d48 <walk_pmd_range>:

> 
> We have to keep in mind that this could also be a ftrace function tracer
> bug, in which case the surrounding of 0xc00a1d14 from the vmlinux image
> will not match the instructions in memory. Can you dump the hex content
> around 0xc00a1d14 just before the vmalloc_sync_all gets called and
> compare with the disassembly ?

The hex content dump shows:

[  150.810119] c00a1d14: b500 f85d eb04 4770

So it seems your guess is right, the content of vmlinux image
surrounding 0xc00a1d14 is not matching the instruction in memory. By
grepping objdump I found that instruction 'f85d eb04' translates to
"ldr.w	lr, [sp], #4".

> 
> Also, another test: what happens if you put a "return;" at the beginning
> of the wrapper/vmalloc.h vmalloc_sync_all wrapper ?
>
By putting "return;" the vmalloc_sync_all wrapper behaves correctly
without any oops.

Regards,
Avik

> Thanks,
> 
> Mathieu
> 