[ltt-dev] kernel oops with LTTng 0.178 with linux 2.6.31.6-rt19

Mathieu Desnoyers compudj at krystal.dyndns.org
Tue Mar 16 12:41:25 EDT 2010 

* Rachana Rao (rachana.rtuser at gmail.com) wrote:
> Hello,
> 
> I disabled the CONFIG_LTT_STATEDUMP and have been able to run lttctl on my
> appln. Thank you !
> 
> However, i am now facing another issue when i try to insmod my fpga driver
> on the board.
> 
> $ insmod fpga.ko
> BUG: unable to handle kernel NULL pointer dereference at 00000004
> IP: [<c1049791>] marker_update_probe_range+0xa1/0x310
> *pde = 00000000
> Oops: 0000 [#1] PREEMPT
> LTT NESTING LEVEL : 0
> last sysfs file:
> Modules linked in: fpga(+)
> Pid: 799, comm: insmod Not tainted (2.6.31.6-rt19 #4) H6037
> EIP: 0060:[<c1049791>] EFLAGS: 00010203 CPU: 0
> EIP is at marker_update_probe_range+0xa1/0x310
> EAX: 00000000 EBX: 00000000 ECX: cff42aa0 EDX: cf2cf070
> ESI: 00000000 EDI: c126bac8 EBP: fdc60fc0 ESP: cf0ddf14
>  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068 preempt:00000000
> Process insmod (pid: 799, ti=cf0dc000 task=cf2cf070 task.ti=cf0dc000)
> Stack:
>  00000000 00000000 cf0ddf7c 00000000 00047fd4 00000001 cff5d000 0000002b
> <0> c126baa8 00000000 c126bac8 fffffffe c1049a2a c10335b6 00000001 cff42aa0
> <0> c1263ce0 c1263cf8 00000001 00000000 c103388a ffffffff 00000000 cff42aa0
> Call Trace:
>  [<c1049a2a>] ? marker_module_notify+0x2a/0x30
>  [<c10335b6>] ? notifier_call_chain+0x36/0x70
>  [<c103388a>] ? __blocking_notifier_call_chain+0x4a/0x70
>  [<c10338c7>] ? blocking_notifier_call_chain+0x17/0x20
>  [<c10436e6>] ? sys_init_module+0x86/0x220
>  [<c1002e44>] ? sysenter_do_call+0x12/0x26
> Code: 08 08 0f 85 54 02 00 00 85 ff 0f 84 44 02 00 00 8b 53 28 8b 43 24 e8
> 0f 1a
>  00 00 89 f0 88 43 0c 83 c3 30 39 dd 0f 86 3f 01 00 00 <8b> 53 04 8b 03 e8
> b5 f0
>  ff ff 85 c0 89 c7 0f 84 db 00 00 00 8b
> EIP: [<c1049791>] marker_update_probe_range+0xa1/0x310 SS:ESP 0068:cf0ddf14
> CR2: 0000000000000004
> ---[ end trace 0a7239d58be85cb5 ]---
> Killed
> $ Clocksource tsc unstable (delta = 130894527 ns)
> 
> Is this related to the tracer or a fault my fpga module ?

Let me guess: your fpga module is a binary module built for a kernel
configuration that knows nothing about the "marker" fields in struct
module, am I right ? You should recompile your fpga module so it fits
your kernel sources.

Hope this helps,

Mathieu

> 
> Thanks in adv
> rachana
> 
> On Tue, Mar 16, 2010 at 7:22 PM, Mathieu Desnoyers <
> compudj at krystal.dyndns.org> wrote:
> 
> > * Rachana Rao (rachana.rtuser at gmail.com) wrote:
> > > Hello,
> > >
> > > I am using LTTng version 0.178 on linux 2.6.31.6 patched with rt prempt
> > > patch-2.6.31.6-rt19
> > > LTT control version is 0.79
> > >
> > > This is running on Celeron M 370 processor (1.5GHz) board.
> >
> > It seems to be caused by the ltt-statedump module, which can be disabled
> > with CONFIG_LTT_STATEDUMP=n.
> >
> > Make sure you have no warnings when ltt-statedump compiles.. maybe
> > there is a spinlock taken there that happens to become a raw spinlock in
> > RT...
> >
> > Thanks,
> >
> > Mathieu
> >
> > >
> > > *I am getting the following oops -
> >  > *
> > > $ lttctl -C -w /home/app1/trace7 trace7
> > > Linux Trace Toolkit Trace Control 0.79-01022010
> > >
> > > Controlling trace : trace7
> > >
> > > lttctl: Creating trace
> > > lttctl: Forking lttd
> > > Linux Trace Toolkit Trace Daemon 0.79-01022010
> > >
> > > ReadingBUG: unable to handle kernel NULL pointer dereference at 00000008
> > > IP: [<c10f4f0b>] plist_add+0x1b/0x90
> > > *pde = 00000000
> > > Oops: 0000 [#1] PREEMPT
> > > LTT NESTING LEVEL : 0
> > > last sysfs file:
> > > Modules linked in:
> > >
> > > Pid: 839, comm: lttctl Not tainted (2.6.31.6-rt19 #3) H6037
> > > EIP: 0060:[<c10f4f0b>] EFLAGS: 00010046 CPU: 0
> > > EIP is at plist_add+0x1b/0x90
> > > EAX: 00000004 EBX: c1266120 ECX: cf261e74 EDX: c1266120
> > > ESI: cf261e7c EDI: cf2ce750 EBP: 00000008 ESP: cf261e18
> > >  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068 preempt:00000002
> > > Process lttctl (pid: 839, ti=cf260000 task=cf2ce750 task.ti=cf260000)
> > > Stack:
> > >  cf261e88 cf261e68 c1265dfc cf2ce750 c1265e0c c103eb41 00000001 00000041
> > > <0> cf261e74 cf261e68 c1266120 00000000 00000282 c1265dfc ffffffff
> > c11dd97c
> > > <0> 00000000 00000282 cec98cfc c1265e04 00000078 c1265dfc c1265dfc
> > c1265e04
> > > Call Trace:
> > >  [<c103eb41>] ? task_blocks_on_rt_mutex+0x141/0x1c0
> > >  [<c11dd97c>] ? rt_spin_lock_slowlock+0x18c/0x200
> > >  [<c10e4d9d>] ? ltt_statedump_start+0x2fd/0x7a0
> > >  [<c104a6ee>] ? marker_probe_cb+0x4e/0xc0
> > >  [<c10de3d2>] ? ltt_trace_start+0x122/0x270
> > >  [<c10e70e8>] ? enabled_write+0xd8/0x160
> > >  [<c10e7010>] ? enabled_write+0x0/0x160
> > >  [<c106dfd0>] ? vfs_write+0x90/0xc0
> > >  [<c106e173>] ? sys_write+0x53/0xf0
> > >  [<c1002eb1>] ? syscall_call+0x7/0xb
> > > Code: 04 89 50 10 8b 7c 24 08 83 c4 0c c3 8d 74 26 00 55 57 56 89 c6 53
> > 89
> > > d3 83
> > >  ec 04 8d 40 0c 3b 46 0c 89 04 24 75 67 8b 2b 8d 45 fc <8b> 50 04 0f 18
> > 02
> > > 90 39
> > >  dd 89 e9 74 1f 8b 3e 3b 7d fc 7c 18 90
> > > EIP: [<c10f4f0b>] plist_add+0x1b/0x90 SS:ESP 0068:cf261e18
> > > CR2: 0000000000000008
> > > ---[ end trace 18077621483d0c39 ]---
> > > note: lttctl[839] exited with preempt_count 2
> > >  from debugfs directory : /mnt/debugfs/ltt/trace7
> > > Writing to trace directory : /home/app1/trace7
> > >
> > > lttctl: Starting trace
> > > Killed
> > >
> > > $ Clocksource tsc unstable (delta = 146049025 ns)
> > >
> > > *Following are excerpts from the kernel config -
> > >
> > > *#
> > > # GCOV-based kernel profiling
> > > #
> > > # CONFIG_GCOV_KERNEL is not set
> > > # CONFIG_SLOW_WORK is not set
> > > CONFIG_LTT=y
> > > CONFIG_LTT_FILTER=y
> > > CONFIG_HAVE_LTT_DUMP_TABLES=y
> > > CONFIG_LTT_RELAY=y
> > > CONFIG_LTT_RELAY_LOCKLESS=y
> > > # CONFIG_LTT_RELAY_IRQOFF is not set
> > > # CONFIG_LTT_RELAY_LOCKED is not set
> > > CONFIG_LTT_SERIALIZE=y
> > > CONFIG_LTT_FAST_SERIALIZE=y
> > > CONFIG_LTT_TRACEPROBES=y
> > > CONFIG_LTT_TRACE_CONTROL=y
> > > CONFIG_LTT_TRACER=y
> > > CONFIG_LTT_ALIGNMENT=y
> > > CONFIG_LTT_CHECK_ARCH_EFFICIENT_UNALIGNED_ACCESS=y
> > > # CONFIG_LTT_DEBUG_EVENT_SIZE is not set
> > > CONFIG_LTT_USERSPACE_EVENT=y
> > > CONFIG_LTT_VMCORE=y
> > > CONFIG_LTT_STATEDUMP=y
> > > # CONFIG_LTT_ASCII is not set
> > > CONFIG_HAVE_IMMEDIATE=y
> > > CONFIG_IMMEDIATE=y
> > > CONFIG_HAVE_GENERIC_DMA_COHERENT=y
> > > CONFIG_SLABINFO=y
> > > CONFIG_RT_MUTEXES=y
> > > CONFIG_BASE_SMALL=0
> > > CONFIG_MODULES=y
> > > CONFIG_MODULE_FORCE_LOAD=y
> > > CONFIG_MODULE_UNLOAD=y
> > > # CONFIG_MODULE_FORCE_UNLOAD is not set
> > > # CONFIG_MODVERSIONS is not set
> > > # CONFIG_MODULE_SRCVERSION_ALL is not set
> > > CONFIG_BLOCK=y
> > > # CONFIG_LBDAF is not set
> > > CONFIG_BLK_DEV_BSG=y
> > > # CONFIG_BLK_DEV_INTEGRITY is not set
> > >
> > > #
> > > # RCU Subsystem
> > > #
> > > # CONFIG_CLASSIC_RCU is not set
> > > # CONFIG_TREE_RCU is not set
> > > CONFIG_PREEMPT_RCU=y
> > > # CONFIG_RCU_TRACE is not set
> > > # CONFIG_TREE_RCU_TRACE is not set
> > > # CONFIG_PREEMPT_RCU_TRACE is not set
> > > # CONFIG_IKCONFIG is not set
> > > CONFIG_LOG_BUF_SHIFT=17
> > > CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> > > CONFIG_HAVE_GET_CYCLES=y
> > > CONFIG_HAVE_TRACE_CLOCK=y
> > > # CONFIG_HAVE_TRACE_CLOCK_GENERIC is not set
> > > # CONFIG_HAVE_TRACE_CLOCK_32_TO_64 is not set
> > > # CONFIG_HAVE_UNSYNCHRONIZED_TSC is not set
> > > # CONFIG_GROUP_SCHED is not set
> > > # CONFIG_CGROUPS is not set
> > > # CONFIG_SYSFS_DEPRECATED_V2 is not set
> > > # CONFIG_RELAY is not set
> > > # CONFIG_NAMESPACES is not set
> > > # CONFIG_BLK_DEV_INITRD is not set
> > > # CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
> > > CONFIG_SYSCTL=y
> > > CONFIG_ANON_INODES=y
> > > CONFIG_EMBEDDED=y
> > > CONFIG_UID16=y
> > > CONFIG_SYSCTL_SYSCALL=y
> > > CONFIG_KALLSYMS=y
> > > # CONFIG_KALLSYMS_EXTRA_PASS is not set
> > > CONFIG_HOTPLUG=y
> > > CONFIG_PRINTK=y
> > > CONFIG_BUG=y
> > > CONFIG_ELF_CORE=y
> > > CONFIG_PCSPKR_PLATFORM=y
> > > CONFIG_BASE_FULL=y
> > > CONFIG_FUTEX=y
> > > CONFIG_EPOLL=y
> > > CONFIG_SIGNALFD=y
> > > CONFIG_TIMERFD=y
> > > CONFIG_EVENTFD=y
> > > CONFIG_SHMEM=y
> > > CONFIG_AIO=y
> > > CONFIG_HAVE_PERF_COUNTERS=y
> > >
> > > *
> > > *
> > > Do let me know if you need the entire .config file or any other details
> > >
> > > I would appreciate any kind of help with this issue.
> > >
> > > Thanks in adv
> > > rachana
> >
> > > _______________________________________________
> > > ltt-dev mailing list
> > > ltt-dev at lists.casi.polymtl.ca
> > > http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev
> >
> >
> > --
> > Mathieu Desnoyers
> > Operating System Efficiency Consultant
> > EfficiOS Inc.
> > http://www.efficios.com
> >

> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev


-- 
Mathieu Desnoyers
Operating System Efficiency Consultant
EfficiOS Inc.
http://www.efficios.com

More information about the lttng-dev mailing list