[ltt-dev] kernel oops with LTTng 0.178 with linux 2.6.31.6-rt19

Tue Mar 16 09:52:59 EDT 2010

* Rachana Rao (rachana.rtuser at gmail.com) wrote:
> Hello,
> 
> I am using LTTng version 0.178 on linux 2.6.31.6 patched with rt prempt
> patch-2.6.31.6-rt19
> LTT control version is 0.79
> 
> This is running on Celeron M 370 processor (1.5GHz) board.

It seems to be caused by the ltt-statedump module, which can be disabled
with CONFIG_LTT_STATEDUMP=n.

Make sure you have no warnings when ltt-statedump compiles.. maybe
there is a spinlock taken there that happens to become a raw spinlock in
RT...

Thanks,

Mathieu

> 
> *I am getting the following oops -
> *
> $ lttctl -C -w /home/app1/trace7 trace7
> Linux Trace Toolkit Trace Control 0.79-01022010
> 
> Controlling trace : trace7
> 
> lttctl: Creating trace
> lttctl: Forking lttd
> Linux Trace Toolkit Trace Daemon 0.79-01022010
> 
> ReadingBUG: unable to handle kernel NULL pointer dereference at 00000008
> IP: [<c10f4f0b>] plist_add+0x1b/0x90
> *pde = 00000000
> Oops: 0000 [#1] PREEMPT
> LTT NESTING LEVEL : 0
> last sysfs file:
> Modules linked in:
> 
> Pid: 839, comm: lttctl Not tainted (2.6.31.6-rt19 #3) H6037
> EIP: 0060:[<c10f4f0b>] EFLAGS: 00010046 CPU: 0
> EIP is at plist_add+0x1b/0x90
> EAX: 00000004 EBX: c1266120 ECX: cf261e74 EDX: c1266120
> ESI: cf261e7c EDI: cf2ce750 EBP: 00000008 ESP: cf261e18
>  DS: 007b ES: 007b FS: 0000 GS: 0033 SS: 0068 preempt:00000002
> Process lttctl (pid: 839, ti=cf260000 task=cf2ce750 task.ti=cf260000)
> Stack:
>  cf261e88 cf261e68 c1265dfc cf2ce750 c1265e0c c103eb41 00000001 00000041
> <0> cf261e74 cf261e68 c1266120 00000000 00000282 c1265dfc ffffffff c11dd97c
> <0> 00000000 00000282 cec98cfc c1265e04 00000078 c1265dfc c1265dfc c1265e04
> Call Trace:
>  [<c103eb41>] ? task_blocks_on_rt_mutex+0x141/0x1c0
>  [<c11dd97c>] ? rt_spin_lock_slowlock+0x18c/0x200
>  [<c10e4d9d>] ? ltt_statedump_start+0x2fd/0x7a0
>  [<c104a6ee>] ? marker_probe_cb+0x4e/0xc0
>  [<c10de3d2>] ? ltt_trace_start+0x122/0x270
>  [<c10e70e8>] ? enabled_write+0xd8/0x160
>  [<c10e7010>] ? enabled_write+0x0/0x160
>  [<c106dfd0>] ? vfs_write+0x90/0xc0
>  [<c106e173>] ? sys_write+0x53/0xf0
>  [<c1002eb1>] ? syscall_call+0x7/0xb
> Code: 04 89 50 10 8b 7c 24 08 83 c4 0c c3 8d 74 26 00 55 57 56 89 c6 53 89
> d3 83
>  ec 04 8d 40 0c 3b 46 0c 89 04 24 75 67 8b 2b 8d 45 fc <8b> 50 04 0f 18 02
> 90 39
>  dd 89 e9 74 1f 8b 3e 3b 7d fc 7c 18 90
> EIP: [<c10f4f0b>] plist_add+0x1b/0x90 SS:ESP 0068:cf261e18
> CR2: 0000000000000008
> ---[ end trace 18077621483d0c39 ]---
> note: lttctl[839] exited with preempt_count 2
>  from debugfs directory : /mnt/debugfs/ltt/trace7
> Writing to trace directory : /home/app1/trace7
> 
> lttctl: Starting trace
> Killed
> 
> $ Clocksource tsc unstable (delta = 146049025 ns)
> 
> *Following are excerpts from the kernel config -
> 
> *#
> # GCOV-based kernel profiling
> #
> # CONFIG_GCOV_KERNEL is not set
> # CONFIG_SLOW_WORK is not set
> CONFIG_LTT=y
> CONFIG_LTT_FILTER=y
> CONFIG_HAVE_LTT_DUMP_TABLES=y
> CONFIG_LTT_RELAY=y
> CONFIG_LTT_RELAY_LOCKLESS=y
> # CONFIG_LTT_RELAY_IRQOFF is not set
> # CONFIG_LTT_RELAY_LOCKED is not set
> CONFIG_LTT_SERIALIZE=y
> CONFIG_LTT_FAST_SERIALIZE=y
> CONFIG_LTT_TRACEPROBES=y
> CONFIG_LTT_TRACE_CONTROL=y
> CONFIG_LTT_TRACER=y
> CONFIG_LTT_ALIGNMENT=y
> CONFIG_LTT_CHECK_ARCH_EFFICIENT_UNALIGNED_ACCESS=y
> # CONFIG_LTT_DEBUG_EVENT_SIZE is not set
> CONFIG_LTT_USERSPACE_EVENT=y
> CONFIG_LTT_VMCORE=y
> CONFIG_LTT_STATEDUMP=y
> # CONFIG_LTT_ASCII is not set
> CONFIG_HAVE_IMMEDIATE=y
> CONFIG_IMMEDIATE=y
> CONFIG_HAVE_GENERIC_DMA_COHERENT=y
> CONFIG_SLABINFO=y
> CONFIG_RT_MUTEXES=y
> CONFIG_BASE_SMALL=0
> CONFIG_MODULES=y
> CONFIG_MODULE_FORCE_LOAD=y
> CONFIG_MODULE_UNLOAD=y
> # CONFIG_MODULE_FORCE_UNLOAD is not set
> # CONFIG_MODVERSIONS is not set
> # CONFIG_MODULE_SRCVERSION_ALL is not set
> CONFIG_BLOCK=y
> # CONFIG_LBDAF is not set
> CONFIG_BLK_DEV_BSG=y
> # CONFIG_BLK_DEV_INTEGRITY is not set
> 
> #
> # RCU Subsystem
> #
> # CONFIG_CLASSIC_RCU is not set
> # CONFIG_TREE_RCU is not set
> CONFIG_PREEMPT_RCU=y
> # CONFIG_RCU_TRACE is not set
> # CONFIG_TREE_RCU_TRACE is not set
> # CONFIG_PREEMPT_RCU_TRACE is not set
> # CONFIG_IKCONFIG is not set
> CONFIG_LOG_BUF_SHIFT=17
> CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
> CONFIG_HAVE_GET_CYCLES=y
> CONFIG_HAVE_TRACE_CLOCK=y
> # CONFIG_HAVE_TRACE_CLOCK_GENERIC is not set
> # CONFIG_HAVE_TRACE_CLOCK_32_TO_64 is not set
> # CONFIG_HAVE_UNSYNCHRONIZED_TSC is not set
> # CONFIG_GROUP_SCHED is not set
> # CONFIG_CGROUPS is not set
> # CONFIG_SYSFS_DEPRECATED_V2 is not set
> # CONFIG_RELAY is not set
> # CONFIG_NAMESPACES is not set
> # CONFIG_BLK_DEV_INITRD is not set
> # CONFIG_CC_OPTIMIZE_FOR_SIZE is not set
> CONFIG_SYSCTL=y
> CONFIG_ANON_INODES=y
> CONFIG_EMBEDDED=y
> CONFIG_UID16=y
> CONFIG_SYSCTL_SYSCALL=y
> CONFIG_KALLSYMS=y
> # CONFIG_KALLSYMS_EXTRA_PASS is not set
> CONFIG_HOTPLUG=y
> CONFIG_PRINTK=y
> CONFIG_BUG=y
> CONFIG_ELF_CORE=y
> CONFIG_PCSPKR_PLATFORM=y
> CONFIG_BASE_FULL=y
> CONFIG_FUTEX=y
> CONFIG_EPOLL=y
> CONFIG_SIGNALFD=y
> CONFIG_TIMERFD=y
> CONFIG_EVENTFD=y
> CONFIG_SHMEM=y
> CONFIG_AIO=y
> CONFIG_HAVE_PERF_COUNTERS=y
> 
> *
> *
> Do let me know if you need the entire .config file or any other details
> 
> I would appreciate any kind of help with this issue.
> 
> Thanks in adv
> rachana

> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev

-- 
Mathieu Desnoyers
Operating System Efficiency Consultant
EfficiOS Inc.
http://www.efficios.com