[ltt-dev] lttng enabled SMP kernel soft locks up

[Mathieu Desnoyers]

* Venkatesh Babu Chitlur Srinivasa (vchitlur) (vchitlur at cisco.com) wrote:
> 
>  I have installed FC13 on Sun Fire x4100 platform. I built and installed vanilla 2.6.35.4 SMP kernel and it booted up fine. Then I patched it with patch-2.6.35.4-lttng-0.230 and built and installed the kernel. But it soft locks up while booting. I am attaching the complete console output. Am I missing anything here?
> 
> Linux version 2.6.35.4 (root at arch-lab4) (gcc version 4.4.4 20100630 (Red Hat 4.4.4-10) (GCC) ) #2 SMP Mon Sep 20 11:36:40 PDT 2010 Command line: ro root=/dev/mapper/vg_archlab4-lv_root rd_LVM_LV=vg_archlab4/lv_root rd_LVM_LV=vg_archlab4/lv_swap rd_NO_LUKS rd_NO_MD rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=us console=ttyS0,9600

[...]

The soft lockup below looks like it's coming from lvm, not lttng. So I'm
a bit surprised that it works without the patchset and doesn't with the
patchset.

Can you provide your .config for both the kernel that works and the
kernel that doesn't ? Maybe some other kernel option got selected. Also,
please try to disable the added LTTng options (diffing between the two
config files will help you there) to see if it helps.

Another point is to make sure you reinstall your modules too (make
modules_install) when you deploy the lttng-patched kernel, and make sure
you re-create your initrd if you have one.

Thanks,

Mathieu

> BUG: soft lockup - CPU#3 stuck for 61s! [lvm:399]
> Modules linked in: scsi_transport_sas pata_amd mptbase usb_storage mptscsih pata_acpi mptsas ata_generic [last unloaded: scsi_wait_scan]
> CPU 3
> Modules linked in: scsi_transport_sas pata_amd mptbase usb_storage mptscsih pata_acpi mptsas ata_generic [last unloaded: scsi_wait_scan]
> 
> Pid: 399, comm: lvm Not tainted 2.6.35.4 #2 Sun Fire X4100 Server/Sun Fire X4100 Server
> RIP: 0010:[<ffffffff810f9c83>]  [<ffffffff810f9c83>] pcpu_alloc+0x702/0x8a3
> RSP: 0018:ffff880009babc08  EFLAGS: 00010286
> RAX: ffff880001e16a00 RBX: ffff880009babca8 RCX: 0000000000000080
> RDX: 0000000000000000 RSI: ffffe8fefe200000 RDI: ffffe8fefe200000
> RBP: ffffffff8100950e R08: 0000000000000000 R09: ffffffff81b86bc0
> R10: 0000000000000713 R11: 00000000000f508e R12: ffffffff81b86bc0
> R13: 0000000000000080 R14: 0000000000000000 R15: 0000000000000000
> FS:  00007fb838bf47a0(0000) GS:ffff880103b00000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: ffffe8fefe200000 CR3: 00000001fa78d000 CR4: 00000000000006e0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Process lvm (pid: 399, threadinfo ffff880009baa000, task ffff880009aaae20)
> Stack:
>  0000000000000003 ffff880000000003 0000008000000100 ffffffff81b86bc0
> <0> 0000000000001000 0000000000000000 00000001f96b246c 0000000000000040
> <0> ffff8801fbb55960 00000001817cdf96 0000000000000080 0000000100000001
> Call Trace:
>  [<ffffffff810f9e3f>] ? __alloc_percpu+0xb/0x10
>  [<ffffffff81060234>] ? __create_workqueue_key+0x96/0x228
>  [<ffffffff813543ff>] ? dm_create+0x433/0x4e5
>  [<ffffffff8135a2e8>] ? dev_create+0x53/0x2af
>  [<ffffffff8135a295>] ? dev_create+0x0/0x2af
>  [<ffffffff8135a21a>] ? ctl_ioctl+0x1c5/0x21d
>  [<ffffffff8135a280>] ? dm_ctl_ioctl+0xe/0x12
>  [<ffffffff8110dd13>] ? vfs_ioctl+0x2d/0xa1
>  [<ffffffff8110e266>] ? do_vfs_ioctl+0x468/0x4a1
>  [<ffffffff8110e31e>] ? sys_ioctl+0x7f/0xa4
>  [<ffffffff81008a82>] ? system_call_fastpath+0x16/0x1b
> Code: 63 ff 4c 63 ad 74 ff ff ff 45 31 f6 eb 1f 48 8b 05 0b ee a8 00 48 8b 73 18 89 d1 4c 01 fe 48 03 34 c8 4c 89 e9 44 88 f0 48 89 f7 <f3> aa 89 d7 4c 89 e6 e8 cb ea ff ff 3b 05 6b ac a8 00 89 c2 72
> Call Trace:
>  [<ffffffff810f9e3f>] ? __alloc_percpu+0xb/0x10
>  [<ffffffff81060234>] ? __create_workqueue_key+0x96/0x228
>  [<ffffffff813543ff>] ? dm_create+0x433/0x4e5
>  [<ffffffff8135a2e8>] ? dev_create+0x53/0x2af
>  [<ffffffff8135a295>] ? dev_create+0x0/0x2af
>  [<ffffffff8135a21a>] ? ctl_ioctl+0x1c5/0x21d
>  [<ffffffff8135a280>] ? dm_ctl_ioctl+0xe/0x12
>  [<ffffffff8110dd13>] ? vfs_ioctl+0x2d/0xa1
>  [<ffffffff8110e266>] ? do_vfs_ioctl+0x468/0x4a1
>  [<ffffffff8110e31e>] ? sys_ioctl+0x7f/0xa4
>  [<ffffffff81008a82>] ? system_call_fastpath+0x16/0x1b
> 

> _______________________________________________
> ltt-dev mailing list
> ltt-dev at lists.casi.polymtl.ca
> http://lists.casi.polymtl.ca/cgi-bin/mailman/listinfo/ltt-dev


-- 
Mathieu Desnoyers
Operating System Efficiency R&D Consultant
EfficiOS Inc.
http://www.efficios.com