[Gluster-users] XFS and MD RAID
Jack Wang
xjtuwjp at gmail.com
Mon Sep 10 01:29:25 UTC 2012
Hi Brian,
below patch should fix your bug.
John reports:
BUG: soft lockup - CPU#2 stuck for 23s! [kworker/u:8:2202] [..] Call Trace:
[<ffffffff8141782a>] scsi_remove_target+0xda/0x1f0
[<ffffffff81421de5>] sas_rphy_remove+0x55/0x60
[<ffffffff81421e01>] sas_rphy_delete+0x11/0x20
[<ffffffff81421e35>] sas_port_delete+0x25/0x160
[<ffffffff814549a3>] mptsas_del_end_device+0x183/0x270
...introduced by commit 3b661a9 "[SCSI] fix hot unplug vs async scan race".
Don't restart lookup of more stargets in the multi-target case, just
arrange to traverse the list once, on the assumption that new targets
are always added at the end. There is no guarantee that the target
will change state in scsi_target_reap() so we can end up spinning if
we restart.
LKML-Reference:
<CAEhu1-6wq1YsNiscGMwP4ud0Q+MrViRzv=kcWCQSBNc8c68N5Q at mail.gmail.com>
Reported-by: John Drescher <drescherjm at gmail.com>
Tested-by: John Drescher <drescherjm at gmail.com>
Signed-off-by: Dan Williams <djbw at fb.com>
---
drivers/scsi/scsi_sysfs.c | 30 ++++++++++++++----------------
1 file changed, 14 insertions(+), 16 deletions(-)
diff --git a/drivers/scsi/scsi_sysfs.c b/drivers/scsi/scsi_sysfs.c
index 093d4f6..ce5224c 100644
--- a/drivers/scsi/scsi_sysfs.c
+++ b/drivers/scsi/scsi_sysfs.c
@@ -1031,33 +1031,31 @@ static void __scsi_remove_target(struct
scsi_target *starget) void scsi_remove_target(struct device *dev) {
struct Scsi_Host *shost = dev_to_shost(dev->parent);
- struct scsi_target *starget, *found;
+ struct scsi_target *starget, *last = NULL;
unsigned long flags;
- restart:
- found = NULL;
+ /* remove targets being careful to lookup next entry before
+ * deleting the last
+ */
spin_lock_irqsave(shost->host_lock, flags);
list_for_each_entry(starget, &shost->__targets, siblings) {
if (starget->state == STARGET_DEL)
continue;
if (starget->dev.parent == dev || &starget->dev == dev) {
- found = starget;
- found->reap_ref++;
- break;
+ /* assuming new targets arrive at the end */
+ starget->reap_ref++;
+ spin_unlock_irqrestore(shost->host_lock, flags);
+ if (last)
+ scsi_target_reap(last);
+ last = starget;
+ __scsi_remove_target(starget);
+ spin_lock_irqsave(shost->host_lock, flags);
}
}
spin_unlock_irqrestore(shost->host_lock, flags);
- if (found) {
- __scsi_remove_target(found);
- scsi_target_reap(found);
- /* in the case where @dev has multiple starget children,
- * continue removing.
- *
- * FIXME: does such a case exist?
- */
- goto restart;
- }
+ if (last)
+ scsi_target_reap(last);
}
EXPORT_SYMBOL(scsi_remove_target);
Jack
2012/9/10 Brian Candler <B.Candler at pobox.com>:
> On Sun, Sep 09, 2012 at 10:22:02AM +0100, Brian Candler wrote:
>> And I've also run this workload on another test box here over a week and not
>> been able to reproduce the problem :-(
>
> I did a full update on the test box (kernel now 3.2.0-30-generic). On
> subsequent testing the machine does crash in a different way, but I think
> this is probably unrelated to my original problem.
>
> After last set of testing I noticed disk /dev/sdd was misbehaving, so I
> rebuilt the RAID0 array without it, and a new XFS filesystem on the
> remaining disks of course. sdd is not mounted, but I've not yet physically
> removed it.
>
> When running a test, the machine locked up in a really bad way - I couldn't
> ssh into it, but I did get a console capture (see attached). This is
> repeatable, I've done it three times now.
>
> After a forced reset, I got some messages from /var/log/syslog (below). The
> errors occur nearly 30 seconds after an error on sdd, and I see
> "scsi_remove_target" in the backtrace.
>
> However, a misbehaving disk ought to be a fairly regular occurrence in a
> RAID array, and I'm sure it shouldn't bring the system to its knees
> (especially a disk which is not being used)
>
> Maybe I'm coming round to the view that a hardware RAID controller, which
> can isolate the system from the quirks of bad drives, might be a good idea
> after all - even though having a single virtual disk makes it hard to do
> SMART queries against individual drives.
>
> This of course assumes the RAID controller manufacturer tests the various
> error scenarios better than the Linux kernel drivers do :-(
>
> So I'd still be grateful for a recommendation of 24-port SATA RAID
> controller with good Linux support.
>
> Regards,
>
> Brian.
>
>
> Sep 9 20:37:59 dev-storage1 kernel: [ 838.093290] sd 4:0:2:0: [sdd] Synchronizing SCSI cache
> Sep 9 20:37:59 dev-storage1 kernel: [ 838.093350] sd 4:0:2:0: [sdd] Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.824696] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u:12:1602]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825049] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825076] CPU 1
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825077] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825095]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825097] Pid: 1602, comm: kworker/u:12 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825099] RIP: 0010:[<ffffffff8165a629>] [<ffffffff8165a629>] _raw_spin_unlock_irqrestore+0x19/0x30
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825104] RSP: 0018:ffff880210d55b28 EFLAGS: 00000282
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825105] RAX: 0000000000000282 RBX: 0000000000000004 RCX: 0000000000000004
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825107] RDX: ffff8802050d8818 RSI: 0000000000000282 RDI: 0000000000000282
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825108] RBP: ffff880210d55b30 R08: ffffffff81cddaa0 R09: 0000000000000100
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825109] R10: 0000000000000004 R11: 0000000000000000 R12: ffff88020ee0a150
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825110] R13: ffff880210d55b30 R14: ffff88020ee0a144 R15: ffff88020ee09d60
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825112] FS: 0000000000000000(0000) GS:ffff88021fc80000(0000) knlGS:0000000000000000
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825113] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825114] CR2: 00007f70a894c000 CR3: 0000000001c05000 CR4: 00000000000406e0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825116] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825117] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825119] Process kworker/u:12 (pid: 1602, threadinfo ffff880210d54000, task ffff8802050aae00)
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825120] Stack:
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825215] ffff8802050d8800 ffff880210d55b60 ffffffff8142d1e7 ffff8802050d8400
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825217] ffff8802050d8400 ffff88020dacc010 ffff88020dacc000 ffff880210d55b90
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825219] ffffffff8142f3c9 ffff8802050d8400 ffff880211940000 ffff880211eaa4c8
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825222] Call Trace:
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825347] [<ffffffff8142d1e7>] scsi_target_reap+0x77/0xf0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825349] [<ffffffff8142f3c9>] scsi_remove_target+0xc9/0xe0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825354] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825357] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825360] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825364] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825369] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825373] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825377] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825381] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825385] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825388] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825391] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825394] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825397] [<ffffffff81085814>] worker_thread+0x164/0x370
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825399] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825401] [<ffffffff8108a03c>] kthread+0x8c/0xa0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825403] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825405] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825407] [<ffffffff81664b70>] ? gs_change+0x13/0x13
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.825407] Code: 48 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 7e 36 9e ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826836] Call Trace:
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826838] [<ffffffff8142d1e7>] scsi_target_reap+0x77/0xf0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826839] [<ffffffff8142f3c9>] scsi_remove_target+0xc9/0xe0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826842] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826845] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826848] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826850] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826854] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826858] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826861] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826865] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826868] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826870] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826874] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826876] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826878] [<ffffffff81085814>] worker_thread+0x164/0x370
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826880] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826882] [<ffffffff8108a03c>] kthread+0x8c/0xa0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826884] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826885] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
> Sep 9 20:38:27 dev-storage1 kernel: [ 865.826887] [<ffffffff81664b70>] ? gs_change+0x13/0x13
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.744859] BUG: soft lockup - CPU#1 stuck for 22s! [kworker/u:12:1602]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745212] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745239] CPU 1
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745239] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745258]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745260] Pid: 1602, comm: kworker/u:12 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745262] RIP: 0010:[<ffffffff8165a629>] [<ffffffff8165a629>] _raw_spin_unlock_irqrestore+0x19/0x30
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745268] RSP: 0018:ffff880210d55b58 EFLAGS: 00000286
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745269] RAX: 0000000000000286 RBX: ffffffff81cddaa0 RCX: 0000000000000004
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745270] RDX: ffff8802050d8808 RSI: 0000000000000286 RDI: 0000000000000286
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745271] RBP: ffff880210d55b60 R08: ffffffff81cddaa0 R09: 0000000000000100
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745273] R10: 0000000000000004 R11: 0000000000000000 R12: 0000000000000004
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745274] R13: 0000000000000000 R14: 0000000000000004 R15: ffff880210d55b30
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745276] FS: 0000000000000000(0000) GS:ffff88021fc80000(0000) knlGS:0000000000000000
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745277] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745278] CR2: 00007f70a894c000 CR3: 0000000001c05000 CR4: 00000000000406e0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745279] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745281] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745282] Process kworker/u:12 (pid: 1602, threadinfo ffff880210d54000, task ffff8802050aae00)
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745283] Stack:
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745378] ffff8802050d8400 ffff880210d55b90 ffffffff8142f3b4 ffff8802050d8400
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745381] ffff880211940000 ffff880211eaa4c8 ffff880211940000 ffff880210d55bb0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745383] ffffffffa00a2e65 ffff8802050d8400 ffff880211eaa480 ffff880210d55bd0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745385] Call Trace:
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745510] [<ffffffff8142f3b4>] scsi_remove_target+0xb4/0xe0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745514] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745518] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745521] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745524] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745530] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745534] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745538] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745541] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745545] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745548] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745551] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745555] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745557] [<ffffffff81085814>] worker_thread+0x164/0x370
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745560] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745562] [<ffffffff8108a03c>] kthread+0x8c/0xa0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745564] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745566] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745568] [<ffffffff81664b70>] ? gs_change+0x13/0x13
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.745569] Code: 48 8b 5d f0 4c 8b 65 f8 c9 c3 0f 1f 84 00 00 00 00 00 55 48 89 e5 53 66 66 66 66 90 48 89 f3 e8 7e 36 9e ff 66 90 48 89 df 57 9d <66> 66 90 66 90 5b 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.746982] Call Trace:
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.746984] [<ffffffff8142f3b4>] scsi_remove_target+0xb4/0xe0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.746987] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.746990] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.746993] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.746995] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.746998] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747002] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747006] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747009] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747013] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747015] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747018] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747020] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747022] [<ffffffff81085814>] worker_thread+0x164/0x370
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747024] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747026] [<ffffffff8108a03c>] kthread+0x8c/0xa0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747028] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747030] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
> Sep 9 20:38:55 dev-storage1 kernel: [ 893.747032] [<ffffffff81664b70>] ? gs_change+0x13/0x13
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.928887] INFO: rcu_sched detected stall on CPU 1 (t=15000 jiffies)
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929229] sending NMI to all CPUs:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929230] NMI backtrace for cpu 1
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929232] CPU 1
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929233] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929268]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929270] Pid: 1602, comm: kworker/u:12 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929274] RIP: 0010:[<ffffffff8101b316>] [<ffffffff8101b316>] native_read_tsc+0x6/0x20
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929280] RSP: 0018:ffff88021fc83d68 EFLAGS: 00000002
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929281] RAX: 00000000a61a449e RBX: 0000000000002710 RCX: 00000000a61a4476
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929283] RDX: 0000000000000313 RSI: 0000000000000100 RDI: 0000000000335923
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929285] RBP: ffff88021fc83d68 R08: 000000000000000a R09: 0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929286] R10: 0000000000000000 R11: 0000000000000000 R12: 00000000a61a4476
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929288] R13: 0000000000000001 R14: 0000000000335923 R15: 0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929290] FS: 0000000000000000(0000) GS:ffff88021fc80000(0000) knlGS:0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929292] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929294] CR2: 00007f70a894c000 CR3: 0000000001c05000 CR4: 00000000000406e0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929296] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929297] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929299] Process kworker/u:12 (pid: 1602, threadinfo ffff880210d54000, task ffff8802050aae00)
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929301] Stack:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929396] ffff88021fc83d98 ffffffff81317bda 0000000000002710 ffffffff81c31b00
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929399] ffffffff81c31c00 ffff88021fc8e7c0 ffff88021fc83da8 ffffffff81317b01
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929402] ffff88021fc83dc8 ffffffff810327b2 ffffffff81cddaa0 ffffffff81c31b00
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929406] Call Trace:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929523] <IRQ>
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929644] [<ffffffff81317bda>] delay_tsc+0x4a/0x80
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929647] [<ffffffff81317b01>] __const_udelay+0x31/0x40
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929651] [<ffffffff810327b2>] arch_trigger_all_cpu_backtrace+0x72/0xa0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929655] [<ffffffff810e04e7>] check_cpu_stall.isra.36+0x97/0xf0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929657] [<ffffffff810e0578>] __rcu_pending+0x38/0x1b0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929660] [<ffffffff810e09e9>] rcu_check_callbacks+0x79/0x1e0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929664] [<ffffffff81078528>] update_process_times+0x48/0x90
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929667] [<ffffffff8109c054>] tick_sched_timer+0x64/0xc0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929671] [<ffffffff8108e738>] __run_hrtimer+0x78/0x1f0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929673] [<ffffffff8109bff0>] ? tick_nohz_handler+0x100/0x100
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929677] [<ffffffff8106e57d>] ? __do_softirq+0xfd/0x210
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929679] [<ffffffff8108f067>] hrtimer_interrupt+0xf7/0x230
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929683] [<ffffffff81664c6c>] ? call_softirq+0x1c/0x30
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929686] [<ffffffff81665609>] smp_apic_timer_interrupt+0x69/0x99
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929689] [<ffffffff816634de>] apic_timer_interrupt+0x6e/0x80
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929690] <EOI>
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929799] [<ffffffff8165a629>] ? _raw_spin_unlock_irqrestore+0x19/0x30
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929802] [<ffffffff8142f3b4>] scsi_remove_target+0xb4/0xe0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929807] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929811] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929815] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929819] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929825] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929831] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929836] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929841] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929846] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929849] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929853] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929857] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929860] [<ffffffff81085814>] worker_thread+0x164/0x370
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929863] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929866] [<ffffffff8108a03c>] kthread+0x8c/0xa0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929868] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929871] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929873] [<ffffffff81664b70>] ? gs_change+0x13/0x13
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.929875] Code: c3 90 90 90 90 55 89 f8 48 89 e5 e6 70 e4 71 5d c3 0f 1f 40 00 55 89 f0 48 89 e5 e6 70 89 f8 e6 71 5d c3 66 90 55 48 89 e5 0f 31 <89> c1 48 89 d0 48 c1 e0 20 48 09 c8 5d c3 66 66 66 2e 0f 1f 84
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931273] Call Trace:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931274] <IRQ> [<ffffffff81317bda>] delay_tsc+0x4a/0x80
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931279] [<ffffffff81317b01>] __const_udelay+0x31/0x40
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931281] [<ffffffff810327b2>] arch_trigger_all_cpu_backtrace+0x72/0xa0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931284] [<ffffffff810e04e7>] check_cpu_stall.isra.36+0x97/0xf0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931287] [<ffffffff810e0578>] __rcu_pending+0x38/0x1b0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931290] [<ffffffff810e09e9>] rcu_check_callbacks+0x79/0x1e0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931292] [<ffffffff81078528>] update_process_times+0x48/0x90
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931295] [<ffffffff8109c054>] tick_sched_timer+0x64/0xc0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931297] [<ffffffff8108e738>] __run_hrtimer+0x78/0x1f0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931300] [<ffffffff8109bff0>] ? tick_nohz_handler+0x100/0x100
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931302] [<ffffffff8106e57d>] ? __do_softirq+0xfd/0x210
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931305] [<ffffffff8108f067>] hrtimer_interrupt+0xf7/0x230
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931308] [<ffffffff81664c6c>] ? call_softirq+0x1c/0x30
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931310] [<ffffffff81665609>] smp_apic_timer_interrupt+0x69/0x99
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931313] [<ffffffff816634de>] apic_timer_interrupt+0x6e/0x80
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931314] <EOI> [<ffffffff8165a629>] ? _raw_spin_unlock_irqrestore+0x19/0x30
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931318] [<ffffffff8142f3b4>] scsi_remove_target+0xb4/0xe0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931322] [<ffffffffa00a2e65>] sas_rphy_remove+0x65/0x70 [scsi_transport_sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931326] [<ffffffffa00a3d96>] sas_rphy_delete+0x16/0x30 [scsi_transport_sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931330] [<ffffffffa00a3dda>] sas_port_delete+0x2a/0x160 [scsi_transport_sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931333] [<ffffffff811ee476>] ? sysfs_remove_link+0x26/0x30
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931337] [<ffffffffa00c77fa>] mpt2sas_transport_port_remove+0x14a/0x1e0 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931342] [<ffffffffa00c0e06>] _scsih_remove_device.part.48+0x136/0x140 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931347] [<ffffffffa00c42f1>] _scsih_sas_topology_change_event.isra.58+0x281/0x350 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931351] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931356] [<ffffffffa00c4599>] _firmware_event_work+0x1d9/0x300 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931358] [<ffffffff8105609a>] ? finish_task_switch+0x4a/0xf0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931363] [<ffffffffa00c43c0>] ? _scsih_sas_topology_change_event.isra.58+0x350/0x350 [mpt2sas]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931366] [<ffffffff81084a6a>] process_one_work+0x11a/0x480
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931369] [<ffffffff81085814>] worker_thread+0x164/0x370
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931371] [<ffffffff810856b0>] ? manage_workers.isra.30+0x130/0x130
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931374] [<ffffffff8108a03c>] kthread+0x8c/0xa0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931376] [<ffffffff81664b74>] kernel_thread_helper+0x4/0x10
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931379] [<ffffffff81089fb0>] ? flush_kthread_worker+0xa0/0xa0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931381] [<ffffffff81664b70>] ? gs_change+0x13/0x13
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931383] NMI backtrace for cpu 3
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931384] CPU 3
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931385] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931414]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931416] Pid: 0, comm: swapper/3 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931419] RIP: 0010:[<ffffffff8136ba5f>] [<ffffffff8136ba5f>] intel_idle+0xbf/0x150
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931423] RSP: 0018:ffff8802121c5e38 EFLAGS: 00000046
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931424] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931426] RDX: 0000000000000000 RSI: ffff8802121c5fd8 RDI: ffffffff81c12e00
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931428] RBP: ffff8802121c5e98 R08: 00000000fffffffc R09: 0000000000000f9e
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931430] R10: 0000000000001f21 R11: 0000000000000000 R12: 0000000000000003
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931431] R13: ffff88021fd9ac70 R14: 0000000000000020 R15: 12b249087f5451ca
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931433] FS: 0000000000000000(0000) GS:ffff88021fd80000(0000) knlGS:0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931435] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931437] CR2: 00007f70a894c000 CR3: 0000000001c05000 CR4: 00000000000406e0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931439] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931440] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931442] Process swapper/3 (pid: 0, threadinfo ffff8802121c4000, task ffff8802121a1700)
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931444] Stack:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931539] 0000000000000000 00000000003ce5da ffff8802121c5e98 ffffffff81506fb7
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931543] 0000000000000000 00000003003ce5da 0000000000000000 ffff88021fd9ac70
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931546] 00000000fffffff0 0000000000000003 0000000000000003 0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931549] Call Trace:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931671] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931673] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931677] [<ffffffff8101222a>] cpu_idle+0xca/0x120
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931680] [<ffffffff81637fed>] start_secondary+0xd9/0xdb
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.931682] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> cc 93 d2 ff 4c 29 f8 48 89 c7 e8 b1 11 d0 ff 4c 69 f0 40 42
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933089] Call Trace:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933091] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933093] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933096] [<ffffffff8101222a>] cpu_idle+0xca/0x120
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933098] [<ffffffff81637fed>] start_secondary+0xd9/0xdb
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933100] NMI backtrace for cpu 0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933102] CPU 0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933103] Modules linked in: xfs ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933160]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933162] Pid: 0, comm: swapper/0 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933167] RIP: 0010:[<ffffffff8136ba5f>] [<ffffffff8136ba5f>] intel_idle+0xbf/0x150
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933173] RSP: 0018:ffffffff81c01e18 EFLAGS: 00000046
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933175] RAX: 0000000000000010 RBX: 0000000000000004 RCX: 0000000000000001
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933178] RDX: 0000000000000000 RSI: ffffffff81c01fd8 RDI: ffffffff81c12e00
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933181] RBP: ffffffff81c01e78 R08: 00000000fffffffd R09: 0000000000000f99
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933184] R10: 0000000000001eb7 R11: 0000000000000000 R12: 0000000000000002
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933186] R13: ffff88021fc1ac70 R14: 0000000000000010 R15: 12b249087f546234
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933189] FS: 0000000000000000(0000) GS:ffff88021fc00000(0000) knlGS:0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933192] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933195] CR2: 00007ffb58064000 CR3: 0000000001c05000 CR4: 00000000000406f0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933198] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933201] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933204] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c0d020)
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.933206] Stack:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.940315] 0000000000000000 00000000003cd540 ffffffff81c01e78 ffffffff81506fb7
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.940319] 0000000000000000 00000000003cd540 0000000000000000 ffff88021fc1ac70
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.940322] 00000000fffffff0 0000000000000002 0000000000000000 000000000008c800
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.940325] Call Trace:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.947289] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.947292] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.947295] [<ffffffff8101222a>] cpu_idle+0xca/0x120
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.947299] [<ffffffff81620c0e>] rest_init+0x72/0x74
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.947302] [<ffffffff81cfbc03>] start_kernel+0x3b0/0x3bd
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.947305] [<ffffffff81cfb388>] x86_64_start_reservations+0x132/0x136
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.947308] [<ffffffff81cfb140>] ? early_idt_handlers+0x140/0x140
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.947311] [<ffffffff81cfb459>] x86_64_start_kernel+0xcd/0xdc
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.947312] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> cc 93 d2 ff 4c 29 f8 48 89 c7 e8 b1 11 d0 ff 4c 69 f0 40 42
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969301] Call Trace:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969304] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969306] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969309] [<ffffffff8101222a>] cpu_idle+0xca/0x120
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969311] [<ffffffff81620c0e>] rest_init+0x72/0x74
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969314] [<ffffffff81cfbc03>] start_kernel+0x3b0/0x3bd
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969316] [<ffffffff81cfb388>] x86_64_start_reservations+0x132/0x136
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969319] [<ffffffff81cfb140>] ? early_idt_handlers+0x140/0x140
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969322] [<ffffffff81cfb459>] x86_64_start_kernel+0xcd/0xdc
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969324] NMI backtrace for cpu 2
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969325] CPU 2
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969326] Modules linked in: xfs
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.969327] INFO: rcu_sched detected stalls on CPUs/tasks: { 1} (detected by 0, t=15010 jiffies)
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984392] ip6table_filter ip6_tables ebtable_nat ebtables ipt_MASQUERADE iptable_nat nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack ipt_REJECT xt_CHECKSUM iptable_mangle xt_tcpudp iptable_filter ip_tables x_tables vesafb kvm_intel kvm binfmt_misc nfsd nfs lockd fscache auth_rpcgss nfs_acl sunrpc dm_multipath bridge stp joydev shpchp mac_hid lp parport raid10 raid456 async_pq async_xor xor async_memcpy async_raid6_recov ixgbe dca mdio mpt2sas scsi_transport_sas raid_class e1000e raid6_pq async_tx raid1 usbhid hid raid0 multipath linear
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984423]
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984425] Pid: 0, comm: swapper/2 Not tainted 3.2.0-30-generic #48-Ubuntu TYAN S5510/S5510
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984428] RIP: 0010:[<ffffffff8136ba5f>] [<ffffffff8136ba5f>] intel_idle+0xbf/0x150
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984432] RSP: 0018:ffff880212189e38 EFLAGS: 00000046
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984434] RAX: 0000000000000020 RBX: 0000000000000008 RCX: 0000000000000001
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984436] RDX: 0000000000000000 RSI: ffff880212189fd8 RDI: ffffffff81c12e00
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984438] RBP: ffff880212189e98 R08: 00000000fffffffc R09: 0000000000000f9b
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984440] R10: 0000000000001ef1 R11: 0000000000000000 R12: 0000000000000003
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984442] R13: ffff88021fd1ac70 R14: 0000000000000020 R15: 12b249087f545cba
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984445] FS: 0000000000000000(0000) GS:ffff88021fd00000(0000) knlGS:0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984447] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984449] CR2: 00007f9905b7e000 CR3: 0000000001c05000 CR4: 00000000000406e0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984451] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984453] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984455] Process swapper/2 (pid: 0, threadinfo ffff880212188000, task ffff880212181700)
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.984457] Stack:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.991838] 0000000000000000 00000000003cdb87 ffff880212189e98 ffffffff81506fb7
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.991842] 0000000000000000 00000002003cdb87 0000000000000000 ffff88021fd1ac70
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.991846] 00000000fffffff0 0000000000000003 0000000000000002 0000000000000000
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.991850] Call Trace:
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.998926] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.998928] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.998930] [<ffffffff8101222a>] cpu_idle+0xca/0x120
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.998932] [<ffffffff81637fed>] start_secondary+0xd9/0xdb
> Sep 9 20:38:59 dev-storage1 kernel: [ 897.998933] Code: 28 e0 ff ff 83 e2 08 75 22 31 d2 48 83 c0 10 48 89 d1 0f 01 c8 0f ae f0 48 8b 86 38 e0 ff ff a8 08 75 08 b1 01 4c 89 f0 0f 01 c9 <e8> cc 93 d2 ff 4c 29 f8 48 89 c7 e8 b1 11 d0 ff 4c 69 f0 40 42
> Sep 9 20:38:59 dev-storage1 kernel: [ 898.021107] Call Trace:
> Sep 9 20:38:59 dev-storage1 kernel: [ 898.021109] [<ffffffff81506fb7>] ? menu_select+0xe7/0x2c0
> Sep 9 20:38:59 dev-storage1 kernel: [ 898.021111] [<ffffffff81505bf1>] cpuidle_idle_call+0xc1/0x280
> Sep 9 20:38:59 dev-storage1 kernel: [ 898.021113] [<ffffffff8101222a>] cpu_idle+0xca/0x120
> Sep 9 20:38:59 dev-storage1 kernel: [ 898.021114] [<ffffffff81637fed>] start_secondary+0xd9/0xdb
> Sep 9 20:39:27 dev-storage1 kernel: [ 925.653613] BUG: soft lockup - CPU#1 stuck for 23s! [kworker/u:12:1602]
> ... etc
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list