[Gluster-users] The continuing story ...

Wed Sep 9 17:47:07 UTC 2009

So I'm really wondering, what is it that is making you believe that
you are going to get a solution for this particular problem on the
gluster mailing lists? You clearly have soft lockup backtraces from
the kernel pointing to various kernel subsystems. Have you even
reported this log dump to the linux-kernel mailing list? That is the
first thing anybody does when they have a kernel backtrace in dmesg -
report to LKML. Unlike what you have been telling that all you have in
your kernel log is just a single line indicating a soft lockup, your
kernel log below has a whole wealth of info and backtraces which are
going to help you get the issue diagnosed a lot faster on the LKML
than here. Apparantly they have not appeared to you as backtraces, but
there are backtraces some of which go through tcpip ARP table
management, ACPI (power save?) cpu shutdown, etc. I fail to see how
any of these are related to glusterfs. No amount of messed up pointers
or race conditions in glusterfsd can ever result in your kernel
landing in this state. This is not windows 3.1 where apps and kernel
run in the same address space. Just because glusterfs was being used
at the time of this lockup, you are putting your efforts in trying to
get a solution on this list, without ever working on the first direct
evidence you have at hand - these kernel backtraces.

Please reply back to this thread only after you have a response from
the appropriate kernel developer indicating that the cause of this
lockup is because of a misbehaving userspace application. After that,
let us give you the benefit of doubt that the misbehaving userspace
process is glusterfsd and then continue any further debugging. It is
not that we do not want to help you, but we really are pointing you to
the right place where your problem can actually get fixed. You have
all the necessary input they need.

Avati

> > Yes I did go through that. Do you still have the complete kernel log?
> > The lines before the "CPU soft lockup" message should be helpful in
> > debugging it further.
> >
>
>  here you have it, almost all are for wrf.exe (except swapper on
>  cpu 3)
>
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] BUG: soft lockup - CPU#1
> stuck for 7781s! [wrf.exe:25309]
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] CPU 1:
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Pid: 25309, comm: wrf.exe
> Not tainted 2.6.26-2-amd64 #1
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RIP:
> 0010:[<ffffffff8020be6b>]  [<ffffffff8020be6b>]
> system_call_after_swapgs+0x2b/0x8f
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RSP: 0018:ffff810215ebdf88
>  EFLAGS: 00000286
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RAX: 0000000000000018 RBX:
> 0000000000000000 RCX: 00002ac4a1f6c657
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RDX: 00002ac4a168da68 RSI:
> 0000000000000000 RDI: 0000000000000001
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RBP: ffff81022d54eda0 R08:
> 0000000002bc5fd8 R09: 00000000084b31b8
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] R10: 00002ac4a26590c0 R11:
> 0000000000000246 R12: ffff81022d54eb10
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] R13: ffff81022d54eb10 R14:
> ffff810142afb560 R15: ffff8101fd9958e0
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] FS: 00002ac4a2627b10(0000)
> GS:ffff81022f0928c0(0000) knlGS:0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] CS:  0010 DS: 0000 ES:
> 0000 CR0: 0000000080050033
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] CR2: 0000000002ac1528 CR3:
> 000000021507a000 CR4: 00000000000006e0
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] DR3: 0000000000000000 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Call Trace:
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]  [<ffffffff8020beca>] ?
> system_call_after_swapgs+0x8a/0x8f
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] BUG: soft lockup - CPU#2
> stuck for 7780s! [wrf.exe:25592]
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] CPU 2:
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Pid: 25592, comm: wrf.exe
> Not tainted 2.6.26-2-amd64 #1
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RIP:
> 0033:[<000000000104e972>]  [<000000000104e972>]
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RSP: 002b:00007fff32aff6a0
>  EFLAGS: 00000287
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RAX: 0000000000000080 RBX:
> 00007fff32b130e0 RCX: 0000000000002448
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RDX: 00007fff32b01ee8 RSI:
> 0000000000000012 RDI: 00002ba977064aac
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RBP: 0000000000000000 R08:
> 00002ba977064868 R09: 00007fff32b01ee8
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] R10: 0000000000000081 R11:
> 00002ba976e61624 R12: 000000000000b258
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] R13: 00000000001b6db0 R14:
> ffffffff80230754 R15: ffff81022b9e1f78
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] FS: 00002ba972258b10(0000)
> GS:ffff81022f0920c0(0000) knlGS:0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] CS:  0010 DS: 0000 ES:
> 0000 CR0: 0000000080050033
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] CR2: 00002ba972c74000 CR3:
> 0000000215091000 CR4: 00000000000006e0
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] DR3: 0000000000000000 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Call Trace:
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] BUG: soft lockup - CPU#3
> stuck for 7781s! [swapper:0]
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] CPU 3:
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Pid: 0, comm: swapper Not
> tainted 2.6.26-2-amd64 #1
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RIP:
> 0010:[<ffffffff8021eb68>]  [<ffffffff8021eb68>] native_safe_halt+0x2/0x3
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RSP: 0018:ffff81022f13def8
>  EFLAGS: 00000246
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RAX: ffff81022f13dfd8 RBX:
> ffff81022c5874a8 RCX: 0000000000000808
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RDX: 0000000000000808 RSI:
> ffff81022c587090 RDI: ffff81022c587020
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] RBP: 00000001141d861b R08:
> ffff8100010536a0 R09: 0000000001141d86
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] R10: 0000000000000000 R11:
> ffff810114362da0 R12: ffff81022f114000
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] R13: ffff810114362da0 R14:
> ffffffff80248fe9 R15: 0000000000000092
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] FS: 0000000000000000(0000)
> GS:ffff81022f0a77c0(0000) knlGS:0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] CS:  0010 DS: 0018 ES:
> 0018 CR0: 000000008005003b
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] CR2: 0000000002ac1528 CR3:
> 00000001142d7000 CR4: 00000000000006e0
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] DR3: 0000000000000000 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]
>  Aug 19 07:04:51 server2 kernel: [1422975.972778] Call Trace:
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]  [<ffffffff803aa693>] ?
> menu_select+0x62/0x7f
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]  [<ffffffffa000be19>] ?
> :processor:acpi_safe_halt+0x2b/0x44
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]  [<ffffffffa000bee5>] ?
> :processor:acpi_idle_enter_c1+0xb3/0x112
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]  [<ffffffff803a9b4b>] ?
> cpuidle_idle_call+0x7a/0xaf
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]  [<ffffffff803a9ad1>] ?
> cpuidle_idle_call+0x0/0xaf
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]  [<ffffffff8020ac79>] ?
> cpu_idle+0x89/0xb3
>  Aug 19 07:04:51 server2 kernel: [1422975.972778]
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] BUG: soft lockup - CPU#4
> stuck for 7780s! [wrf.exe:25593]
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] CPU 4:
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] Pid: 25593, comm: wrf.exe
> Not tainted 2.6.26-2-amd64 #1
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] RIP:
> 0033:[<0000000000d3dfcc>]  [<0000000000d3dfcc>]
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] RSP: 002b:00007fff37a9e7a0
>  EFLAGS: 00000287
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] RAX: 000000000c537530 RBX:
> 00007fff37aa4360 RCX: 000000000b3fd060
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] RDX: 000000000c544c80 RSI:
> 00002b8172938e6c RDI: 00002b8172938c2c
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] RBP: ffffffff8020c53e R08:
> 00002b8174738c2c R09: 00002b8174338c2c
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] R10: 00002b8174538c2c R11:
> 00002b8173138c2c R12: 00007fff37aa4360
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] R13: 000000000af96610 R14:
> 0000000000000072 R15: 00002b8171f6e004
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] FS: 00002b816d384b10(0000)
> GS:ffff81022f126ec0(0000) knlGS:0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] CS:  0010 DS: 0000 ES:
> 0000 CR0: 0000000080050033
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] CR2: 00002b816d6b6300 CR3:
> 000000022ddad000 CR4: 00000000000006e0
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] DR3: 0000000000000000 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
>  Aug 19 07:04:51 server2 kernel: [1422975.984739]
>  Aug 19 07:04:51 server2 kernel: [1422975.984739] Call Trace:
>  Aug 19 07:04:51 server2 kernel: [1422975.984739]
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] BUG: soft lockup - CPU#5
> stuck for 7781s! [wrf.exe:25594]
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] CPU 5:
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] Pid: 25594, comm: wrf.exe
> Not tainted 2.6.26-2-amd64 #1
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] RIP:
> 0033:[<0000000000d3ded6>]  [<0000000000d3ded6>]
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] RSP: 002b:00007fff3cb2ca10
>  EFLAGS: 00000287
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] RAX: 0000000013b87710 RBX:
> 00007fff3cb32660 RCX: 000000000f0b0710
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] RDX: 0000000013b94fd0 RSI:
> 000000000d8ea620 RDI: 000000000d8ea3dc
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] RBP: 0000000000000000 R08:
> 000000000f93657c R09: 000000000f530ebc
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] R10: 000000000f733a1c R11:
> 000000000e1a7f9c R12: 0000000000009dde
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] R13: 00000000076e1180 R14:
> ffffffff80230754 R15: ffff8101ff095f78
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] FS: 00002ad768238b10(0000)
> GS:ffff81022f1266c0(0000) knlGS:0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] CS:  0010 DS: 0000 ES:
> 0000 CR0: 0000000080050033
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] CR2: 00002ad768c54000 CR3:
> 000000021510b000 CR4: 00000000000006e0
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] DR3: 0000000000000000 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
>  Aug 19 07:04:51 server2 kernel: [1422975.990964]
>  Aug 19 07:04:51 server2 kernel: [1422975.990964] Call Trace:
>  Aug 19 07:04:51 server2 kernel: [1422975.990964]
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] BUG: soft lockup - CPU#6
> stuck for 7780s! [wrf.exe:25591]
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] CPU 6:
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] Modules linked in: nfs
> nfsd lockd nfs_acl auth_rpcgss sunrpc exportfs iptable_mangle iptable_nat
> nf_nat nf_conntrack_ipv4 nf_conntrack iptable_filter ip_tables x_tables
> cpufreq_ondemand cpufreq_userspace cpufreq_powersave cpufreq_stats
> freq_table cpufreq_conservative ipv6 fuse dm_snapshot dm_mirror dm_log
> dm_mod loop snd_pcsp snd_pcm rng_core shpchp i5000_edac edac_core psmouse
> snd_timer pci_hotplug snd soundcore snd_page_alloc serio_raw evdev button
> dcdbas usbhid hid ff_memless ext3 jbd mbcache sg sr_mod cdrom ata_generic
> ide_pci_generic ide_core sd_mod ata_piix uhci_hcd bnx2 libata dock ehci_hcd
> mptsas mptscsih mptbase scsi_transport_sas scsi_mod firmware_class thermal
> processor fan thermal_sys
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] Pid: 25591, comm: wrf.exe
> Not tainted 2.6.26-2-amd64 #1
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] RIP:
> 0010:[<ffffffff8042957d>]  [<ffffffff8042957d>]
> _spin_unlock_irqrestore+0x7/0xe
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] RSP: 0000:ffff81022f203c48
>  EFLAGS: 00000286
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] RAX: ffffffff805234e0 RBX:
> 0000000000000001 RCX: 0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] RDX: ffffffff806709b0 RSI:
> 0000000000000286 RDI: 0000000000000286
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] RBP: ffff81022f203bc0 R08:
> 00000000000000c1 R09: ffff81022b977700
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] R10: 000000000000012c R11:
> ffffffff803fa1ef R12: ffffffff8020ccf2
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] R13: ffff81022f203bc0 R14:
> ffff81022d1cf558 R15: ffff810001077880
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] FS: 00002b77bbbdfb10(0000)
> GS:ffff81022f1e0dc0(0000) knlGS:0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] CS:  0010 DS: 0000 ES:
> 0000 CR0: 0000000080050033
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] CR2: 00002b677b245000 CR3:
> 000000022cd4f000 CR4: 00000000000006e0
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] DR0: 0000000000000000 DR1:
> 0000000000000000 DR2: 0000000000000000
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] DR3: 0000000000000000 DR6:
> 00000000ffff0ff0 DR7: 0000000000000400
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]
>  Aug 19 07:04:51 server2 kernel: [1422975.994958] Call Trace:
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  <IRQ>
> [<ffffffff8023d2c1>] ? del_timer+0x56/0x5f
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff80248f18>] ?
> hrtimer_interrupt+0x123/0x159
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff80248d4a>] ?
> ktime_get_ts+0x22/0x4b
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff803bfe9e>] ?
> neigh_del_timer+0x18/0x41
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff803c0067>] ?
> neigh_update+0x1a0/0x3e7
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff803fa1a1>] ?
> arp_process+0x50c/0x55a
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff803b58f2>] ?
> __alloc_skb+0x7f/0x12d
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffffa00bf5e9>] ?
> :bnx2:bnx2_poll+0xdf0/0x103b
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff80219dc1>] ?
> native_smp_call_function_mask+0xd9/0x108
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff80249cc1>] ?
> notes_read+0x8/0x14
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff80231257>] ?
> task_tick_fair+0x22/0x8d
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff8024ac46>] ?
> getnstimeofday+0x39/0x98
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff803bc36e>] ?
> net_rx_action+0xab/0x1da
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff802393eb>] ?
> __do_softirq+0x5c/0xd1
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff8020d2cc>] ?
> call_softirq+0x1c/0x28
>  Aug 19 07:04:51 server2 kernel: [1422975.994958]  [<ffffffff8020f3d0>] ?
> do_softirq+0x3c/0x81
>  Aug 19 07:04:51 server2 kernel: [1422976.003381]  [<ffffffff8023934b>] ?
> irq_exit+0x3f/0x83
>  Aug 19 07:04:51 server2 kernel: [1422976.003381]  [<ffffffff8020f630>] ?
> do_IRQ+0xb9/0xd9
>  Aug 19 07:04:51 server2 kernel: [1422976.003381]  [<ffffffff8020c46d>] ?
> ret_from_intr+0x0/0x19
>  Aug 19 07:04:51 server2 kernel: [1422976.003381]  <EOI>
>