[Gluster-users] machine lockup on heavy load

[Arend-Jan Wijtzes]

Guys,

I thought I share this on the list because we're using GlusterFS supplied patched
fuse lib. I'm going to try the latest fuse lib now and see if it fixes things. 
There's nothing in the gluster logs. 


fuse-2.7.3glfs10
glusterfs--mainline--3.0--patch-359.


>From the system log:

 BUG: unable to handle kernel paging request at virtual address 00100104
  printing eip:
 f8ab3426
 *pde = 254f2001
 Oops: 0002 [#1]
 SMP 
 Modules linked in: fuse w83627hf w83781d hwmon_vid hwmon i2c_isa i2c_piix4 i2c_i801 i2c_dev i2c_core
 CPU:    0
 EIP:    0060:[pg0+944403494/1067394048]    Not tainted VLI
 EFLAGS: 00010246   (2.6.18 #5) 
 EIP is at request_end+0x3b/0xe9 [fuse]
 eax: 00100100   ebx: d301b320   ecx: d301b328   edx: 00200200
 esi: e5e56480   edi: 00000000   ebp: e5103f10   esp: e5103ef4
 ds: 007b   es: 007b   ss: 0068
 Process glusterfs (pid: 8196, ti=e5102000 task=e6b82560 task.ti=e5102000)
 Stack: d301b320 e5e56480 00000000 f8ab4157 e5e56480 d301b320 00000030 e5e56480 
        00000001 d301b320 e5103f6c 00000000 00100008 b6cba000 c3436660 00000000 
        fffb9038 00000fc8 00000001 e5ce3ec0 b6cb9008 e5103fa4 00101000 f8ab41c5 
 Call Trace:
  [pg0+944406871/1067394048] fuse_dev_readv+0x1a8/0x1f1 [fuse]
  [pg0+944406981/1067394048] fuse_dev_read+0x25/0x29 [fuse]
  [vfs_read+136/286] vfs_read+0x88/0x11e
  [sys_read+59/99] sys_read+0x3b/0x63
  [syscall_call+7/11] syscall_call+0x7/0xb
 Code: 8b bb 64 01 00 00 8d 4b 08 c7 83 64 01 00 00 00 00 00 00 89 02 89 50 04 c7 03 00 01 10 00 8b 43 08 c7 43 04 00 02 20 00 8b 51 04 <89> 50 04 89 02 c7 41 04 00 0
0 01 
 EIP: [pg0+944403494/1067394048] request_end+0x3b/0xe9 [fuse] SS:ESP 0068:e5103ef4
  <3>BUG: soft lockup detected on CPU#0!
  [softlockup_tick+154/168] softlockup_tick+0x9a/0xa8
  [update_process_times+56/93] update_process_times+0x38/0x5d
  [smp_apic_timer_interrupt+90/97] smp_apic_timer_interrupt+0x5a/0x61
  [apic_timer_interrupt+31/36] apic_timer_interrupt+0x1f/0x24
  [__func__.3+27/113] _spin_lock+0xa/0xf
  [pg0+944404797/1067394048] request_send+0x15/0x50 [fuse]
  [pg0+944416580/1067394048] fuse_flush+0xcd/0xf1 [fuse]
  [filp_close+53/87] filp_close+0x35/0x57
  [sys_close+89/109] sys_close+0x59/0x6d
  [syscall_call+7/11] syscall_call+0x7/0xb
 BUG: soft lockup detected on CPU#1!
  [softlockup_tick+154/168] softlockup_tick+0x9a/0xa8
  [update_process_times+56/93] update_process_times+0x38/0x5d
  [smp_apic_timer_interrupt+90/97] smp_apic_timer_interrupt+0x5a/0x61
  [apic_timer_interrupt+31/36] apic_timer_interrupt+0x1f/0x24
  [__func__.3+27/113] _spin_lock+0xa/0xf
  [pg0+944404873/1067394048] request_send_nowait+0x11/0x53 [fuse]
  [pg0+944417777/1067394048] fuse_readpages+0x7f/0xa0 [fuse]
  [read_pages+37/197] read_pages+0x25/0xc5
  [get_page_from_freelist+112/136] get_page_from_freelist+0x70/0x88
  [__alloc_pages+96/651] __alloc_pages+0x60/0x28b
  [__do_page_cache_readahead+277/309] __do_page_cache_readahead+0x115/0x135
  [blockable_page_cache_readahead+70/149] blockable_page_cache_readahead+0x46/0x95
  [make_ahead_window+120/153] make_ahead_window+0x78/0x99
  [page_cache_readahead+282/312] page_cache_readahead+0x11a/0x138
  [do_generic_mapping_read+282/1029] do_generic_mapping_read+0x11a/0x405
  [__generic_file_aio_read+408/457] __generic_file_aio_read+0x198/0x1c9
  [file_read_actor+0/234] file_read_actor+0x0/0xea
  [generic_file_read+172/197] generic_file_read+0xac/0xc5
  [autoremove_wake_function+0/58] autoremove_wake_function+0x0/0x3a
  [border+32/96] mutex_lock+0xf/0x1a
  [generic_file_llseek+145/159] generic_file_llseek+0x91/0x9f
  [vfs_read+136/286] vfs_read+0x88/0x11e
  [sys_read+59/99] sys_read+0x3b/0x63
  [syscall_call+7/11] syscall_call+0x7/0xb