[Gluster-users] FUSE I/O Lockup while accessing glusterfs shares

[Hugo Cisneiros (Eitch)]

Hi,

We have a web server installation that is using glusterFS. There are 3
GlusterFS servers and 4 GlusterFS clients acessing one share. We're
having a problem that randomly, the httpd processes begin to hung
while waiting for I/O. This begins to happen with the following kernel
call trace:

kernel: INFO: task httpd.worker:18974 blocked for more than 120 seconds.
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
this message.
kernel: httpd.worker  D ffffffff80150462     0 18974   4818
18975 18972 (NOTLB)
kernel: ffff8108c0ee7c78 0000000000000086 ffff8101f3b2b000 ffff810116d0f200
kernel: ffff81037392c6d8 000000000000000a ffff8106b73d57a0 ffff810116dd4100
kernel: 00004f0baea8a21e 00000000000026f5 ffff8106b73d5988 0000000300000000
kernel: Call Trace:
kernel: [<ffffffff88443cda>] :fuse:fuse_dentry_revalidate+0x13e/0x1c1
kernel: [<ffffffff80063c6f>] __mutex_lock_slowpath+0x60/0x9b
kernel: [<ffffffff80063cb9>] .text.lock.mutex+0xf/0x14
kernel: [<ffffffff8000cec2>] do_lookup+0x90/0x1e6
kernel: [<ffffffff80009bb2>] __link_path_walk+0x3a6/0xf42
kernel: [<ffffffff8000e9e2>] link_path_walk+0x42/0xb2
kernel: [<ffffffff8000ccb2>] do_path_lookup+0x275/0x2f1
kernel: [<ffffffff8001280e>] getname+0x15b/0x1c2
kernel: [<ffffffff80023876>] __user_walk_fd+0x37/0x4c
kernel: [<ffffffff80028846>] vfs_stat_fd+0x1b/0x4a
kernel: [<ffffffff8000c5fe>] _atomic_dec_and_lock+0x39/0x57
kernel: [<ffffffff800235a8>] sys_newstat+0x19/0x31
kernel: [<ffffffff80012b54>] __fput+0x191/0x1bd
kernel: [<ffffffff8002c9e4>] mntput_no_expire+0x19/0x89
kernel: [<ffffffff80023b84>] filp_close+0x5c/0x64
kernel: [<ffffffff8001dfa6>] sys_close+0x88/0xbd
kernel: [<ffffffff8005d116>] system_call+0x7e/0x83

Looking at the glusterfs client logs, the following entries begin to
show up before this call trace begins:

[fuse-bridge.c:585:fuse_lookup] glusterfs-fuse: 89023783: LOOKUP
46912563008752/2010 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024196: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024203: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024207: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024210: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024213: GETATTR
46912563008752 (fuse_loc_fill() failed)
[fuse-bridge.c:793:fuse_getattr] glusterfs-fuse: 89024218: GETATTR
46912563008752 (fuse_loc_fill() failed)

The same message repeats 1116 times. Talking with a friend, we noticed
that the 4 web servers are reading *and* writing very concurrently and
because of this, there are many lookups, reads and unlinks returning
"no such file or directory" on the log, like these:

[fuse-bridge.c:859:fuse_fd_cbk] glusterfs-fuse: 88969079: OPEN()
/cache/70ea5f0d4df01648be4f70d9813f24b7.meta => -1 (No such file or
directory)
[fuse-bridge.c:1215:fuse_unlink_cbk] glusterfs-fuse: 88969080:
UNLINK() /cache/70ea5f0d4df01648be4f70d9813f24b7.meta => -1 (No such
file or directory)
[fuse-bridge.c:491:fuse_entry_cbk] glusterfs-fuse:
LOOKUP(/cache/index.html) inode (ptr=0x2aaab313f9f0, ino=1528908,
gen=5480759143103581058) found conflict (ptr=0x2aaadebc8690,
ino=1528908, gen=5480759143103581058)

After the kernel call trace appears, the fuse filesystem becomes very
unresponsive and unstable. Who tries to read the mounted directory,
randomly gets its process locked (with status waiting for I/O) and
hangs forever. We know that maybe glusterfs is not made for high
concurrency for both writing and reading, so we're working on getting
an alternate solution for this behavior (it's a disk cache, we're
transfering for something like memcached). But maybe this error helps
to find something unexpected going on... :)

Here's the client configuration:

volume vol01
  type protocol/client
  option transport-type tcp
  option remote-host server01.domain
  option remote-subvolume vol
end-volume

volume vol02
  type protocol/client
  option transport-type tcp
  option remote-host server02.domain
  option remote-subvolume vol
end-volume

volume vol03
  type protocol/client
  option transport-type tcp
  option remote-host server03.domain
  option remote-subvolume vol
end-volume

volume vol-mirror
    type cluster/replicate
    subvolumes vol01 vol02 vol03
end-volume

volume vol-iocache
  type performance/io-cache
  option cache-size 128MB
  option cache-timeout 2
  subvolumes vol-mirror
end-volume

volume vol-quickread
    type performance/quick-read
    option cache-timeout 1
    option max-file-size 64kB
    subvolumes vol-iocache
end-volume

volume vol-writebehind
    type performance/write-behind
    option cache-size 4MB
    subvolumes vol-quickread
end-volume

volume vol-statprefetch
    type performance/stat-prefetch
    subvolumes vol-writebehind
end-volume

Thanks,

-- 
[]'s
Hugo
www.devin.com.br