[Gluster-users] Apache hanging

[Neil Garb]

Hi there

I have a 4-node GlusterFS (3.0.2 -- the one bundled with Ubuntu 10.04)
cluster set up in RAID 10, and it's working beautifully.  I have an
rsync job that is busy copying about 7TB onto that cluster, and I can
mount the cluster and browse etc., so it seems to be configured
correctly.

I am having big issues with Apache, though.  Whenever I try to access
the cluster or write to it -- even small files -- the Apache process
hangs.  After 2 minutes, I get this in /var/log/messages:

[ 6120.883314] INFO: task apache2:3349 blocked for more than 120 seconds.
[ 6120.883321] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
disables this message.
[ 6120.883326] apache2       D 0000000000000000     0  3349      1 0x00000004
[ 6120.883342]  ffff8801ddaa7e48 0000000000000282 ffffffff81662ca0
ffff8801ddaa7dc8
[ 6120.883345]  0000000000000000 ffff8801ddaa7e10 ffff8801a1386478
ffff8801ddaa7fd8
[ 6120.883347]  ffff8801a13860c0 ffff8801a13860c0 ffff8801a13860c0
ffff8801ddaa7fd8
[ 6120.883350] Call Trace:
[ 6120.883352]  [<ffffffff81059e80>] ? autoremove_wake_function+0x0/0x40
[ 6120.883357]  [<ffffffffa005ebd5>] request_wait_answer+0xd5/0x290 [fuse]
[ 6120.883360]  [<ffffffff81059e80>] ? autoremove_wake_function+0x0/0x40
[ 6120.883364]  [<ffffffffa005ee14>] fuse_request_send+0x84/0xa0 [fuse]
[ 6120.883368]  [<ffffffffa00627d4>] fuse_setlk+0x164/0x1b0 [fuse]
[ 6120.883372]  [<ffffffffa0062a44>] fuse_file_flock+0x34/0x50 [fuse]
[ 6120.883375]  [<ffffffff8112f3df>] sys_flock+0x10f/0x1d0
[ 6120.883378]  [<ffffffff81009ba8>] system_call_fastpath+0x16/0x1b
[ 6120.883381]  [<ffffffff81009b40>] ? system_call+0x0/0x52

I have mounted the cluster on that Apache server using the glusterfs
command without --disable-direct-io-mode.

I have done some searching and found mod_glusterfs, but I've also read
that that's deprecated.  Can anyone see something that I'm doing
wrong?

- Neil