[Gluster-devel] 2.0.9 hanging bug

[Brian Hirt]

So 3.0.5 has turned out to be even more of a nightmare than 2.0.9 -- at least 2.0.9 worked until a machine shutdown unexpectedly.

At the stroke of midnight all my clients got stale nfs handles and were no longer able to access files on the filesystem.   Had to kill glusterfs and all programs with open files, unmount and remount.

Here is a sample of the errors in the client logs:
	[2010-09-23 00:00:00] W [fuse-bridge.c:725:fuse_attr_cbk] glusterfs-fuse: 3408010: LOOKUP() / => -1 (Stale NFS file handle)
	[2010-09-23 00:00:16] W [fuse-bridge.c:725:fuse_attr_cbk] glusterfs-fuse: 3408056: LOOKUP() / => -1 (Stale NFS file handle)

There are no errors or warnings in the server logs.

clients are running 3.0.5 and ubuntu 9.04/9.10

does anyone have any idea what going on?  i'm losing all confidence is this product.


On Sep 22, 2010, at 11:35 AM, Brian Hirt wrote:

> We have a four node set up where remote1/remote2 and remote3/remote4 replicate and then the filesystem is distributed between them.  Well our remote3 crashed hard and all of the clients that mounted the filesystem started to hang when trying to access anything on the mount point.  
> 
> I never expected this to happen -- the whole point of a setup like this is to allow for a machine to fail and still have your filesystem available and accessible.
> 
> I've upgraded to 3.0.5 in the hopes that it's more reliable, but need to ask:   Does anyone know if this bug has been fixed in the 3.x branch?  
> 
> Please advise
> 
> Thanks,
> 
> Brian
> 
> Client log entries at start of failure
> [2010-09-21 20:51:24] E [client-protocol.c:309:call_bail] remote3: bailing out frame WRITE(14) frame sent = 2010-09-21 20:21:23. frame-timeout = 1800
> [2010-09-21 20:51:24] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:21:23. frame-timeout = 1800
> [2010-09-21 20:51:24] W [client-protocol.c:6045:protocol_client_interpret] remote3: no frame for callid=629454 type=4 op=36
> [2010-09-21 20:51:44] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:21:41. frame-timeout = 1800
> [2010-09-21 20:51:44] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:21:41. frame-timeout = 1800
> [2010-09-21 20:52:04] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:22:01. frame-timeout = 1800
> [2010-09-21 20:52:04] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:22:01. frame-timeout = 1800
> [2010-09-21 20:52:34] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:22:28. frame-timeout = 1800
> [2010-09-21 20:52:34] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:22:28. frame-timeout = 1800
> [2010-09-21 20:52:54] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:22:48. frame-timeout = 1800
> [2010-09-21 20:52:54] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:22:48. frame-timeout = 1800
> [2010-09-21 20:53:04] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:23:00. frame-timeout = 1800
> [2010-09-21 20:53:04] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:23:00. frame-timeout = 1800
> [2010-09-21 20:53:24] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:23:21. frame-timeout = 1800
> [2010-09-21 20:53:24] E [client-protocol.c:309:call_bail] remote3: bailing out frame FINODELK(36) frame sent = 2010-09-21 20:23:21. frame-timeout = 1800
> 
> 
> 
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel