[Gluster-users] Hanging writes after upgrading "clients" to debian squeeze

Sun Feb 5 20:49:47 UTC 2012

- no ip tables involved
- the server is running 3.2.0 as well, of course I could upgrade but this probably means some downtime which I can not afford right now
- did not find something in the logs, but there are a lot of files so I might miss something, logs on the client or server side?
- some debug flags or verbose logging possible?

the bricks log on the server side just says "client connected" so there is not a lot of value in that. On the client side I have the following:

[2012-02-05 19:38:53.324172] I [fuse-bridge.c:3214:fuse_thread_proc] 0-fuse: unmounting /home/XXXstorage
[2012-02-05 19:38:53.324221] I [glusterfsd.c:712:cleanup_and_exit] 0-glusterfsd: shutting down
[2012-02-05 19:38:58.709783] W [write-behind.c:3023:init] 0-XXXstorage-write-behind: disabling write-behind for first 0 bytes
[2012-02-05 19:38:58.711289] I [client.c:1935:notify] 0-XXXstorage-client-0: parent translators are ready, attempting connect on transport
[2012-02-05 19:38:58.711489] I [client.c:1935:notify] 0-XXXstorage-client-1: parent translators are ready, attempting connect on transport
Given volfile:
+------------------------------------------------------------------------------+
  1: volume XXXstorage-client-0
  2:     type protocol/client
  3:     option remote-host 10.10.100.40
  4:     option remote-subvolume /brick1
  5:     option transport-type tcp
  6: end-volume
  7:
  8: volume XXXstorage-client-1
  9:     type protocol/client
 10:     option remote-host 10.10.100.41
 11:     option remote-subvolume /brick1
 12:     option transport-type tcp
 13: end-volume
 14:
 15: volume XXXstorage-replicate-0
 16:     type cluster/replicate
 17:     subvolumes XXXstorage-client-0 XXXstorage-client-1
 18: end-volume
 19:
 20: volume XXXstorage-write-behind
 21:     type performance/write-behind
 22:     subvolumes XXXstorage-replicate-0
 23: end-volume
 24:
 25: volume XXXstorage-read-ahead
 26:     type performance/read-ahead
 27:     subvolumes XXXstorage-write-behind
 28: end-volume
 29:
 30: volume XXXstorage-io-cache
 31:     type performance/io-cache
 32:     subvolumes XXXstorage-read-ahead
 33: end-volume
 34:
 35: volume XXXstorage-stat-prefetch
 36:     type performance/stat-prefetch
 37:     subvolumes XXXstorage-io-cache
 38: end-volume
 39:
 40: volume XXXstorage
 41:     type debug/io-stats
 42:     option latency-measurement off
 43:     option count-fop-hits off
 44:     subvolumes XXXstorage-stat-prefetch
 45: end-volume

+------------------------------------------------------------------------------+
[2012-02-05 19:38:58.712460] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-XXXstorage-client-1: changing port to 24015 (from 0)
[2012-02-05 19:38:58.712527] I [rpc-clnt.c:1531:rpc_clnt_reconfig] 0-XXXstorage-client-0: changing port to 24012 (from 0)
[2012-02-05 19:39:02.709882] I [client-handshake.c:1080:select_server_supported_programs] 0-XXXstorage-client-1: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2012-02-05 19:39:02.710112] I [client-handshake.c:1080:select_server_supported_programs] 0-XXXstorage-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version (310)
[2012-02-05 19:39:02.710355] I [client-handshake.c:913:client_setvolume_cbk] 0-XXXstorage-client-1: Connected to 10.10.100.41:24015, attached to remote volume '/brick1'.
[2012-02-05 19:39:02.710395] I [afr-common.c:2514:afr_notify] 0-XXXstorage-replicate-0: Subvolume 'XXXstorage-client-1' came back up; going online.
[2012-02-05 19:39:02.712314] I [fuse-bridge.c:3316:fuse_graph_setup] 0-fuse: switched to graph 0
[2012-02-05 19:39:02.712387] I [client-handshake.c:913:client_setvolume_cbk] 0-XXXstorage-client-0: Connected to 10.10.100.40:24012, attached to remote volume '/brick1'.
[2012-02-05 19:39:02.712436] I [fuse-bridge.c:2897:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
[2012-02-05 19:39:02.713253] I [afr-common.c:836:afr_fresh_lookup_cbk] 0-XXXstorage-replicate-0: added root inode

I cannot see any problems. I was tailing a few logs while I issued a write which hangs. Nothing gets logged.

-----Ursprüngliche Nachricht-----
Von: Whit Blauvelt [mailto:whit.gluster at transpect.com] 
Gesendet: Sonntag, 5. Februar 2012 21:25
An: Brian Candler
Cc: Stefan Becker; gluster-users at gluster.org
Betreff: Re: [Gluster-users] Hanging writes after upgrading "clients" to debian squeeze

On Sun, Feb 05, 2012 at 07:36:55PM +0000, Brian Candler wrote:
> On Sun, Feb 05, 2012 at 08:02:08PM +0100, Stefan Becker wrote:
> >    After the debian upgrade I can
> >    still mount my volumes. Reading is fine as well but it hangs on writes.
> 
> Could it be that on the post-upgrade machines one brick is reachable but not
> the other?  Compare iptables rules between the pre-upgrade and post-upgrade
> machines?  Compare tcpdump or ntop between them?

If you can, try dropping iptables out of the picture entirely. If you are
running it, and have it logging what it drops, the docs say "Ensure that TCP
ports 111, 24007,24008, 24009-(24009 + number of bricks across all volumes)
are open on all Gluster servers. If you will be using NFS, open additional
ports 38465 to 38467." So I'd check your logs to see if iptables is dropping
any traffic to/from the IPs in question on those ports.

Or us "netstat -tc" while doing some file operations, and you should see the
traffic on the IPs/ports. Another utility to see the same thing is "iptraf."  

Whit	