[Gluster-users] One node goes offline, the other node can't see the replicated volume anymore

Tue Jul 16 01:16:21 UTC 2013

Here is a tail from /var/log/glusterfs/firewall-scripts.log.   The entries at 00:39:56 GMT must correspond to /var/log/messages at 19:39:56 USA Central time.   There are no firewall rules and the default policy is ACCEPT.   With no rc.local file, my application is out of the picture right now, so nobody is sleeping or taking themselves offline or anything like that.  Just a straight up boot using fstab to mount the filesystems.   
.
.
.
[2013-07-16 00:36:40.495183] I [afr-common.c:2120:afr_discovery_cbk] 0-firewall-scripts-replicate-0: selecting local read_child
firewall-scripts-client-0
[2013-07-16 00:39:12.584585] W [socket.c:514:__socket_rwv] 0-glusterfs: readv failed (No data available)
[2013-07-16 00:39:12.584853] W [socket.c:1962:__socket_proto_state_machine] 0-glusterfs: reading from socket failed. Error (No d
ata available), peer (127.0.0.1:24007)
[2013-07-16 00:39:12.714225] W [glusterfsd.c:970:cleanup_and_exit] (-->/usr/lib64/libc.so.6(clone+0x6d) [0x7fd5c27f913d] (-->/us
r/lib64/libpthread.so.0(+0x33c1607c53) [0x7fd5c2e8fc53] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7fd5c3b7de35]))) 0-
: received signum (15), shutting down
[2013-07-16 00:39:12.714289] I [fuse-bridge.c:5212:fini] 0-fuse: Unmounting '/firewall-scripts'.
[2013-07-16 00:39:12.715170] I [fuse-bridge.c:4583:fuse_thread_proc] 0-fuse: unmounting /firewall-scripts
[2013-07-16 00:39:56.667185] I [glusterfsd.c:1878:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0
beta4 (/usr/sbin/glusterfs --volfile-id=/firewall-scripts --volfile-server=localhost /firewall-scripts)
[2013-07-16 00:39:56.726785] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled
[2013-07-16 00:39:56.726993] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread
[2013-07-16 00:40:05.794794] I [socket.c:3480:socket_init] 0-firewall-scripts-client-1: SSL support is NOT enabled
[2013-07-16 00:40:05.794927] I [socket.c:3495:socket_init] 0-firewall-scripts-client-1: using system polling thread
[2013-07-16 00:40:05.801351] I [socket.c:3480:socket_init] 0-firewall-scripts-client-0: SSL support is NOT enabled
[2013-07-16 00:40:05.801486] I [socket.c:3495:socket_init] 0-firewall-scripts-client-0: using system polling thread
[2013-07-16 00:40:05.801611] I [client.c:2154:notify] 0-firewall-scripts-client-0: parent translators are ready, attempting conn
ect on transport
[2013-07-16 00:40:05.817724] I [client.c:2154:notify] 0-firewall-scripts-client-1: parent translators are ready, attempting conn
ect on transport
Given volfile:
+------------------------------------------------------------------------------+
  1: volume firewall-scripts-client-0
  2:     type protocol/client
  3:     option password fb3955b7-a6ca-49bb-b886-d4b6609392f8
  4:     option username de6eacd1-31bc-4bdb-a049-776cd840059e
  5:     option transport-type tcp
  6:     option remote-subvolume /gluster-fw1
  7:     option remote-host 192.168.253.1
  8: end-volume
  9:
 10: volume firewall-scripts-client-1
 11:     type protocol/client
 12:     option password fb3955b7-a6ca-49bb-b886-d4b6609392f8
 13:     option username de6eacd1-31bc-4bdb-a049-776cd840059e
 14:     option transport-type tcp
 15:     option remote-subvolume /gluster-fw2
 16:     option remote-host 192.168.253.2
 17: end-volume
 18:
 19: volume firewall-scripts-replicate-0
 20:     type cluster/replicate
 21:     subvolumes firewall-scripts-client-0 firewall-scripts-client-1
 22: end-volume
 23:
 24: volume firewall-scripts-dht
 25:     type cluster/distribute
 26:     subvolumes firewall-scripts-replicate-0
 27: end-volume
 28:
 29: volume firewall-scripts-write-behind
 30:     type performance/write-behind
 31:     subvolumes firewall-scripts-dht
 32: end-volume
 33:
 34: volume firewall-scripts-read-ahead
 35:     type performance/read-ahead
 36:     subvolumes firewall-scripts-write-behind
 37: end-volume
 38:
 39: volume firewall-scripts-io-cache
 40:     type performance/io-cache
 41:     subvolumes firewall-scripts-read-ahead
 42: end-volume
 43:
 44: volume firewall-scripts-quick-read
 45:     type performance/quick-read
 46:     subvolumes firewall-scripts-io-cache
 47: end-volume
 48:
 49: volume firewall-scripts-open-behind
 50:     type performance/open-behind
 51:     subvolumes firewall-scripts-quick-read
 52: end-volume
 53:
 54: volume firewall-scripts-md-cache
 55:     type performance/md-cache
 56:     subvolumes firewall-scripts-open-behind
 57: end-volume
 58:
 59: volume firewall-scripts
 60:     type debug/io-stats
 61:     option count-fop-hits off
 62:     option latency-measurement off
 63:     subvolumes firewall-scripts-md-cache
 64: end-volume

+------------------------------------------------------------------------------+
[2013-07-16 00:40:05.975356] E [client-handshake.c:1741:client_query_portmap_cbk] 0-firewall-scripts-client-0: failed to get the
 port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2013-07-16 00:40:05.975588] W [socket.c:514:__socket_rwv] 0-firewall-scripts-client-0: readv failed (No data available)
[2013-07-16 00:40:05.975708] I [client.c:2097:client_rpc_notify] 0-firewall-scripts-client-0: disconnected
[2013-07-16 00:40:06.027821] E [client-handshake.c:1741:client_query_portmap_cbk] 0-firewall-scripts-client-1: failed to get the
 port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.
[2013-07-16 00:40:06.028010] W [socket.c:514:__socket_rwv] 0-firewall-scripts-client-1: readv failed (No data available)
[2013-07-16 00:40:06.028103] I [client.c:2097:client_rpc_notify] 0-firewall-scripts-client-1: disconnected
[2013-07-16 00:40:06.028148] E [afr-common.c:3735:afr_notify] 0-firewall-scripts-replicate-0: All subvolumes are down. Going off
line until atleast one of them comes back up.
[2013-07-16 00:40:06.048172] I [fuse-bridge.c:4723:fuse_graph_setup] 0-fuse: switched to graph 0
[2013-07-16 00:40:06.049068] I [fuse-bridge.c:3680:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.
13 kernel 7.21
[2013-07-16 00:40:06.051158] W [fuse-bridge.c:665:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (No such file or director
y)
[2013-07-16 00:40:06.077420] I [fuse-bridge.c:4583:fuse_thread_proc] 0-fuse: unmounting /firewall-scripts
[2013-07-16 00:40:06.078427] W [glusterfsd.c:970:cleanup_and_exit] (-->/usr/lib64/libc.so.6(clone+0x6d) [0x7f1f0078d13d] (-->/us
r/lib64/libpthread.so.0(+0x33c1607c53) [0x7f1f00e23c53] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xd5) [0x7f1f01b11e35]))) 0-
: received signum (15), shutting down
[2013-07-16 00:40:06.078501] I [fuse-bridge.c:5212:fini] 0-fuse: Unmounting '/firewall-scripts'.
[2013-07-16 00:53:39.844556] I [glusterfsd.c:1878:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.0
beta4 (/usr/sbin/glusterfs --volfile-id=/firewall-scripts --volfile-server=localhost /firewall-scripts)
[2013-07-16 00:53:39.858957] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled
[2013-07-16 00:53:39.859117] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread
[2013-07-16 00:53:39.907716] I [socket.c:3480:socket_init] 0-firewall-scripts-client-1: SSL support is NOT enabled
[2013-07-16 00:53:39.907881] I [socket.c:3495:socket_init] 0-firewall-scripts-client-1: using system polling thread
[2013-07-16 00:53:39.909563] I [socket.c:3480:socket_init] 0-firewall-scripts-client-0: SSL support is NOT enabled
[2013-07-16 00:53:39.909655] I [socket.c:3495:socket_init] 0-firewall-scripts-client-0: using system polling thread
[2013-07-16 00:53:39.909778] I [client.c:2154:notify] 0-firewall-scripts-client-0: parent translators are ready, attempting conn
ect on transport
[2013-07-16 00:53:39.920933] I [client.c:2154:notify] 0-firewall-scripts-client-1: parent translators are ready, attempting conn
ect on transport
Given volfile:
+------------------------------------------------------------------------------+
  1: volume firewall-scripts-client-0
  2:     type protocol/client
  3:     option password fb3955b7-a6ca-49bb-b886-d4b6609392f8
  4:     option username de6eacd1-31bc-4bdb-a049-776cd840059e
  5:     option transport-type tcp
  6:     option remote-subvolume /gluster-fw1
  7:     option remote-host 192.168.253.1
  8: end-volume
  9:
 10: volume firewall-scripts-client-1
 11:     type protocol/client
 12:     option password fb3955b7-a6ca-49bb-b886-d4b6609392f8
 13:     option username de6eacd1-31bc-4bdb-a049-776cd840059e
 14:     option transport-type tcp
 15:     option remote-subvolume /gluster-fw2
 16:     option remote-host 192.168.253.2
 17: end-volume
 18:
 19: volume firewall-scripts-replicate-0
 20:     type cluster/replicate
 21:     subvolumes firewall-scripts-client-0 firewall-scripts-client-1
 22: end-volume
 23:
 24: volume firewall-scripts-dht
 25:     type cluster/distribute
 26:     subvolumes firewall-scripts-replicate-0
 27: end-volume
 28:
 29: volume firewall-scripts-write-behind
 30:     type performance/write-behind
 31:     subvolumes firewall-scripts-dht
 32: end-volume
 33:
 34: volume firewall-scripts-read-ahead
 35:     type performance/read-ahead
 36:     subvolumes firewall-scripts-write-behind
 37: end-volume
 38:
 39: volume firewall-scripts-io-cache
 40:     type performance/io-cache
 41:     subvolumes firewall-scripts-read-ahead
 42: end-volume
 43:
 44: volume firewall-scripts-quick-read
 45:     type performance/quick-read
 46:     subvolumes firewall-scripts-io-cache
 47: end-volume
 48:
 49: volume firewall-scripts-open-behind
 50:     type performance/open-behind
 51:     subvolumes firewall-scripts-quick-read
 52: end-volume
 53:
 54: volume firewall-scripts-md-cache
 55:     type performance/md-cache
 56:     subvolumes firewall-scripts-open-behind
 57: end-volume
 58:
 59: volume firewall-scripts
 60:     type debug/io-stats
 61:     option count-fop-hits off
 62:     option latency-measurement off
 63:     subvolumes firewall-scripts-md-cache
 64: end-volume

+------------------------------------------------------------------------------+
[2013-07-16 00:53:39.933009] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-firewall-scripts-client-0: changing port to 49152 (from 0)
[2013-07-16 00:53:39.933178] W [socket.c:514:__socket_rwv] 0-firewall-scripts-client-0: readv failed (No data available)
[2013-07-16 00:53:39.950457] I [rpc-clnt.c:1676:rpc_clnt_reconfig] 0-firewall-scripts-client-1: changing port to 49152 (from 0)
[2013-07-16 00:53:39.950621] W [socket.c:514:__socket_rwv] 0-firewall-scripts-client-1: readv failed (No data available)
[2013-07-16 00:53:39.966646] I [client-handshake.c:1658:select_server_supported_programs] 0-firewall-scripts-client-0: Using Pro
gram GlusterFS 3.3, Num (1298437), Version (330)
[2013-07-16 00:53:39.966994] I [client-handshake.c:1658:select_server_supported_programs] 0-firewall-scripts-client-1: Using Pro
gram GlusterFS 3.3, Num (1298437), Version (330)
[2013-07-16 00:53:39.967417] I [client-handshake.c:1456:client_setvolume_cbk] 0-firewall-scripts-client-0: Connected to 192.168.
253.1:49152, attached to remote volume '/gluster-fw1'.
[2013-07-16 00:53:39.967498] I [client-handshake.c:1468:client_setvolume_cbk] 0-firewall-scripts-client-0: Server and Client lk-
version numbers are not same, reopening the fds
[2013-07-16 00:53:39.967800] I [afr-common.c:3698:afr_notify] 0-firewall-scripts-replicate-0: Subvolume 'firewall-scripts-client
-0' came back up; going online.
[2013-07-16 00:53:39.967959] I [client-handshake.c:450:client_set_lk_version_cbk] 0-firewall-scripts-client-0: Server lk version
 = 1
[2013-07-16 00:53:39.968068] I [client-handshake.c:1456:client_setvolume_cbk] 0-firewall-scripts-client-1: Connected to 192.168.
253.2:49152, attached to remote volume '/gluster-fw2'.
[2013-07-16 00:53:39.968114] I [client-handshake.c:1468:client_setvolume_cbk] 0-firewall-scripts-client-1: Server and Client lk-
version numbers are not same, reopening the fds
[2013-07-16 00:53:39.982434] I [fuse-bridge.c:4723:fuse_graph_setup] 0-fuse: switched to graph 0
[2013-07-16 00:53:39.982877] I [client-handshake.c:450:client_set_lk_version_cbk] 0-firewall-scripts-client-1: Server lk version
 = 1
[2013-07-16 00:53:39.983205] I [fuse-bridge.c:3680:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.
13 kernel 7.21
[2013-07-16 00:53:39.984516] I [afr-common.c:2057:afr_set_root_inode_on_first_lookup] 0-firewall-scripts-replicate-0: added root
 inode
[2013-07-16 00:53:39.985412] I [afr-common.c:2120:afr_discovery_cbk] 0-firewall-scripts-replicate-0: selecting local read_child
firewall-scripts-client-0
[root at chicago-fw1 ~]#