[Gluster-users] Crossover cable: single point of failure?

[Daniel Manser]

> Can you please share NFS and brick logs from the duration of the link
> going down? Gluster should have worked in the situation you 
> described.

 Brick log on gluster1:
 [2011-06-10 13:12:08.57634] W [socket.c:204:__socket_rwv] 
 0-tcp.vmware-server: readv failed (Connection timed out)
 [2011-06-10 13:12:08.57674] W 
 [socket.c:1494:__socket_proto_state_machine] 0-tcp.vmware-server: 
 reading from socket failed. Error (Connection timed out), peer 
 (192.168.150.252:1022)
 [2011-06-10 13:12:08.57712] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/debian one.vmx
 [2011-06-10 13:12:08.57778] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/vmware-3.log
 [2011-06-10 13:12:08.57796] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/debian one.vmdk
 [2011-06-10 13:12:08.57820] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/debian one-flat.vmdk
 [2011-06-10 13:12:08.57848] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/debian one.nvram
 [2011-06-10 13:12:08.57866] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/.lck-98894e05b84ec3a1
 [2011-06-10 13:12:08.57887] I [server.c:438:server_rpc_notify] 
 0-vmware-server: disconnected connection from 192.168.150.252:1022
 [2011-06-10 13:12:08.57933] I 
 [server-helpers.c:783:server_connection_destroy] 0-vmware-server: 
 destroyed connection of 
 gluster2-4038-2011/06/10-08:31:09:180937-vmware-client-0
 [2011-06-10 13:12:19.3036] I [server-handshake.c:534:server_setvolume] 
 0-vmware-server: accepted client from 192.168.150.252:1021

 Brick log on gluster2:
 [2011-06-10 13:12:01.467191] W [socket.c:204:__socket_rwv] 
 0-tcp.vmware-server: readv failed (Connection timed out)
 [2011-06-10 13:12:01.467236] W 
 [socket.c:1494:__socket_proto_state_machine] 0-tcp.vmware-server: 
 reading from socket failed. Error (Connection timed out), peer 
 (192.168.150.251:1021)
 [2011-06-10 13:12:01.467279] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/debian one.vmx
 [2011-06-10 13:12:01.467345] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/debian one.vmxf
 [2011-06-10 13:12:01.467362] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/debian one.vmdk
 [2011-06-10 13:12:01.467379] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/debian one-flat.vmdk
 [2011-06-10 13:12:01.467413] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/vmware-4.log
 [2011-06-10 13:12:01.467431] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/debian one.nvram
 [2011-06-10 13:12:01.467447] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/vmware-5.log
 [2011-06-10 13:12:01.467463] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/vmware-6.log
 [2011-06-10 13:12:01.467478] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/vmware.log
 [2011-06-10 13:12:01.467494] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/.lck-98894e05b84ec3a1
 [2011-06-10 13:12:01.467510] I [server-helpers.c:485:do_fd_cleanup] 
 0-vmware-server: fd cleanup on /debian one/.lck-b14dcb6d11640f7e
 [2011-06-10 13:12:01.467526] I [server.c:438:server_rpc_notify] 
 0-vmware-server: disconnected connection from 192.168.150.251:1021
 [2011-06-10 13:12:01.467546] I 
 [server-helpers.c:783:server_connection_destroy] 0-vmware-server: 
 destroyed connection of 
 gluster1-3974-2011/06/10-08:31:35:778159-vmware-client-1
 [2011-06-10 13:12:18.705503] I 
 [server-handshake.c:534:server_setvolume] 0-vmware-server: accepted 
 client from 192.168.150.251:1021

 NFS log on gluster2:
 [2011-06-10 13:11:56.708490] W [socket.c:204:__socket_rwv] 
 0-testvolume-client-0: readv failed (Connection timed out)
 [2011-06-10 13:11:56.708530] W 
 [socket.c:1494:__socket_proto_state_machine] 0-testvolume-client-0: 
 reading from socket failed. Error (Connection timed out), peer 
 (192.168.150.251:24009)
 [2011-06-10 13:11:56.708564] I [client.c:1883:client_rpc_notify] 
 0-testvolume-client-0: disconnected
 [2011-06-10 13:11:56.709490] W [socket.c:204:__socket_rwv] 
 0-vmware-client-0: readv failed (Connection timed out)
 [2011-06-10 13:11:56.709521] W 
 [socket.c:1494:__socket_proto_state_machine] 0-vmware-client-0: reading 
 from socket failed. Error (Connection timed out), peer 
 (192.168.150.251:24013)
 [2011-06-10 13:11:56.709553] I [client.c:1883:client_rpc_notify] 
 0-vmware-client-0: disconnected
 [2011-06-10 13:12:07.701752] I 
 [client-handshake.c:1080:select_server_supported_programs] 
 0-testvolume-client-0: Using Program GlusterFS-3.1.0, Num (1298437), 
 Version (310)
 [2011-06-10 13:12:07.702059] I 
 [client-handshake.c:913:client_setvolume_cbk] 0-testvolume-client-0: 
 Connected to 192.168.150.251:24009, attached to remote volume 
 '/mnt/gvolumes/testvolume'.
 [2011-06-10 13:12:07.705397] I 
 [client-handshake.c:1080:select_server_supported_programs] 
 0-vmware-client-0: Using Program GlusterFS-3.1.0, Num (1298437), Version 
 (310)
 [2011-06-10 13:12:07.705656] I 
 [client-handshake.c:913:client_setvolume_cbk] 0-vmware-client-0: 
 Connected to 192.168.150.251:24013, attached to remote volume 
 '/mnt/gvolumes/vmware'.
 [2011-06-10 13:12:07.705684] I 
 [client-handshake.c:779:client_post_handshake] 0-vmware-client-0: 6 fds 
 open - Delaying child_up until they are re-opened
 [2011-06-10 13:12:07.706312] I 
 [client-lk.c:617:decrement_reopen_fd_count] 0-vmware-client-0: last fd 
 open'd/lock-self-heal'd - notifying CHILD-UP

 The NFS log on gluster1 has a lot of lines. I have put it on pastebin 
 [0]. 192.168.150.251 is the crossover link IP for gluster1, 
 192.168.150.252 for gluster2.

 Thanks

 [0] http://pastebin.com/tpwevMSu