[Gluster-devel] GlusterFS 1.3.0-pre2.3: robustness

Gerry Reno greno at verizon.net
Thu Apr 5 01:12:51 UTC 2007


Ok, I setup a four brick AFR test and it is working.  Well at least as 
long as nothing goes wrong.  But if I take down one of the servers 
(glusterfsd) and try to write to any of the mounts then the whole thing 
hangs.  This is the type of situation that I am hoping will be handled 
gracefully.  I've had tcp/ip stacks hang, servers hang and need rebooted 
and once back up the whole cluster needs to be able to recover itself. 

So with the current state of GlusterFS is this a bug or is there 
functionality planned that would support this level of robustness.

[root at grp-01-30-01 glusterfs]# touch /mnt/glusterfs3/file3
[root at grp-01-30-01 glusterfs]# find /mnt/glusterfs?
/mnt/glusterfs0
/mnt/glusterfs0/file0
/mnt/glusterfs0/file3
/mnt/glusterfs0/file2
/mnt/glusterfs0/file1
/mnt/glusterfs1
/mnt/glusterfs1/file0
/mnt/glusterfs1/file3
/mnt/glusterfs1/file2
/mnt/glusterfs1/file1
/mnt/glusterfs2
/mnt/glusterfs2/file0
/mnt/glusterfs2/file3
/mnt/glusterfs2/file2
/mnt/glusterfs2/file1
/mnt/glusterfs3
/mnt/glusterfs3/file0
/mnt/glusterfs3/file3
/mnt/glusterfs3/file2
/mnt/glusterfs3/file1
[root at grp-01-30-01 glusterfs]# find /root/export?  #underlying filesystems
/root/export0
/root/export0/file1
/root/export0/file2
/root/export0/file3
/root/export0/file0
/root/export1
/root/export1/file1
/root/export1/file2
/root/export1/file3
/root/export1/file0
/root/export2
/root/export2/file1
/root/export2/file2
/root/export2/file3
/root/export2/file0
/root/export3
/root/export3/file1
/root/export3/file2
/root/export3/file3
/root/export3/file0


=================================
now kill one server:
[root at grp-01-30-01 glusterfs]# kill -9 23879
[root at grp-01-30-01 glusterfs]# ps -e | grep gluster
23881 ?        00:00:00 glusterfsd
23883 ?        00:00:00 glusterfsd
23885 ?        00:00:00 glusterfsd
23888 ?        00:00:00 glusterfs
23891 ?        00:00:00 glusterfs
23894 ?        00:00:00 glusterfs
23897 ?        00:00:00 glusterfs
[root at grp-01-30-01 glusterfs]# find /mnt/glusterfs?
/mnt/glusterfs0
/mnt/glusterfs0/file0
/mnt/glusterfs0/file2
/mnt/glusterfs0/file1
/mnt/glusterfs1
/mnt/glusterfs1/file0
/mnt/glusterfs1/file2
/mnt/glusterfs1/file1
/mnt/glusterfs2
/mnt/glusterfs2/file0
/mnt/glusterfs2/file2
/mnt/glusterfs2/file1
/mnt/glusterfs3
/mnt/glusterfs3/file0
/mnt/glusterfs3/file2
/mnt/glusterfs3/file1

=== now touch a new file =====
[root at grp-01-30-01 glusterfs]# touch /mnt/glusterfs1/file-0down

=== ^C done here
[root at grp-01-30-01 glusterfs]# find /mnt/glusterfs?
/mnt/glusterfs0
/mnt/glusterfs0/file0
/mnt/glusterfs0/file2
/mnt/glusterfs0/file1

=== ^C done here

Both commands hung.

Gerry


LOG OUTPUT:

[Apr 04 20:55:24] [ERROR/common-utils.c:54/full_rw()] 
libglusterfs:full_rw: 0 bytes r/w instead of 113
[Apr 04 20:55:24] 
[DEBUG/protocol.c:244/gf_block_unserialize_transport()] 
libglusterfs/protocol:gf_block_unserialize_transport: full_read of 
header failed
[Apr 04 20:55:24] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:24] [ERROR/common-utils.c:54/full_rw()] 
libglusterfs:full_rw: 0 bytes r/w instead of 113
[Apr 04 20:55:24] 
[DEBUG/protocol.c:244/gf_block_unserialize_transport()] 
libglusterfs/protocol:gf_block_unserialize_transport: full_read of 
header failed
[Apr 04 20:55:24] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:24] [ERROR/common-utils.c:54/full_rw()] 
libglusterfs:full_rw: 0 bytes r/w instead of 113
[Apr 04 20:55:24] 
[DEBUG/protocol.c:244/gf_block_unserialize_transport()] 
libglusterfs/protocol:gf_block_unserialize_transport: full_read of 
header failed
[Apr 04 20:55:24] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:24] [ERROR/common-utils.c:54/full_rw()] 
libglusterfs:full_rw: 0 bytes r/w instead of 113
[Apr 04 20:55:24] 
[DEBUG/protocol.c:244/gf_block_unserialize_transport()] 
libglusterfs/protocol:gf_block_unserialize_transport: full_read of 
header failed
[Apr 04 20:55:24] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x916da08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:55:38] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:55:38] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:55:38] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:55:38] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:55:38] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:55:38] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x867ba08
[Apr 04 20:56:13] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:13] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:13] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:13] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:13] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:13] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:56:13] [DEBUG/afr.c:65/afr_get_num_copies()] afr:matched! 
pattern = *, filename = file-0down,
[Apr 04 20:56:13] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:13] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:13] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:13] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:13] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:13] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x9992a08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08
[Apr 04 20:56:37] [DEBUG/tcp-client.c:174/tcp_connect()] transport: tcp: 
:try_connect: socket fd = 10
[Apr 04 20:56:37] [DEBUG/tcp-client.c:196/tcp_connect()] transport: tcp: 
:try_connect: finalized on port `1023'
[Apr 04 20:56:37] [DEBUG/tcp-client.c:255/tcp_connect()] 
tcp/client:connect on 10 in progress (non-blocking)
[Apr 04 20:56:37] [ERROR/tcp-client.c:284/tcp_connect()] 
tcp/client:non-blocking connect() returned: 111 (Connection refused)
[Apr 04 20:56:37] [ERROR/client-protocol.c:183/client_protocol_xfer()] 
protocol/client: client_protocol_xfer: :transport_submit failed
[Apr 04 20:56:37] 
[DEBUG/client-protocol.c:2543/client_protocol_cleanup()] 
protocol/client:cleaning up state in transport object 0x885fa08






More information about the Gluster-devel mailing list