[Gluster-devel] brick stops responding

[Emmanuel Dreyfus]

Hi

With 3.5beta1, at some point I have a brick that stops responding, and 
reconection never happens. 

Client log:

[2014-01-16 12:57:47.896907] C [client-handshake.c:127:rpc_client_ping_timer_expired] 0-gfs35b1-client-0: server 192.0.2.99:49152 has not responded in the last 42 seconds, disconnecting.
[2014-01-16 12:57:47.897315] E [rpc-clnt.c:369:saved_frames_unwind]  0-gfs35b1-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2014-01-16 12:56:33.651927 (xid=0xded6c2)
[2014-01-16 12:57:47.897466] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-gfs35b1-client-0: remote operation failed: Socket is not connected
[2014-01-16 12:57:47.897643] E [rpc-clnt.c:369:saved_frames_unwind]  0-gfs35b1-client-0: forced unwinding frame type(GlusterFS 3.3) op(INODELK(29)) called at 2014-01-16 12:56:33.651998 (xid=0xded6c3)
[2014-01-16 12:57:47.897756] W [client-rpc-fops.c:1529:client3_3_inodelk_cbk] 0-gfs35b1-client-0: remote operation failed: Socket is not connected
(many times, andit never reconects)

Brick log:

[2014-01-16 12:57:48.036133] I [client_t.c:166:gf_client_get] 0-gfs35b1-server: client_uid=bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0
[2014-01-16 12:57:48.041763] I [server-handshake.c:575:server_setvolume] 0-gfs35b1-server: accepted client from bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0 (version: 3.5beta1)
[2014-01-16 12:58:52.609335] W [socket.c:522:__socket_rwv] 0-tcp.gfs35b1-server: writev on 192.0.2.6:1018 failed (Broken pipe)
[2014-01-16 12:58:52.828718] I [server.c:521:server_rpc_notify] 0-gfs35b1-server: disconnecting connectionfrom bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0
[2014-01-16 13:00:17.465098] I [client_t.c:166:gf_client_get] 0-gfs35b1-server: client_uid=bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0
[2014-01-16 13:00:17.465240] I [server-handshake.c:575:server_setvolume] 0-gfs35b1-server: accepted client from bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0 (version: 3.5beta1)
[2014-01-16 13:03:07.220384] I [client_t.c:166:gf_client_get] 0-gfs35b1-server: client_uid=bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0
[2014-01-16 13:03:07.220499] I [server-handshake.c:575:server_setvolume] 0-gfs35b1-server: accepted client from bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0 (version: 3.5beta1)
[2014-01-16 13:05:56.938213] I [client_t.c:166:gf_client_get] 0-gfs35b1-server: client_uid=bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0
[2014-01-16 13:05:56.938357] I [server-handshake.c:575:server_setvolume] 0-gfs35b1-server: accepted client from bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0 (version: 3.5beta1)
[2014-01-16 13:07:21.750081] I [client_t.c:166:gf_client_get] 0-gfs35b1-server: client_uid=bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0
[2014-01-16 13:07:21.768110] I [server-handshake.c:575:server_setvolume] 0-gfs35b1-server: accepted client from bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0 (version: 3.5beta1)
[2014-01-16 13:10:11.457756] I [client_t.c:166:gf_client_get] 0-gfs35b1-server: client_uid=bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0
[2014-01-16 13:10:11.457897] I [server-handshake.c:575:server_setvolume] 0-gfs35b1-server: accepted client from bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0 (version: 3.5beta1)
[2014-01-16 13:13:01.196338] I [client_t.c:166:gf_client_get] 0-gfs35b1-server: client_uid=bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0
[2014-01-16 13:13:01.196482] I [server-handshake.c:575:server_setvolume] 0-gfs35b1-server: accepted client from bacasable.example.net-17263-2014/01/15-18:24:08:99795-gfs35b1-client-0-0 (version: 3.5beta1)

Any idea?


-- 
Emmanuel Dreyfus
manu at netbsd.org