[Gluster-users] GlusterFS disconnection

Tue Aug 30 08:15:09 UTC 2011

Hi,

    We have glusterFS setup with version 3.1.4 installed and configured in
replica mode.

The setup was configured in the following manner. The gluster volume info
looks like

# gluster volume info
Volume Name: gluster-fs1
Type: Replicate
Status: Started
Number of Bricks: 2
Transport-type: rdma
Bricks:
Brick1: jr4-1-ib:/data/gluster/brick-md2
Brick2: jr4-2-ib:/data/gluster/brick-md2
We are having strange problem with gluster disconnection. We see the
following error from the servers side.

[2011-08-30 07:41:02.868432] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 07:41:05.870965] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 07:41:05.872927] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 07:41:08.875478] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 07:41:08.877490] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 07:41:11.880046] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 07:41:11.882048] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 07:41:14.884589] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 07:41:14.886616] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 07:41:17.889162] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 07:41:17.891141] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 07:41:20.893646] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 07:41:20.895656] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 07:41:23.898240] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 07:41:23.900252] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
brick logs shows the following error messgaes

[2011-08-29 18:16:00.308719] E [rpcsvc.c:1554:rpcsvc_submit_generic]
0-rpc-service: failed to submit message (XID: 0x13460x, Program:
GlusterFS-3.1.0, Pro
gVers: 310, Proc: 34) to rpc-transport (rdma.gluster-fs1-server)
[2011-08-29 18:16:00.308737] E [server.c:137:server_submit_reply] 0-: Reply
submission failed
[2011-08-29 18:16:00.308751] I
[server-helpers.c:756:server_connection_destroy] 0-gluster-fs1-server:
destroyed connection of n1710-1749-2011/08/26-15:43:
13:560803-gluster-fs1-client-0
[2011-08-29 18:16:00.308773] E [rpcsvc.c:1554:rpcsvc_submit_generic]
0-rpc-service: failed to submit message (XID: 0x15372x, Program:
GlusterFS-3.1.0, Pro
gVers: 310, Proc: 34) to rpc-transport (rdma.gluster-fs1-server)
[2011-08-29 18:16:00.308800] E [server.c:137:server_submit_reply] 0-: Reply
submission failed
[2011-08-29 18:16:00.308819] I
[server-helpers.c:756:server_connection_destroy] 0-gluster-fs1-server:
destroyed connection of n1711-1788-2011/08/26-15:43:
38:361566-gluster-fs1-client-0
[2011-08-29 18:16:00.309051] E [rpcsvc.c:1554:rpcsvc_submit_generic]
0-rpc-service: failed to submit message (XID: 0x1463x, Program:
GlusterFS-3.1.0, Prog
Vers: 310, Proc: 34) to rpc-transport (rdma.gluster-fs1-server)
[2011-08-29 18:16:00.309070] E [server.c:137:server_submit_reply] 0-: Reply
submission failed
[2011-08-29 18:16:00.309143] I
[server-helpers.c:756:server_connection_destroy] 0-gluster-fs1-server:
destroyed connection of n1722-1765-2011/08/26-21:13:
31:834472-gluster-fs1-client-0
[2011-08-29 18:16:00.310517] E [rpcsvc.c:1554:rpcsvc_submit_generic]
0-rpc-service: failed to submit message (XID: 0x1599091x, Program:
GlusterFS-3.1.0, P
rogVers: 310, Proc: 34) to rpc-transport (rdma.gluster-fs1-server)
[2011-08-29 18:16:00.310539] E [rpc-transport.c:976:rpc_transport_ref]
0-rpc_transport: invalid argument: this
[2011-08-29 18:16:00.310543] E [server.c:137:server_submit_reply] 0-: Reply
submission failed
[2011-08-29 18:16:00.310564] E [rpc-transport.c:996:rpc_transport_unref]
0-rpc_transport: invalid argument: this
[2011-08-29 18:16:00.310607] E [rpc-transport.c:976:rpc_transport_ref]
0-rpc_transport: invalid argument: this

and on the client side we see the following error messages

[2011-08-30 10:15:52.149071] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 10:15:53.152158] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 10:15:54.888495] W [fuse-bridge.c:413:fuse_attr_cbk]
0-glusterfs-fuse: 9817037: LOOKUP() / => -1 (Transport endpoint is not
connected)
[2011-08-30 10:15:55.155257] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 10:15:56.158282] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 10:15:58.161525] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)
[2011-08-30 10:15:59.164618] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-1: tcp connect to 172.31.100.228:24009 failed
(Connection refused)
[2011-08-30 10:16:01.167819] E [rdma.c:4428:tcp_connect_finish]
0-gluster-fs1-client-0: tcp connect to 172.31.100.227:24009 failed
(Connection refused)

Currently i only option we see is to restart the gluster services on the
gluster brick nodes, which allows to automatically connect the glusterfs.
Could you please suggest us what would be the reason for the same.

Regards,
Ramana Kasaraneni.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110830/1877135e/attachment.html>