[Gluster-users] Volume mounts read-only

[Jamie Lawrence]

Hello Glusterites,

I have a volume that will not mount read/write. V3.10.3 on Centos 7, this is a replica-3 volume, mounted with the fuse client. This is in support of an Ovirt installation, but I've isolated the problem to Gluster. `gluster peer status` looks normal, as does a `gluster v status`. Server.allow-insecure is set to on.

Notable from the client volume logs are the 'connection refused' messages[1], that seem to match the gluster server logs[2], but I'm not exactly sure what that's telling me, or what I can do about it.  Two other hosts currently have the volume mounted and can write just fine; Because of operational reasons, I have not tested to see what would happen if I un/remount it on the currently functioning hosts. I trialed it disabling iptables, no change. 

I'm not sure what else to experiment with, and couldn't find anything relevant in the googles.  Thus this email.

Has anyone seen this before? TIA,

-j


[1] rhev-data-center-mnt-glusterSD-sc5-gluster-10g-1:ovirt__engine.log
[2017-07-21 18:19:07.210232] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket
[2017-07-21 18:19:08.280799] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
[2017-07-21 18:19:08.286512] E [socket.c:2309:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused)
[2017-07-21 18:19:09.260107] W [fuse-bridge.c:1067:fuse_setattr_cbk] 0-glusterfs-fuse: 1721: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only file system)
[2017-07-21 18:19:10.286556] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
[2017-07-21 18:19:10.292217] E [socket.c:2309:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused)
[2017-07-21 18:19:11.206829] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
[2017-07-21 18:19:11.212077] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
[2017-07-21 18:19:11.216493] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket
[2017-07-21 18:19:11.221265] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket
[2017-07-21 18:19:12.292228] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
[2017-07-21 18:19:12.297970] E [socket.c:2309:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused)
[2017-07-21 18:19:14.297828] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
[2017-07-21 18:19:14.303567] E [socket.c:2309:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused)
[2017-07-21 18:19:15.217792] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
[2017-07-21 18:19:15.223013] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
[2017-07-21 18:19:15.227406] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket
[2017-07-21 18:19:15.232193] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket


[2] glustershd.log:
[2017-07-21 18:31:36.890242] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
[2017-07-21 18:31:36.895707] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket
[2017-07-21 18:31:39.896196] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
[2017-07-21 18:31:39.901850] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket
[2017-07-21 18:31:40.901569] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
[2017-07-21 18:31:40.907006] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket
[2017-07-21 18:31:43.907508] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
[2017-07-21 18:31:43.913141] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket
[2017-07-21 18:31:44.912807] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
[2017-07-21 18:31:44.918243] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket