[Gluster-users] Volume mounts read-only

[Pavel Szalbot]

Hi, are you sure there are not two hosts with the same IP address?
Happened to me once and logs looked pretty similar (frequent
disconnects, all other clients seemed fine, gluster cluster reported
no issues).
-ps


On Fri, Jul 21, 2017 at 9:13 PM, Jamie Lawrence
<jlawrence at squaretrade.com> wrote:
> Hello Glusterites,
>
> I have a volume that will not mount read/write. V3.10.3 on Centos 7, this is a replica-3 volume, mounted with the fuse client. This is in support of an Ovirt installation, but I've isolated the problem to Gluster. `gluster peer status` looks normal, as does a `gluster v status`. Server.allow-insecure is set to on.
>
> Notable from the client volume logs are the 'connection refused' messages[1], that seem to match the gluster server logs[2], but I'm not exactly sure what that's telling me, or what I can do about it.  Two other hosts currently have the volume mounted and can write just fine; Because of operational reasons, I have not tested to see what would happen if I un/remount it on the currently functioning hosts. I trialed it disabling iptables, no change.
>
> I'm not sure what else to experiment with, and couldn't find anything relevant in the googles.  Thus this email.
>
> Has anyone seen this before? TIA,
>
> -j
>
>
> [1] rhev-data-center-mnt-glusterSD-sc5-gluster-10g-1:ovirt__engine.log
> [2017-07-21 18:19:07.210232] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket
> [2017-07-21 18:19:08.280799] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
> [2017-07-21 18:19:08.286512] E [socket.c:2309:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused)
> [2017-07-21 18:19:09.260107] W [fuse-bridge.c:1067:fuse_setattr_cbk] 0-glusterfs-fuse: 1721: SETATTR() /__DIRECT_IO_TEST__ => -1 (Read-only file system)
> [2017-07-21 18:19:10.286556] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
> [2017-07-21 18:19:10.292217] E [socket.c:2309:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused)
> [2017-07-21 18:19:11.206829] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
> [2017-07-21 18:19:11.212077] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
> [2017-07-21 18:19:11.216493] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket
> [2017-07-21 18:19:11.221265] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket
> [2017-07-21 18:19:12.292228] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
> [2017-07-21 18:19:12.297970] E [socket.c:2309:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused)
> [2017-07-21 18:19:14.297828] I [rpc-clnt.c:2001:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
> [2017-07-21 18:19:14.303567] E [socket.c:2309:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused)
> [2017-07-21 18:19:15.217792] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
> [2017-07-21 18:19:15.223013] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
> [2017-07-21 18:19:15.227406] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket
> [2017-07-21 18:19:15.232193] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket
>
>
> [2] glustershd.log:
> [2017-07-21 18:31:36.890242] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
> [2017-07-21 18:31:36.895707] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket
> [2017-07-21 18:31:39.896196] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
> [2017-07-21 18:31:39.901850] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket
> [2017-07-21 18:31:40.901569] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
> [2017-07-21 18:31:40.907006] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket
> [2017-07-21 18:31:43.907508] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-1: changing port to 49155 (from 0)
> [2017-07-21 18:31:43.913141] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-1: connection to 172.16.0.152:49155 failed (Connection refused); disconnecting socket
> [2017-07-21 18:31:44.912807] I [rpc-clnt.c:2000:rpc_clnt_reconfig] 0-ovirt_engine-client-0: changing port to 49220 (from 0)
> [2017-07-21 18:31:44.918243] E [socket.c:2316:socket_connect_finish] 0-ovirt_engine-client-0: connection to 172.16.0.151:49220 failed (Connection refused); disconnecting socket
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users