[Gluster-users] socket.c:2161:socket_connect_finish (Connection refused)

[Olav Peeters]

Hi,
I upgraded from glusterfs 3.4 to 3.5 about 8 days ago. Everything was 
running fine until this morning. In a fuse mount we were having write 
issues. Creating and deleting files became an issue all of a sudden 
without any new changes to the cluster.

In /var/log/glusterfs/glustershd.log every couple of seconds I'm getting 
this:

[2014-06-10 22:23:52.055128] I [rpc-clnt.c:1685:rpc_clnt_reconfig] 
0-sr_vol01-client-13: changing port to 49156 (from 0)
[2014-06-10 22:23:52.060153] E [socket.c:2161:socket_connect_finish] 
0-sr_vol01-client-13: connection to ip-of-one-of-the-gluster-nodes:49156 
failed (Connection refused)

# gluster volume status sr_vol01
shows that two bricks of the 18 are offline.

rebalance fails

Iptables was stopped on all nodes

If I cd into the two bricks which are offline according to the gluster v 
status, I can read/write without any problems... The disks are clearly 
fine. They are mounted, they are available.

I cannot find much info online about the error.
Does anyone have an idea what could be wrong?
How can I get the two bricks back online?

Cheers,
Olav