[Gluster-infra] failed (No route to host) disconnecting socket

[Ezequiel Suarez]

I have openshift 3.11 with 4 gluster nodes (glusterfs 6.0). A few days
ago users reported problems with pv (disconnection).

I see the log in one of the gluster nodes and I find these messages.

[2022-05-11 01:52:17.684622] I [rpc-clnt.c:2013:rpc_clnt_reconfig]
0-vol_f2581329a8e5ef03cac6c6fa92cf6396-client-1: changing port to
49254 (from 0)
[2022-05-11 01:52:17.687888] I [rpc-clnt.c:2013:rpc_clnt_reconfig]
0-vol_f6324c9528f1761a26f49d4dbf3cb568-client-2: changing port to
49256 (from 0)
[2022-05-11 01:52:17.694184] I [rpc-clnt.c:2013:rpc_clnt_reconfig]
0-vol_eaff3cce15e6c814172440c72692b915-client-5: changing port to
49253 (from 0)
[2022-05-11 01:52:18.671410] E [socket.c:2376:socket_connect_finish]
0-vol_eaff3cce15e6c814172440c72692b915-client-2: connection to
10.254.7.214:49253 failed (No route to host); disconnecting socket
[2022-05-11 01:52:20.699985] E [socket.c:2376:socket_connect_finish]
0-vol_f2581329a8e5ef03cac6c6fa92cf6396-client-1: connection to
10.254.7.214:49254 failed (No route to host); disconnecting socket
[2022-05-11 01:52:20.701326] E [socket.c:2376:socket_connect_finish]
0-vol_f6324c9528f1761a26f49d4dbf3cb568-client-2: connection to
10.254.7.214:49256 failed (No route to host); disconnecting socket
[2022-05-11 01:52:21.097323] I [rpc-clnt.c:2013:rpc_clnt_reconfig]
0-vol_eaff3cce15e6c814172440c72692b915-client-2: changing port to
49253 (from 0)
[2022-05-11 01:52:21.103126] I [rpc-clnt.c:2013:rpc_clnt_reconfig]
0-vol_e8b901f78c0e84f3c422887068dfccd6-client-0: changing port to
49252 (from 0)
[2022-05-11 01:52:21.104614] E [socket.c:2376:socket_connect_finish]
0-vol_eaff3cce15e6c814172440c72692b915-client-2: connection to
10.254.7.214:49253 failed (No route to host); disconnecting socket
[2022-05-11 01:52:21.109317] I [rpc-clnt.c:2013:rpc_clnt_reconfig]
0-vol_f2581329a8e5ef03cac6c6fa92cf6396-client-1: changing port to
49254 (from 0)
[2022-05-11 01:52:21.122744] I [rpc-clnt.c:2013:rpc_clnt_reconfig]
0-vol_f6324c9528f1761a26f49d4dbf3cb568-client-2: changing port to
49256 (from 0)
[2022-05-11 01:52:22.123479] E [socket.c:2376:socket_connect_finish]
0-vol_f2581329a8e5ef03cac6c6fa92cf6396-client-1: connection to
10.254.7.214:49254 failed (No route to host); disconnecting socket


We have tried all the gluster commands to check the status of the
cluster and volumes and found no errors.

Can someone help me or give me an idea of where to look?

Thanks
Ezequiel Suarez