[Gluster-users] transport endpoint not connected on just 2 files

[Kingsley Tart]

Hi,

I've got a strange issue where on all clients I've tested on (tested on
4) I have "transport endpoint is not connected" on two files in a
directory, whereas other files can be read fine.

Any ideas?

On one of the servers (all same version):

# gluster --version
glusterfs 9.1

On one of the clients (same thing with all of them) - problem with
files "gw3" and "gw11":

[root at gw6 btl]# cd /mnt/runqueues/runners/
[root at gw6 runners]# ls -la
ls: cannot access gw11: Transport endpoint is not connected
ls: cannot access gw3: Transport endpoint is not connected
total 8
drwxr-xr-x  2 root root 4096 May 26 09:48 .
drwxr-xr-x 13 root root 4096 Apr 12  2021 ..
-rw-r--r--  1 root root    0 May 26 09:49 gw1
-rw-r--r--  1 root root    0 May 26 09:49 gw10
-?????????  ? ?    ?       ?            ? gw11
-rw-r--r--  1 root root    0 May 26 09:49 gw2
-?????????  ? ?    ?       ?            ? gw3
-rw-r--r--  1 root root    0 May 26 09:49 gw4
-rw-r--r--  1 root root    0 May 26 09:49 gw6
-rw-r--r--  1 root root    0 May 26 09:49 gw7
[root at gw6 runners]# cat *
cat: gw11: Transport endpoint is not connected
cat: gw3: Transport endpoint is not connected
[root at gw6 runners]#


Querying on a server shows those two problematic files:

# gluster volume heal gw-runqueues info
Brick gluster9a:/data/brick/gw-runqueues
/runners
/runners/gw11
/runners/gw3
Status: Connected
Number of entries: 3

Brick gluster9b:/data/brick/gw-runqueues
/runners
/runners/gw11
/runners/gw3
Status: Connected
Number of entries: 3

Brick gluster9c:/data/brick/gw-runqueues
Status: Connected
Number of entries: 0


However several hours later there's no obvious change. The servers have
hardly any load and the volume is tiny. From a client:

# find /mnt/runqueues | wc -l
35


glfsheal-gw-runqueues.log from server gluster9a:
https://pastebin.com/7mPszBBM

glfsheal-gw-runqueues.log from server gluster9b:
https://pastebin.com/rxXs5Tcv


Any pointers would be much appreciated!

Cheers,
Kingsley.