<html dir="ltr"><head></head><body style="text-align:left; direction:ltr;"><div>Hi,</div><div><br></div><div>Thanks - sorry for the late reply - I was suddenly swamped with other work then it was a UK holiday.</div><div><br></div><div>I've tried rsync -A -X with the volume stopped, then restarted it. Will see whether it heals.</div><div><br></div><div>Cheers,</div><div>Kingsley.</div><div><br></div><div>On Mon, 2022-05-30 at 18:41 +0000, Strahil Nikolov wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Make a backup from all bricks. Based on the info 2 of the bricks have the same copy while brickC has another copy (gfid mismatch).</div><div><br></div><div>I would use mtime to identify the latest version and use that, but I have no clue what kind of application you have.</div><div><br></div><div>Usually, It's not recommended to manipulate bricks directly, but in this case it might be necessary. The simplest way is to move the file on brick C (the only one that is different) away, but if you need exactly that one, you can rsync/scp it to the other 2 bricks.</div><div><br></div><div><br></div><div>Best Regards,</div><div>Strahil Nikolov</div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>On Fri, May 27, 2022 at 11:45, Kingsley Tart</div><div><<a href="mailto:gluster@gluster.dogwind.com">gluster@gluster.dogwind.com</a>> wrote:</div><div>Hi, thanks.</div><div><br></div><div>OK that's interesting. Picking one of the files, on bricks A and B I see this (and all of the values are identical between bricks A and B):</div><div><br></div><div>trusted.afr.dirty=0x000000000000000000000000</div><div>trusted.afr.gw-runqueues-client-2=0x000000010000000200000000</div><div>trusted.gfid=0xa40bb83ff3784ae09c997d272296a7a9</div><div>trusted.gfid2path.06eddbe9be9c7c75=0x30323665396561652d613661662d346365642d623863632d6261353037333339646364372f677733</div><div>trusted.glusterfs.mdata=0x01000000000000000000000000628ec57700000000007168bb00000000628ec576000000000000000000000000628ec5760000000000000000</div><div><br></div><div>and on brick C I see this:</div><div><br></div><div>trusted.gfid=0xd73992aee03e4021824b1baced973df3</div><div>trusted.gfid2path.06eddbe9be9c7c75=0x30323665396561652d613661662d346365642d623863632d6261353037333339646364372f677733</div><div>trusted.glusterfs.mdata=0x01000000000000000000000000628ec5230000000030136ca000000000628ec523000000000000000000000000628ec5230000000000000000</div><div><br></div><div>So brick C is missing the trusted.afr attributes and the trusted.gfid and mdata differ.</div><div><br></div><div>What do I need to do to fix this?</div><div><br></div><div>Cheers,</div><div>Kingsley.</div><div><br></div><div>On Fri, 2022-05-27 at 03:59 +0000, Strahil Nikolov wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>Check the file attributes on all bricks:</div><div><br></div><div>getfattr -d -e hex -m. /data/brick/gw-runqueues/<path to file></div><div><br></div><div><br></div><div>Best Regards,</div><div>Strahil Nikolov</div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>On Thu, May 26, 2022 at 16:05, Kingsley Tart</div><div><<a href="mailto:gluster@gluster.dogwind.com">gluster@gluster.dogwind.com</a>> wrote:</div><div>Hi,</div><div><br></div><div>I've got a strange issue where on all clients I've tested on (tested on</div><div>4) I have "transport endpoint is not connected" on two files in a</div><div>directory, whereas other files can be read fine.</div><div><br></div><div>Any ideas?</div><div><br></div><div>On one of the servers (all same version):</div><div><br></div><div># gluster --version</div><div>glusterfs 9.1</div><div><br></div><div>On one of the clients (same thing with all of them) - problem with</div><div>files "gw3" and "gw11":</div><div><br></div><div>[<a href="mailto:root@gw6">root@gw6</a> btl]# cd /mnt/runqueues/runners/</div><div>[<a href="mailto:root@gw6">root@gw6</a> runners]# ls -la</div><div>ls: cannot access gw11: Transport endpoint is not connected</div><div>ls: cannot access gw3: Transport endpoint is not connected</div><div>total 8</div><div>drwxr-xr-x 2 root root 4096 May 26 09:48 .</div><div>drwxr-xr-x 13 root root 4096 Apr 12 2021 ..</div><div>-rw-r--r-- 1 root root 0 May 26 09:49 gw1</div><div>-rw-r--r-- 1 root root 0 May 26 09:49 gw10</div><div>-????????? ? ? ? ? ? gw11</div><div>-rw-r--r-- 1 root root 0 May 26 09:49 gw2</div><div>-????????? ? ? ? ? ? gw3</div><div>-rw-r--r-- 1 root root 0 May 26 09:49 gw4</div><div>-rw-r--r-- 1 root root 0 May 26 09:49 gw6</div><div>-rw-r--r-- 1 root root 0 May 26 09:49 gw7</div><div>[<a href="mailto:root@gw6">root@gw6</a> runners]# cat *</div><div>cat: gw11: Transport endpoint is not connected</div><div>cat: gw3: Transport endpoint is not connected</div><div>[<a href="mailto:root@gw6">root@gw6</a> runners]#</div><div><br></div><div><br></div><div>Querying on a server shows those two problematic files:</div><div><br></div><div># gluster volume heal gw-runqueues info</div><div>Brick gluster9a:/data/brick/gw-runqueues</div><div>/runners</div><div>/runners/gw11</div><div>/runners/gw3</div><div>Status: Connected</div><div>Number of entries: 3</div><div><br></div><div>Brick gluster9b:/data/brick/gw-runqueues</div><div>/runners</div><div>/runners/gw11</div><div>/runners/gw3</div><div>Status: Connected</div><div>Number of entries: 3</div><div><br></div><div>Brick gluster9c:/data/brick/gw-runqueues</div><div>Status: Connected</div><div>Number of entries: 0</div><div><br></div><div><br></div><div>However several hours later there's no obvious change. The servers have</div><div>hardly any load and the volume is tiny. From a client:</div><div><br></div><div># find /mnt/runqueues | wc -l</div><div>35</div><div><br></div><div><br></div><div>glfsheal-gw-runqueues.log from server gluster9a:</div><div><a href="https://pastebin.com/7mPszBBM">https://pastebin.com/7mPszBBM</a></div><div><br></div><div>glfsheal-gw-runqueues.log from server gluster9b:</div><div><a href="https://pastebin.com/rxXs5Tcv">https://pastebin.com/rxXs5Tcv</a></div><div><br></div><div><br></div><div>Any pointers would be much appreciated!</div><div><br></div><div>Cheers,</div><div>Kingsley.</div><div><br></div><div>________</div><div><br></div><div><br></div><div><br></div><div>Community Meeting Calendar:</div><div><br></div><div>Schedule -</div><div>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC</div><div>Bridge: <a href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a></div><div>Gluster-users mailing list</div><div><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a></div><div><a href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div></blockquote><div><br></div><div>________</div><div><br></div><div><br></div><div><br></div><div>Community Meeting Calendar:</div><div><br></div><div>Schedule -</div><div>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC</div><div>Bridge: <a href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a></div><div>Gluster-users mailing list</div><div><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a></div><div><a href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div></blockquote></blockquote></blockquote></body></html>