Hi Zenon,<div><br></div><div>First step would be to ensure that all clients are connected to all bricks - this will reduce the chance of new problems.</div><div><br></div><div><br>For some reason there are problems with the broken node. Did you reduce the replica to 2 before reinstalling the broken node and re-adding it to the TSP ?</div><div><br></div><div>Try to get the attributes and the blames of a few files.</div><div>The following article (check all 3 parts) could help you understand the logic and give you hints where to look at:</div><div><br></div><div><a id="linkextractor__1721674149866" data-yahoo-extracted-link="true" href="https://ravispeaks.wordpress.com/2019/04/05/glusterfs-afr-the-complete-guide/" class="lEnhancr_1721674151810">https://ravispeaks.wordpress.com/2019/04/05/glusterfs-afr-the-complete-guide/</a></div><div><br></div><div><br><div id="ymail_android_signature">Best Regards,<br>Strahil Nikolov</div> <br> <blockquote style="margin: 0 0 20px 0;"> <div style="font-family:Roboto, sans-serif; color:#6D00F6;"> <div>On Wed, Jun 26, 2024 at 20:46, Zenon Panoussis</div><div><oracle@provocation.net> wrote:</div> </div> <div style="padding: 10px 0 0 20px; margin: 10px 0 0 0; border-left: 1px solid #6D00F6;"> <br clear="none">I should add that in /var/lib/glusterd/vols/gv0/gv0-shd.vol and<br clear="none">in all other configs in /var/lib/glusterd/ on all three machines<br clear="none">the nodes are consistently named<br clear="none"><br clear="none">client-2: zephyrosaurus<br clear="none">client-3: alvarezsaurus<br clear="none">client-4: nanosaurus<br clear="none"><br clear="none">This is normal. It was the second time that a brick was removed,<br clear="none">so client-0 and client-1 are gone.<br clear="none"><br clear="none">So the problem is the file attibutes themselves. And there I see<br clear="none">things like<br clear="none"><br clear="none">trusted.afr.gv0-client-0=0x000000000000000000000000<br clear="none">trusted.afr.gv0-client-1=0x000000000000000000000ab0<br clear="none">trusted.afr.gv0-client-3=0x000000000000000000000000<br clear="none">trusted.afr.gv0-client-4=0x000000000000000000000000<br clear="none"><br clear="none">and<br clear="none"><br clear="none">trusted.afr.gv0-client-3=0x000000000000000000000000<br clear="none">trusted.afr.gv0-client-4=0x000000000000000000000000<br clear="none"><br clear="none">and other such, where the only thing that is consistent, is inconsistency.<br clear="none"><br clear="none">When a brick is removed, shouldn't all files on the remaining bricks<br clear="none">be re-fattr'ed to remove the pointers to the non-existent brick?<br clear="none"><br clear="none">I guess I can do this manually, but it will still leave me with<br clear="none">those files where the value of all trusted.afr.gv0-client(s) is<br clear="none">zero. How does healing deal with those?<div class="yqt6139481903" id="yqtfd98061"><br clear="none"><br clear="none">Cheers,<br clear="none"><br clear="none">Z<br clear="none"><br clear="none"><br clear="none">-- <br clear="none">Слава Україні!<br clear="none">Путлер хуйло!<br clear="none">________<br clear="none"><br clear="none"><br clear="none"><br clear="none">Community Meeting Calendar:<br clear="none"><br clear="none">Schedule -<br clear="none">Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">Bridge: <a shape="rect" href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br clear="none">Gluster-users mailing list<br clear="none"><a shape="rect" ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br clear="none"><a shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br clear="none"></div> </div> </blockquote></div>