<div dir="ltr">Hi Chris,<div><br></div><div>By looking at the data provided (hope the other entry is also a file and not the parent of the file for which the stat & getfattrs are provided) it seems like the parent(s) of these entries are missing the entry pending markers on the good bricks, which is necessary to create these files on the bad node. Can you try the following steps and let us know whether you have any luck with this?</div><div><br></div><div>- Find the actual path of the files on one of the bricks where it exists using the below command</div><div>find <brickpath> -samefile <brickpath/.glusterfs/<first two bits of gfid>/<next 2 bits of gfid>/<full gfid></div><div>- Run lookup on the files from a client mount point</div><div>- Run gluster volume heal <volname></div><div>- Check the heal info to see whether these files gets healed or not</div><div><br></div><div>Regards,</div><div>Karthik</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Feb 1, 2020 at 2:25 PM Christian Reiss <<a href="mailto:email@christian-reiss.de">email@christian-reiss.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hey folks,<br>
<br>
in our production setup with 3 nodes (HCI) we took one host down <br>
(maintenance, stop gluster, poweroff via ssh/ovirt engine). Once it was <br>
up the gluster hat 2k healing entries that went down in a matter on 10 <br>
minutes to 2.<br>
<br>
Those two give me a headache:<br>
<br>
[root@node03:~] # gluster vol heal ssd_storage info<br>
Brick node01:/gluster_bricks/ssd_storage/ssd_storage<br>
<gfid:a121e4fb-0984-4e41-94d7-8f0c4f87f4b6><br>
<gfid:6f8817dc-3d92-46bf-aa65-a5d23f97490e><br>
Status: Connected<br>
Number of entries: 2<br>
<br>
Brick node02:/gluster_bricks/ssd_storage/ssd_storage<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick node03:/gluster_bricks/ssd_storage/ssd_storage<br>
<gfid:a121e4fb-0984-4e41-94d7-8f0c4f87f4b6><br>
<gfid:6f8817dc-3d92-46bf-aa65-a5d23f97490e><br>
Status: Connected<br>
Number of entries: 2<br>
<br>
No paths, only gfid. We took down node2, so it does not have the file:<br>
<br>
[root@node01:~] # md5sum <br>
/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
75c4941683b7eabc223fc9d5f022a77c <br>
/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
<br>
[root@node02:~] # md5sum <br>
/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
md5sum: <br>
/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6: <br>
No such file or directory<br>
<br>
[root@node03:~] # md5sum <br>
/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
75c4941683b7eabc223fc9d5f022a77c <br>
/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
<br>
The other two files are md5-identical.<br>
<br>
These flags are identical, too:<br>
<br>
[root@node01:~] # getfattr -d -m . -e hex <br>
/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
getfattr: Removing leading '/' from absolute path names<br>
# file: <br>
gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000<br>
trusted.afr.dirty=0x000000000000000000000000<br>
trusted.afr.ssd_storage-client-1=0x0000004f0000000100000000<br>
trusted.gfid=0xa121e4fb09844e4194d78f0c4f87f4b6<br>
trusted.gfid2path.d4cf876a215b173f=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f38366461303238392d663734662d343230302d393238342d3637386537626437363139352e31323030<br>
trusted.glusterfs.mdata=0x010000000000000000000000005e349b1e000000001139aa2a000000005e349b1e000000001139aa2a000000005e34994900000000304a5eb2<br>
<br>
getfattr: Removing leading '/' from absolute path names<br>
# file: <br>
gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
security.selinux=0x73797374656d5f753a6f626a6563745f723a676c7573746572645f627269636b5f743a733000<br>
trusted.afr.dirty=0x000000000000000000000000<br>
trusted.afr.ssd_storage-client-1=0x0000004f0000000100000000<br>
trusted.gfid=0xa121e4fb09844e4194d78f0c4f87f4b6<br>
trusted.gfid2path.d4cf876a215b173f=0x62653331383633382d653861302d346336642d393737642d3761393337616138343830362f38366461303238392d663734662d343230302d393238342d3637386537626437363139352e31323030<br>
trusted.glusterfs.mdata=0x010000000000000000000000005e349b1e000000001139aa2a000000005e349b1e000000001139aa2a000000005e34994900000000304a5eb2<br>
<br>
<br>
The only thing I can see is the different change times, really:<br>
<br>
[root@node01:~] # stat <br>
/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
File: <br>
‘/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6’<br>
Size: 67108864 Blocks: 54576 IO Block: 4096 regular file<br>
Device: fd09h/64777d Inode: 16152829909 Links: 2<br>
Access: (0660/-rw-rw----) Uid: ( 0/ root) Gid: ( 0/ root)<br>
Context: system_u:object_r:glusterd_brick_t:s0<br>
Access: 2020-01-31 22:16:57.812620635 +0100<br>
Modify: 2020-02-01 07:19:24.183045141 +0100<br>
Change: 2020-02-01 07:19:24.186045203 +0100<br>
Birth: -<br>
<br>
[root@node03:~] # stat <br>
/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6<br>
File: <br>
‘/gluster_bricks/ssd_storage/ssd_storage/.glusterfs/a1/21/a121e4fb-0984-4e41-94d7-8f0c4f87f4b6’<br>
Size: 67108864 Blocks: 54576 IO Block: 4096 regular file<br>
Device: fd09h/64777d Inode: 16154259424 Links: 2<br>
Access: (0660/-rw-rw----) Uid: ( 0/ root) Gid: ( 0/ root)<br>
Context: system_u:object_r:glusterd_brick_t:s0<br>
Access: 2020-01-31 22:16:57.811800217 +0100<br>
Modify: 2020-02-01 07:19:24.180939487 +0100<br>
Change: 2020-02-01 07:19:24.184939586 +0100<br>
Birth: -<br>
<br>
<br>
<br>
Now, I dont dare simply proceeding without some advice.<br>
Anyone got a clue on who to resolve this issue? File #2 is identical to <br>
this one, from a problem point of view.<br>
<br>
Have a great weekend!<br>
-Chris.<br>
<br>
-- <br>
with kind regards,<br>
mit freundlichen Gruessen,<br>
<br>
Christian Reiss<br>
<br>
________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>