<div dir="ltr"><div>Hello Mikhail,</div><pre class="gmail-bz_comment_text" id="gmail-comment_text_21" style="white-space:pre-wrap;margin:1em;padding-bottom:1em;color:rgb(0,0,0)">Yes, If you delete any file from the gluster mount, every time the entries in .shard/.remove_me are checked and tried for deletion of shards.
The issue seems to be related to link-to-file and your last comment(<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1568521#c20">https://bugzilla.redhat.com/show_bug.cgi?id=1568521#c20</a>) quite useful to analyze the situation. I don't see similar information in the attached logs.
Can you please give a full set of the above-attached logs? Also, would you please check your brick status whether a few of them are completely full?
Regards </pre><div><span style="color:rgb(0,0,0);white-space:pre-wrap">Vh</span> </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 4, 2020 at 1:39 PM Михаил Гусев <<a href="mailto:gusevmk.uni@yandex.ru">gusevmk.uni@yandex.ru</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>Events:</div><div>1) Generate many files test-* on client side to /mnt/glusterfs-mountpoint (via for i in .. ; do dd if=/dev/zero .. ). File size = 100 G or 1T (amount 38 tb, gluster volume - 48 tb)</div><div>2) Start operation rm -rf /mnt/glusterfs-mountpoin/test-*</div><div>3) There are no files /mnt/glusterfs-mountpoin/test-*, but shards are still on bricks file systems.</div><div>Logs of glusterfs server and client (server-log.txt and client-log.txt - unix generated) (period - rm -rf operation) in attachment</div><div> </div><div>04.12.2020, 09:31, "Vinayakswami Hariharmath" <<a href="mailto:vharihar@redhat.com" target="_blank">vharihar@redhat.com</a>>:</div><blockquote><div>Hello,<div> </div><div>Bit of background about sharded file deletion:</div><div>There are 2 parts of sharded files 1. base file (1st shard or reference shard) 2. shards of the base file stored as GFID.index</div><div>When we delete a sharded file</div><div>1. firstly entry of a base file created (In the name of GFID) under .shard/.remove_me</div><div>2. next, the base file will be unlinked</div><div>3. in the background, the associated shards will be cleaned, and then finally reference entry present at .shard/.remove_me will be removed</div><div> </div><div>The reference created under .shard/.remove_me always referred to build the path to delete the associated shards. So the background thread picks up the ".shard/remove_me" entries, builds the shards path, and deletes them. </div><div> </div><div>So with your description, It looks like steps 1 and 2 are done but the background thread is getting ESTALE while cleaning up those .shard/.remove_me entries, the shards left undelete and space is not freed up. </div><div> </div><div>It looks strange, why you are getting ESTALE though the entry is present at .shard/.remove_me. Can you please post the complete logs of the time you performed the 1st deletion? History of events also helpful to analyze the issue</div><div> </div><div>Regards</div><div>Vh</div></div> <div><div>On Fri, Dec 4, 2020 at 11:33 AM Михаил Гусев <<a href="mailto:gusevmk.uni@yandex.ru" rel="noopener noreferrer" target="_blank">gusevmk.uni@yandex.ru</a>> wrote:</div><blockquote style="border-left:1px solid rgb(204,204,204);margin:0px 0px 0px 0.8ex;padding-left:1ex"><div><div><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Hello, I think my problem is related with this issue: </span><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1568521" rel="noopener noreferrer" style="background-color:rgb(255,255,255);color:rgb(153,0,153);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1568521</a><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">After remove files from client (rm -rf /mountpoint/*), I get many error in gluster mnt log (client side):</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">E [MSGID: 133021] [shard.c:3761:shard_delete_shards] 0-<volume name>-shard: Failed to clean up shards of gfid 4b5afa</span><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">49-5446-49</span><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">e2-a7ba-1b4f2ffadb12 [Stale file handle]</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Files in mountpoint was deleted, but there are no available space after this operation.</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">I did check - there are no files in directory .glusterfs/unlink, but many files are in .shard/.remove_me/</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">As far as I understand glusterd server process have to check files in .shard/.remove_me/ and if no mapping in .glusterfs/unlink, shards must be removed.. But seems it not working. </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"># glusterd --version </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">glusterfs 8.2</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"># cat /etc/redhat-release </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Red Hat Enterprise Linux release 8.3 (Ootpa)</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">gluster volume type: dispersed-distribute (sharding is enabled)</span></div></div>________<br><br><br><br>Community Meeting Calendar:<br><br>Schedule -<br>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noopener noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org" rel="noopener noreferrer" target="_blank">Gluster-users@gluster.org</a><br><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noopener noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div></blockquote></blockquote></div>