<div dir="ltr">Hello,<div><br></div><div>Bit of background about sharded file deletion:</div><div>There are 2 parts of sharded files 1. base file (1st shard or reference shard) 2. shards of the base file stored as GFID.index</div><div>When we delete a sharded file</div><div>1. firstly entry of a base file created (In the name of GFID) under .shard/.remove_me</div><div>2. next, the base file will be unlinked</div><div>3. in the background, the associated shards will be cleaned, and then finally reference entry present at .shard/.remove_me will be removed</div><div><br></div><div>The reference created under .shard/.remove_me always referred to build the path to delete the associated shards. So the background thread picks up the ".shard/remove_me" entries, builds the shards path, and deletes them. </div><div><br></div><div>So with your description, It looks like steps 1 and 2 are done but the background thread is getting ESTALE while cleaning up those .shard/.remove_me entries, the shards left undelete and space is not freed up. </div><div><br></div><div>It looks strange, why you are getting ESTALE though the entry is present at .shard/.remove_me. Can you please post the complete logs of the time you performed the 1st deletion? History of events also helpful to analyze the issue</div><div><br></div><div>Regards</div><div>Vh</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Dec 4, 2020 at 11:33 AM Михаил Гусев <<a href="mailto:gusevmk.uni@yandex.ru">gusevmk.uni@yandex.ru</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Hello, I think my problem is related with this issue: </span><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1568521" rel="noopener noreferrer" style="background-color:rgb(255,255,255);color:rgb(153,0,153);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1568521</a><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">After remove files from client (rm -rf /mountpoint/*), I get many error in gluster mnt log (client side):</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">E [MSGID: 133021] [shard.c:3761:shard_delete_shards] 0-<volume name>-shard: Failed to clean up shards of gfid 4b5afa</span><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">49-5446-49</span><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">e2-a7ba-1b4f2ffadb12 [Stale file handle]</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Files in mountpoint was deleted, but there are no available space after this operation.</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">I did check - there are no files in directory .glusterfs/unlink, but many files are in .shard/.remove_me/</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">As far as I understand glusterd server process have to check files in .shard/.remove_me/ and if no mapping in .glusterfs/unlink, shards must be removed.. But seems it not working. </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"># glusterd --version </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">glusterfs 8.2</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"># cat /etc/redhat-release </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Red Hat Enterprise Linux release 8.3 (Ootpa)</span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"> </span><br style="background-color:rgb(255,255,255);color:rgb(0,0,0);font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="background-color:rgb(255,255,255);color:rgb(0,0,0);float:none;font-family:arial,sans-serif;font-size:15px;font-style:normal;font-weight:400;text-decoration-style:initial;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">gluster volume type: dispersed-distribute (sharding is enabled)</span></div></div>________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div>