[Gluster-users] [EXT] cannot remove empty directory on gluster file system

[Stefan Solbrig]

Hi,

This happened to me many times.  In my case, the reason was this:

On the bricks, there were gluster link files present (The ones with zero bytes and permissions =1000 that are created if the real file is on a different brick than the brick that corresponds to the hash)  but the link target had gone. 

My solution was: go to the respective directories directly on the bricks, see if there link files present but no other files, and the delete the link files manually.   This worked for me.  (I have a distribute-only system.  No guaranteee....)
However, this might leave stale hard links in  /path/to/brick/.glusterfs   but you can find these with
find /path/to/brick/.glusterfs  -type f -links 1   but be careful to delete these as well... double check that there really don't link anywhere.

I haven't discovered the original reason why there are stale glusterfs link files.  I suspect they are left when a unlink operation fails silently if one of the bricks is temporariliy down,  but I'm not sure.

best wishes,
Stefan

> Am 25.03.2020 um 21:08 schrieb Strahil Nikolov <hunter86_bg at yahoo.com>:
> 
> You can also check if there is a process holding a file that was deleted there:
> lsof /tier2/OPA/archive/GOFS/RECOVERY20190416/GlobNative/20190505
> 
> If it's not that one , I'm out of ideas :)
> 
> It's not recommended to delete it from the bricks , so avoid that if possible.
> 
> Best Regards,
> Strahil Nikolov
> 
> 
> 
> 
> 
> 
> В сряда, 25 март 2020 г., 21:12:58 Гринуич+2, Mauro Tridici <mauro.tridici at cmcc.it> написа: 
> 
> 
> 
> 
> 
> Hi Sttrahil,
> 
> thank you for your answer.
> Directory is empty and no immutable bit has been assigned to it.
> 
> [athena-login2][/tier2/OPA/archive/GOFS/RECOVERY20190416/GlobNative/20190505]> ls -la
> total 8
> drwxr-xr-x 2 das oclab_prod 4096 Mar 25 10:02 .
> drwxr-xr-x 3 das oclab_prod 4096 Mar 25 10:02 ..
> 
> Any other idea related this issue?
> Many thanks,
> Mauro
> 
> 
>> On 25 Mar 2020, at 18:32, Strahil Nikolov <hunter86_bg at yahoo.com> wrote:
>> 
>> On March 25, 2020 3:32:59 PM GMT+02:00, Mauro Tridici <mauro.tridici at cmcc.it> wrote:
>>> Dear All,
>>> 
>>> some users tht use regularly our gluster file system are experiencing a
>>> strange error during attempting to remove a empty directory.
>>> All bricks are up and running, no perticular error has been detected,
>>> but they are not able to remove it successfully.
>>> 
>>> This is the error they are receiving:
>>> 
>>> [athena-login2][/tier2/OPA/archive/GOFS]> rm -rf RECOVERY20190416/
>>> rm: cannot remove `RECOVERY20190416/GlobNative/20190505': Directory not
>>> empty
>>> 
>>> I tried to delete this directory from root user without success.
>>> Do you have some suggestions to solve this issue?
>>> 
>>> Thank you in advance.
>>> Kind Regards,
>>> Mauro
>> 
>> What do you have in 'RECOVERY20190416/GlobNative/20190505' ?
>> 
>> Maybe you got an immutable bit (chattr +i) on any file/folder  ?
>> 
>> Best Regards,
>> Sttrahil Nikolov
> 
> 
> 
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users