[Gluster-devel] cannot delete non-empty directory

Mon Feb 9 17:17:25 UTC 2015

  
/backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24
[root at gfs01bkp C24]# ls -al
total 1
drwx------ 2 jcowan users 39 Feb  6 12:41 .
drwxrw-rw- 4 jcowan users 62 Feb  6 19:19 ..

[root at gfs01bkp C24]# ls -al 
/data/brick*/homegfs_bkp/backup.0/old_shelf4/Aegis/\!\!\!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/
total 4
drwxrw-rw-+ 2 jcowan users 4096 Feb  6 12:41 .
drwxrw-rw-+ 3 jcowan users   29 Feb  6 12:41 ..
---------T  5 jcowan users    0 Nov 19 23:30 
c24-airbox_vr_z=25_zoom.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=26.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=27.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=28.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 
c24-airbox_vr_z=29.5_zoom.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=30.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=31.jpeg
---------T  5 jcowan users    0 Nov 19 23:30 c24-airbox_vr_z=32.5.jpeg
[root at gfs01bkp C24]# getfattr -d -m . -e text 
/data/brick*bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/\!\!\!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/*
getfattr: Removing leading '/' from absolute path names
# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=25_zoom.jpeg
trusted.gfid="îr'V*N©ÍÆF¿"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=26.jpeg
trusted.gfid="Là¾}®ÀLdza¥U"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=27.jpeg
trusted.gfid="©.ñªû2@¬ºÜdíÁ?%_"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=28.jpeg
trusted.gfid="0¥
                 /DªÒx?Ïý"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=29.5_zoom.jpeg
trusted.gfid="¼9T'$²Cí¯Eÿx!1"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=30.jpeg
trusted.gfid="tè
8rð"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=31.jpeg
trusted.gfid="x´Å
                  EÅ¦¡ZmØWà"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"

# file: 
data/brick01bkp/homegfs_bkp/backup.0/old_shelf4/Aegis/!!!Programs/Nextel_Cup/SHR/Backup/shr/Airbox/C24/z_slices/c24-airbox_vr_z=32.5.jpeg
trusted.gfid="d+0ÇxþM¯GxÑ@>Â"
trusted.glusterfs.dht.linkto="homegfs_bkp-client-1"


------ Original Message ------
From: "Shyam" <srangana at redhat.com>
To: "David F. Robinson" <david.robinson at corvidtec.com>; "Gluster Devel" 
<gluster-devel at gluster.org>; "gluster-users at gluster.org" 
<gluster-users at gluster.org>; "Susant Palai" <spalai at redhat.com>
Sent: 2/9/2015 11:11:20 AM
Subject: Re: [Gluster-devel] cannot delete non-empty directory

>On 02/08/2015 12:19 PM, David F. Robinson wrote:
>>I am seeing these messsages after I delete large amounts of data using
>>gluster 3.6.2.
>>cannot delete non-empty directory:
>>old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
>>*_From the FUSE mount (as root), the directory shows up as empty:_*
>># pwd
>>/backup/homegfs/backup.0/old_shelf4/Aegis/!!!Programs/RavenCFD/Storage/Jimmy_Old/src_vj1.5_final
>>
>># ls -al
>>total 5
>>d--------- 2 root root 4106 Feb 6 13:55 .
>>drwxrws--- 3 601 dmiller 72 Feb 6 13:55 ..
>>However, when you look at the bricks, the files are still there (none 
>>on
>>brick01bkp, all files are on brick02bkp). All of the files are 
>>0-length
>>and have ------T permissions.
>
>These files are linkto files that are created by DHT, which basically 
>mean the files were either renamed, or the brick layout changed (I 
>suspect the former to be the cause).
>
>These files should have been deleted when the files that they point to 
>were deleted, looks like this did not happen.
>
>Can I get the following information for some of the files here,
>- getfattr -d -m . -e text <path to file on brick>
>   - The output of trusted.glusterfs.dht.linkto xattr should state where 
>the real file belongs, in this case as there are only 2 bricks, it 
>should be brick01bkp subvol
>- As the second brick is empty, we should be able to safely delete 
>these files from the brick and proceed to do an rmdir on the mount 
>point of the volume as the directory is now empty.
>- Please check, the one sub-directory that is showing up in this case 
>as well, "save1"
>
>>Any suggestions on how to fix this and how to prevent it from 
>>happening?
>
>I believe there are renames happening here, possibly by the archive 
>creator, one way to prevent the rename from creating a linkto file is 
>to use the DHT set parameter to set a pattern so that file name hash 
>considers only the static part of the name.
>
>The set parameter is, cluster.extra-hash-regex.
>
>A link on a similar problem and how to use this set parameter (there a 
>few in the gluster forums) would be, 
>http://www.gluster.org/pipermail/gluster-devel/2014-November/042863.html
>
>Additionally, there is a bug here, the unlink of the file should have 
>cleaned up the linkto as well, so that all of the above is not 
>required, we have noticed this with NFS and FUSE mounts (ref bugs, 
>1117923, 1139992), and investigation is in progress on the same. We 
>will step up the priority on this so that we have a clean fix that can 
>be used to prevent this in the future.
>
>Shyam