[Gluster-users] Unable to delete files but getfattr shows file is part of glusterfs

[Franco Broi]

Hi John

I got this yesterday, it was copied to this list.

Cheers,


On Tue, 2014-06-17 at 04:55 -0400, Susant Palai wrote:
Hi Franco:
>    The following patches address the ENOTEMPTY issue.
> 
>                  1. http://review.gluster.org/#/c/7733/ 
>                  2. http://review.gluster.org/#/c/7599/ 
> 
> I think the above patches will be available in 3.5.1 which will be a minor upgrade.(Need ack from Niels de Vos.)
> 


On Wed, 2014-06-18 at 13:26 +0930, SINCOCK John wrote: 
> Thanks Pranith, I hope Susant can help!
> 
>  
> 
> From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com] 
> Sent: Wednesday, 18 June 2014 11:48 AM
> To: SINCOCK John; gluster-users at gluster.org
> Cc: Susant Palai
> Subject: Re: [Gluster-users] Unable to delete files but getfattr shows
> file is part of glusterfs
> 
> 
>  
> 
> I think Susant knows about this issue
> 
> CCed him
> 
> Pranith
> 
> On 06/15/2014 07:16 PM, SINCOCK John wrote:
> 
> 
>         Hi Everybody,
>         
>          
>         
>         We have recently setup a 2-node gluster, and are having some
>         problems deleting some files!
>         
>         We haven’t accessed the gluster bricks directly, all access
>         has been through the glusterfs mount 
>         
>         The 1st node in our cluster also handles sharing the glusterfs
>         volume via samba, to windows clients.
>         
>          
>         
>         Environment is:
>         
>         Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13
>         13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
>         
>         CentOS release 6.4 (Final)
>         
>          
>         
>         glusterfs-libs-3.4.1-3.el6.x86_64
>         
>         glusterfs-cli-3.4.1-3.el6.x86_64
>         
>         glusterfs-fuse-3.4.1-3.el6.x86_64
>         
>         glusterfs-server-3.4.1-3.el6.x86_64
>         
>         glusterfs-3.4.1-3.el6.x86_64
>         
>          
>         
>                        4 bricks on each node
>         
>                        Gluster is setup without replication or
>         striping
>         
>          
>         
>                        Bricks on one of the two nodes (the other node
>         is setup the same):
>         
>         /dev/sda1             9.1T  4.7T  4.5T
>         51% /mnt/glusterfs/bricks/1
>         
>         /dev/sda2             9.1T  3.7T  5.5T
>         40% /mnt/glusterfs/bricks/2
>         
>         /dev/sda3             9.1T  3.7T  5.5T
>         40% /mnt/glusterfs/bricks/3
>         
>         /dev/sda4             9.1T  4.1T  5.0T
>         46% /mnt/glusterfs/bricks/4
>         
>          
>         
>         Gluster volume mounted on 1st node:
>         
>         g-unit-1:/vol00        73T   31T   43T  42% /gluster/vol00
>         
>                        Mount options:
>         
>         g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs
>         (rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=131072)
>         
>          
>         
>         So the problem is this:
>         
>         [root at g-unit-1 ~]# rm
>         -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final
>         
>         rm: cannot remove
>         `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages':
>         Directory not empty
>         
>         rm: cannot remove
>         `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT':
>         Directory not empty
>         
>         [root at g-unit-1 ~]#
>         
>          
>         
>         And the delete leaves behind 2000 or so files:
>         
>         [root at g-unit-1 ~]#
>         find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/ -type f |
>         wc -l
>         
>         2185
>         
>         Almost all the left behind files are on the 2nd node,
>         g-unit-2, but there are a few on g-unit-1
>         
>          
>         
>         From the 2nd node, g-unit-2:
>         
>         [root at g-unit-2 ~]# getfattr -m . -d -e
>         hex /mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi
>         
>         getfattr: Removing leading '/' from absolute path names
>         
>         # file:
>         mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi
>         
>         trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f
>         
>         So the file is there, is tagged with some gluster info, and
>         has definitely been copied onto the volume properly, via the
>         glusterfs mount. It hasn’t been copied directly onto the
>         brick.
>         
>          
>         
>         But from g-unit-1, accessing via the glusterfs mount, and
>         trying to delete this file:
>         
>         [root at g-unit-1 ~]#
>         rm /gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi
>         
>         rm: remove regular file
>         `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y
>         
>         rm: cannot remove
>         `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi': No such file or directory
>         
>          
>         
>         What the hell!?!!!!!
>         
>          
>         
>         Can someone please suggest the best course of action to
>         resolve this and prevent it happening in the future?
>         
>          
>         
>         Thanks for any assistance.
>         
>          
>         
>         John
>         
>          
>         
>          
>         
>          
>         
>          
>         
>          
>         
>          
>         
>         
>         
>         
>         _______________________________________________
>         Gluster-users mailing list
>         Gluster-users at gluster.org
>         http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
>  
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users