[Gluster-users] Unable to delete files but getfattr shows file is part of glusterfs

[Susant Palai]

Hi,
   Can you  upload the logs ?

Susant 


----- Original Message -----
From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
To: "SINCOCK John" <J.Sincock at fugro.com>, gluster-users at gluster.org
Cc: "Susant Palai" <spalai at redhat.com>
Sent: Wednesday, 18 June, 2014 7:48:19 AM
Subject: Re: [Gluster-users] Unable to delete files but getfattr shows file is part of glusterfs

I think Susant knows about this issue

CCed him

Pranith
On 06/15/2014 07:16 PM, SINCOCK John wrote:
>
> Hi Everybody,
>
> We have recently setup a 2-node gluster, and are having some problems 
> deleting some files!
>
> We haven't accessed the gluster bricks directly, all access has been 
> through the glusterfs mount
>
> The 1^st node in our cluster also handles sharing the glusterfs volume 
> via samba, to windows clients.
>
> Environment is:
>
> Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13 UTC 
> 2013 x86_64 x86_64 x86_64 GNU/Linux
>
> CentOS release 6.4 (Final)
>
> glusterfs-libs-3.4.1-3.el6.x86_64
>
> glusterfs-cli-3.4.1-3.el6.x86_64
>
> glusterfs-fuse-3.4.1-3.el6.x86_64
>
> glusterfs-server-3.4.1-3.el6.x86_64
>
> glusterfs-3.4.1-3.el6.x86_64
>
>                4 bricks on each node
>
>                Gluster is setup without replication or striping
>
>                Bricks on one of the two nodes (the other node is setup 
> the same):
>
> /dev/sda1        9.1T  4.7T  4.5T  51% /mnt/glusterfs/bricks/1
>
> /dev/sda2 9.1T  3.7T  5.5T  40% /mnt/glusterfs/bricks/2
>
> /dev/sda3 9.1T  3.7T  5.5T  40% /mnt/glusterfs/bricks/3
>
> /dev/sda4 9.1T  4.1T  5.0T  46% /mnt/glusterfs/bricks/4
>
> Gluster volume mounted on 1st node:
>
> g-unit-1:/vol00 73T   31T   43T  42% /gluster/vol00
>
>                Mount options:
>
> g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs 
> (rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=131072)
>
> *So the problem is this:*
>
> [root at g-unit-1 ~]# rm -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final
>
> rm: cannot remove 
> `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages': Directory not 
> empty
>
> rm: cannot remove 
> `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT': Directory not 
> empty
>
> [root at g-unit-1 ~]#
>
> And the delete leaves behind 2000 or so files:
>
> [root at g-unit-1 ~]# find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/ 
> -type f | wc -l
>
> 2185
>
> Almost all the left behind files are on the 2^nd node, g-unit-2, but 
> there are a few on g-unit-1
>
> From the 2^nd node, g-unit-2:
>
> [root at g-unit-2 ~]# getfattr -m . -d -e hex 
> /mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: 
> mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi
>
> trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f
>
> So the file is there, is tagged with some gluster info, and has 
> definitely been copied onto the volume properly, via the glusterfs 
> mount. It hasn't been copied directly onto the brick.
>
> But from g-unit-1, accessing via the glusterfs mount, and trying to 
> delete this file:
>
> [root at g-unit-1 ~]# rm 
> /gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi
>
> rm: remove regular file 
> `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y
>
> rm: cannot remove 
> `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi': 
> No such file or directory
>
> What the hell!?!!!!!
>
> Can someone please suggest the best course of action to resolve this 
> and prevent it happening in the future?
>
> Thanks for any assistance.
>
> John
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users