[Gluster-users] Unable to delete files but getfattr shows file is part of glusterfs
Franco Broi
franco.broi at iongeo.com
Wed Jun 18 08:26:12 UTC 2014
The major problem with 3.5 for me is that you can't have 3.4 clients,
well actually you can but you have to add them slowly, try too many at
once and the mount operation hangs. The only way to get out of this is
to stop and start the volume but then the existing 3.4 mounts all appear
to be empty.
Cheers,
PS. I'm ex-Fugro (FSI) Perth...
On Wed, 2014-06-18 at 17:49 +0930, SINCOCK John wrote:
> Thanks Franco,
> that's interesting, I've read some more emails to this list now, and yes this does look like the same issue that other people are seeing.
> So I hope those patches will fix the problem.
>
> I am not in any hurry to upgrade to 3.5 considering there still seem to be some other issues with 3.5 - but this seems to be quite a serious issue, if files copied to a gluster are not accessible if their parent directory did not end up on the longest-lived node!
>
> I'll have to keep a close eye on this issue and upgrade as soon as the 3.5 looks stable
>
> Thanks very much for the info.
>
> Cheers and Regards,
> John
>
>
>
>
> -----Original Message-----
> From: Franco Broi [mailto:franco.broi at iongeo.com]
> Sent: Wednesday, 18 June 2014 3:15 PM
> To: SINCOCK John
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] Unable to delete files but getfattr shows file is part of glusterfs
>
>
> Hi John
>
> I got this yesterday, it was copied to this list.
>
> Cheers,
>
>
> On Tue, 2014-06-17 at 04:55 -0400, Susant Palai wrote:
> Hi Franco:
> > The following patches address the ENOTEMPTY issue.
> >
> > 1. http://review.gluster.org/#/c/7733/
> > 2. http://review.gluster.org/#/c/7599/
> >
> > I think the above patches will be available in 3.5.1 which will be a
> > minor upgrade.(Need ack from Niels de Vos.)
> >
>
>
> On Wed, 2014-06-18 at 13:26 +0930, SINCOCK John wrote:
> > Thanks Pranith, I hope Susant can help!
> >
> >
> >
> > From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com]
> > Sent: Wednesday, 18 June 2014 11:48 AM
> > To: SINCOCK John; gluster-users at gluster.org
> > Cc: Susant Palai
> > Subject: Re: [Gluster-users] Unable to delete files but getfattr shows
> > file is part of glusterfs
> >
> >
> >
> >
> > I think Susant knows about this issue
> >
> > CCed him
> >
> > Pranith
> >
> > On 06/15/2014 07:16 PM, SINCOCK John wrote:
> >
> >
> > Hi Everybody,
> >
> >
> >
> > We have recently setup a 2-node gluster, and are having some
> > problems deleting some files!
> >
> > We haven’t accessed the gluster bricks directly, all access
> > has been through the glusterfs mount
> >
> > The 1st node in our cluster also handles sharing the glusterfs
> > volume via samba, to windows clients.
> >
> >
> >
> > Environment is:
> >
> > Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13
> > 13:06:13 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
> >
> > CentOS release 6.4 (Final)
> >
> >
> >
> > glusterfs-libs-3.4.1-3.el6.x86_64
> >
> > glusterfs-cli-3.4.1-3.el6.x86_64
> >
> > glusterfs-fuse-3.4.1-3.el6.x86_64
> >
> > glusterfs-server-3.4.1-3.el6.x86_64
> >
> > glusterfs-3.4.1-3.el6.x86_64
> >
> >
> >
> > 4 bricks on each node
> >
> > Gluster is setup without replication or
> > striping
> >
> >
> >
> > Bricks on one of the two nodes (the other node
> > is setup the same):
> >
> > /dev/sda1 9.1T 4.7T 4.5T
> > 51% /mnt/glusterfs/bricks/1
> >
> > /dev/sda2 9.1T 3.7T 5.5T
> > 40% /mnt/glusterfs/bricks/2
> >
> > /dev/sda3 9.1T 3.7T 5.5T
> > 40% /mnt/glusterfs/bricks/3
> >
> > /dev/sda4 9.1T 4.1T 5.0T
> > 46% /mnt/glusterfs/bricks/4
> >
> >
> >
> > Gluster volume mounted on 1st node:
> >
> > g-unit-1:/vol00 73T 31T 43T 42% /gluster/vol00
> >
> > Mount options:
> >
> > g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs
> >
> > (rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=1310
> > 72)
> >
> >
> >
> > So the problem is this:
> >
> > [root at g-unit-1 ~]# rm
> > -Rf /gluster/vol00/data/CentOS3/CentOS_6.4_Final
> >
> > rm: cannot remove
> > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages':
> > Directory not empty
> >
> > rm: cannot remove
> > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT':
> > Directory not empty
> >
> > [root at g-unit-1 ~]#
> >
> >
> >
> > And the delete leaves behind 2000 or so files:
> >
> > [root at g-unit-1 ~]#
> > find /gluster/vol00/data/CentOS3/CentOS_6.4_Final/ -type f |
> > wc -l
> >
> > 2185
> >
> > Almost all the left behind files are on the 2nd node,
> > g-unit-2, but there are a few on g-unit-1
> >
> >
> >
> > From the 2nd node, g-unit-2:
> >
> > [root at g-unit-2 ~]# getfattr -m . -d -e
> > hex
> > /mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64
> > .efi
> >
> > getfattr: Removing leading '/' from absolute path names
> >
> > # file:
> >
> > mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.
> > efi
> >
> > trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f
> >
> > So the file is there, is tagged with some gluster info, and
> > has definitely been copied onto the volume properly, via the
> > glusterfs mount. It hasn’t been copied directly onto the
> > brick.
> >
> >
> >
> > But from g-unit-1, accessing via the glusterfs mount, and
> > trying to delete this file:
> >
> > [root at g-unit-1 ~]#
> > rm
> > /gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi
> >
> > rm: remove regular file
> >
> > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y
> >
> > rm: cannot remove
> >
> > `/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi':
> > No such file or directory
> >
> >
> >
> > What the hell!?!!!!!
> >
> >
> >
> > Can someone please suggest the best course of action to
> > resolve this and prevent it happening in the future?
> >
> >
> >
> > Thanks for any assistance.
> >
> >
> >
> > John
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >
> >
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
More information about the Gluster-users
mailing list