[Gluster-users] Unable to delete files but getfattr shows file is part of glusterfs

[SINCOCK John]

Thanks Pranith, I hope Susant can help!

 

From: Pranith Kumar Karampuri [mailto:pkarampu at redhat.com] 
Sent: Wednesday, 18 June 2014 11:48 AM
To: SINCOCK John; gluster-users at gluster.org
Cc: Susant Palai
Subject: Re: [Gluster-users] Unable to delete files but getfattr shows
file is part of glusterfs

 

I think Susant knows about this issue

CCed him

Pranith

On 06/15/2014 07:16 PM, SINCOCK John wrote:

	Hi Everybody,

	 

	We have recently setup a 2-node gluster, and are having some
problems deleting some files!

	We haven't accessed the gluster bricks directly, all access has
been through the glusterfs mount 

	The 1st node in our cluster also handles sharing the glusterfs
volume via samba, to windows clients.

	 

	Environment is:

	Kernel: 2.6.32-431.1.2.0.1.el6.x86_64 #1 SMP Fri Dec 13 13:06:13
UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

	CentOS release 6.4 (Final)

	 

	glusterfs-libs-3.4.1-3.el6.x86_64

	glusterfs-cli-3.4.1-3.el6.x86_64

	glusterfs-fuse-3.4.1-3.el6.x86_64

	glusterfs-server-3.4.1-3.el6.x86_64

	glusterfs-3.4.1-3.el6.x86_64

	 

	               4 bricks on each node

	               Gluster is setup without replication or striping

	 

	               Bricks on one of the two nodes (the other node is
setup the same):

	/dev/sda1             9.1T  4.7T  4.5T  51%
/mnt/glusterfs/bricks/1

	/dev/sda2             9.1T  3.7T  5.5T  40%
/mnt/glusterfs/bricks/2

	/dev/sda3             9.1T  3.7T  5.5T  40%
/mnt/glusterfs/bricks/3

	/dev/sda4             9.1T  4.1T  5.0T  46%
/mnt/glusterfs/bricks/4

	 

	Gluster volume mounted on 1st node:

	g-unit-1:/vol00        73T   31T   43T  42% /gluster/vol00

	               Mount options:

	g-unit-1:/vol00 on /gluster/vol00 type fuse.glusterfs
(rw,nosuid,nodev,noatime,default_permissions,allow_other,max_read=131072
)

	 

	So the problem is this:

	[root at g-unit-1 ~]# rm -Rf
/gluster/vol00/data/CentOS3/CentOS_6.4_Final

	rm: cannot remove
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/Packages': Directory not
empty

	rm: cannot remove
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT': Directory not
empty

	[root at g-unit-1 ~]#

	 

	And the delete leaves behind 2000 or so files:

	[root at g-unit-1 ~]# find
/gluster/vol00/data/CentOS3/CentOS_6.4_Final/ -type f | wc -l

	2185

	Almost all the left behind files are on the 2nd node, g-unit-2,
but there are a few on g-unit-1

	 

	From the 2nd node, g-unit-2:

	[root at g-unit-2 ~]# getfattr -m . -d -e hex
/mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.e
fi

	getfattr: Removing leading '/' from absolute path names

	# file:
mnt/glusterfs/bricks/2/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.ef
i

	trusted.gfid=0xa9b0a66cf39b49c7890b2da3f631929f

	So the file is there, is tagged with some gluster info, and has
definitely been copied onto the volume properly, via the glusterfs
mount. It hasn't been copied directly onto the brick.

	 

	But from g-unit-1, accessing via the glusterfs mount, and trying
to delete this file:

	[root at g-unit-1 ~]# rm
/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi

	rm: remove regular file
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi'? y

	rm: cannot remove
`/gluster/vol00/data/CentOS3/CentOS_6.4_Final/EFI/BOOT/BOOTX64.efi': No
such file or directory

	 

	What the hell!?!!!!!

	 

	Can someone please suggest the best course of action to resolve
this and prevent it happening in the future?

	 

	Thanks for any assistance.

	 

	John

	 

	 

	 

	 

	 

	 

	
	
	
	

	_______________________________________________
	Gluster-users mailing list
	Gluster-users at gluster.org
	http://supercolony.gluster.org/mailman/listinfo/gluster-users

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140618/29b0064b/attachment.html>