[Bugs] [Bug 1344559] New: conservative merge happening on a x3 volume for a deleted file
bugzilla at redhat.com
bugzilla at redhat.com
Fri Jun 10 04:12:56 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1344559
Bug ID: 1344559
Summary: conservative merge happening on a x3 volume for a
deleted file
Product: GlusterFS
Version: 3.8.0
Component: replicate
Keywords: Triaged
Severity: urgent
Assignee: bugs at gluster.org
Reporter: ravishankar at redhat.com
CC: bugs at gluster.org, nchilaka at redhat.com
Depends On: 1341190, 1341650
Description of problem:
=========================
When I set the object limit of a directory and bring down a brick and then
exceed quota object limits and then delete an existing file, I see that a
conservative merge happens for that file once the brick is brought online,
instead of deleting the file from the offline brick
Version-Release number of selected component (if applicable):
=====
3.7.9-6
Steps to Reproduce:
1.created a 2x3 volume
2.enabled quota, set the hard and soft timeouts to 0
3.fuse mounted the vol
4. created a dir /dir11
5. set the quota obj-limit to 10
6. created file f{1..5}--> f1 got created on subvol-1 and remaining on sbvol2
7. brought down brick6( the 3rd brick of 2nd subvol
8. created f{6..9} using touch -->passed
9. 10th file create failed as below
touch f10
touch: cannot touch ‘f10’: Disk quota exceeded
10. now did an rm of f5
[root at dhcp35-103 dir11]# rm f5
rm: remove regular empty file ‘f5’? y
==> that means now the quota level has not exceeded limit
11. more file creates
[root at dhcp35-103 dir11]# touch f10
[root at dhcp35-103 dir11]# touch f11
touch: cannot touch ‘f11’: Disk quota exceeded
12. bring back down brick online using start force
[root at dhcp35-103 dir11]# ####now going to bring bricks back###
12. checked that instead of f5 getting deleted from brick6, a conserveative
merge happend and f5 is created on all bricks.
==>also the quota limit has now crossed the limit from backend (heal) operation
Path Hard-limit Soft-limit Files
Dirs Available Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------
/dir11 10 80%(8) 10
1 0 Yes Yes
Getfattr details of subvol 2 :
brick4(1st of subvol2):
==>before bringing brick6 down
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
==>after bringing brick6 down and creating some files and quota limit exceeded
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
=-==>after delete of file f5
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.afr.distrepx3-client-5=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or
directory
=> after created f10(which could accomdate due to deelte of f5 whih passed
while f11 failed due to quota exceeded##
[root at nchilaka-node1 glusterfs]#
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000002
trusted.afr.distrepx3-client-5=0x000000000000000000000002
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
===>after bringing brick6 online using force start
[root at nchilaka-node1 glusterfs]# ####now going to bring bricks back###
[root at nchilaka-node1 glusterfs]#
[root at nchilaka-node1 glusterfs]#
[root at nchilaka-node1 glusterfs]# ##brick is back online
[root at nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node1 glusterfs]#
[root at nchilaka-node1 glusterfs]#
[root at nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node1 glusterfs]#
[root at nchilaka-node1 glusterfs]#
[root at nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.bit-rot.version=0x0200000000000000574d56f500040650
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001
=======>getfattr for brick5(2nd brick of subvol2)
[root at nchilaka-node2 glusterfs]# ################ dir11 start##############
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
[root at nchilaka-node2 glusterfs]# #getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
[root at nchilaka-node2 glusterfs]# #########bringing down brick
[root at nchilaka-node2 glusterfs]# ps -ef|grep 4308
root 4197 27604 0 17:22 pts/0 00:00:00 grep 4308
[root at nchilaka-node2 glusterfs]# kill -9 4308
-bash: kill: (4308) - No such process
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]# ##deleted f5 and created some files and quota
limit exceeded#
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]# #getfattr of deleted file #######;getfattr -d
-m . -e hex /rhs/brick2/dist*/dir11/f5
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001
[root at nchilaka-node2 glusterfs]# ##sorry tried to delete f7 which was not even
part of filsystem during that time
[root at nchilaka-node2 glusterfs]# ##retrying delete of f5
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]# ##deleted f5
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.afr.distrepx3-client-5=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or
directory
[root at nchilaka-node2 glusterfs]# ##created f10 whih passed while f11 failed due
to quota exceeded##
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000002
trusted.afr.distrepx3-client-5=0x000000000000000000000002
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
[root at nchilaka-node2 glusterfs]# ####now going to bring bricks back###
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]# ##brick is back online
[root at nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]#
[root at nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.bit-rot.version=0x0200000000000000574d56f70008f91f
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001
===>brick6 (3rd brick of subvol2) The brick which is brought down as part of
test
[root at nchilaka-node4 glusterfs]# ################ dir11 start##############
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
[root at nchilaka-node4 glusterfs]# #getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
[root at nchilaka-node4 glusterfs]# #########bringing down brick
[root at nchilaka-node4 glusterfs]# ps -ef|grep 4308
root 4308 1 0 17:18 ? 00:00:00 /usr/sbin/glusterfsd -s
10.70.42.123 --volfile-id distrepx3.10.70.42.123.rhs-brick2-distrepx3 -p
/var/lib/glusterd/vols/distrepx3/run/10.70.42.123-rhs-brick2-distrepx3.pid -S
/var/run/gluster/3e4dee7a2dcfb7735da718783949470e.socket --brick-name
/rhs/brick2/distrepx3 -l /var/log/glusterfs/bricks/rhs-brick2-distrepx3.log
--xlator-option *-posix.glusterd-uuid=51bc4ded-1ebb-4ec3-b5a3-2b5de0b025d1
--brick-port 49162 --xlator-option distrepx3-server.listen-port=49162
root 4404 27788 0 17:22 pts/0 00:00:00 grep 4308
[root at nchilaka-node4 glusterfs]# kill -9 4308
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]# ##deleted f5 and created some files and quota
limit exceeded#
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]# #getfattr of deleted file #######;getfattr -d
-m . -e hex /rhs/brick2/dist*/dir11/f5
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001
[root at nchilaka-node4 glusterfs]# ##sorry tried to delete f7 which was not even
part of filsystem during that time
[root at nchilaka-node4 glusterfs]# ##retrying delete of f5
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]# ##deleted f5
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5getfattr: Removing leading '/' from absolute path
names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001
[root at nchilaka-node4 glusterfs]# ##created f10 whih passed while f11 failed due
to quota exceeded##
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001
[root at nchilaka-node4 glusterfs]# ####now going to bring bricks back###
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]# ##brick is back online
[root at nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.distrepx3-client-3=0x000000000000000000000000
trusted.afr.distrepx3-client-4=0x000000000000000000000000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001
[root at nchilaka-node4 glusterfs]#
[root at nchilaka-node4 glusterfs]# #############creating new vol testnew
[root at nchilaka-node4 glusterfs]# rm -rf /rhs/brick1/testnew
[root at nchilaka-node4 glusterfs]#
===>other cmd o/p
[root at nchilaka-node1 ~]# gluster v status distrepx3
Status of volume: distrepx3
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.70.43.22:/rhs/brick1/distrepx3 49160 0 Y 5130
Brick 10.70.42.188:/rhs/brick1/distrepx3 49158 0 Y 31389
Brick 10.70.42.123:/rhs/brick1/distrepx3 49161 0 Y 2296
Brick 10.70.43.22:/rhs/brick2/distrepx3 49161 0 Y 5149
Brick 10.70.42.188:/rhs/brick2/distrepx3 49159 0 Y 31408
Brick 10.70.42.123:/rhs/brick2/distrepx3 49162 0 Y 4776
NFS Server on localhost 2049 0 Y 11839
Self-heal Daemon on localhost N/A N/A Y 11847
Quota Daemon on localhost N/A N/A Y 11855
NFS Server on 10.70.42.123 2049 0 Y 4796
Self-heal Daemon on 10.70.42.123 N/A N/A Y 4804
Quota Daemon on 10.70.42.123 N/A N/A Y 4813
NFS Server on 10.70.42.204 2049 0 Y 27323
Self-heal Daemon on 10.70.42.204 N/A N/A Y 27331
Quota Daemon on 10.70.42.204 N/A N/A Y 27339
NFS Server on 10.70.42.188 2049 0 Y 4573
Self-heal Daemon on 10.70.42.188 N/A N/A Y 4581
Quota Daemon on 10.70.42.188 N/A N/A Y 4590
Task Status of Volume distrepx3
------------------------------------------------------------------------------
There are no active volume tasks
[root at nchilaka-node1 ~]# gluster v quota distrepx3 list-obj
Path Hard-limit Soft-limit Files
Dirs Available Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------
/dir1 35 80%(28) 90
1 0 Yes Yes
/dir2 200 80%(160) 300
1 0 Yes Yes
/dir3 1100 80%(880) 1210
1 0 Yes Yes
/dir4 50 80%(40) 40
1 9 Yes No
/dir5 100 80%(80) 199
1 0 Yes Yes
/dir6 100 80%(80) 99
1 0 Yes Yes
/dir7 10 80%(8) 10
1 0 Yes Yes
/dir8 10 80%(8) 9
1 0 Yes Yes
/dir10 10 80%(8) 9
1 0 Yes Yes
/dir11 10 80%(8) 10
1 0 Yes Yes
[root at nchilaka-node1 ~]#
[root at nchilaka-node1 ~]#
[root at nchilaka-node1 ~]# gluster v heal distrepx3 info
Brick 10.70.43.22:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0
Brick 10.70.42.188:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0
Brick 10.70.42.123:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0
Brick 10.70.43.22:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0
Brick 10.70.42.188:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0
Brick 10.70.42.123:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0
Volume Name: distrepx3
Type: Distributed-Replicate
Volume ID: 353f80dc-aa40-468d-b6ae-8604b79e3397
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.43.22:/rhs/brick1/distrepx3
Brick2: 10.70.42.188:/rhs/brick1/distrepx3
Brick3: 10.70.42.123:/rhs/brick1/distrepx3
Brick4: 10.70.43.22:/rhs/brick2/distrepx3
Brick5: 10.70.42.188:/rhs/brick2/distrepx3
Brick6: 10.70.42.123:/rhs/brick2/distrepx3
Options Reconfigured:
features.soft-timeout: 0
features.hard-timeout: 0
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on
--- Additional comment from Ravishankar N on 2016-06-01 05:30:31 EDT ---
RCA:
I was able to re-create the issue on a 1x3 plain replica (RHGS 3.1.3) with the
same steps given in the description. If one of the bricks of a replica is down,
AFR marks dirty xattrs for even entry transactions ('touch f10' in this case)
in the pre-op. In the post-op, since both the up bricks return EDQUOT, we
consider it a symmetric error and don't do the post-op, retaining the dirty
xattr for the parent directory. When self-heal is triggered when all bricks are
up, it considers directories with dirty xattr as a candidate for conservative
merge. Hence the deleted file is re created on the source bricks instead of
being deleted from the sink.
-
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1341190
[Bug 1341190] conservative merge happening on a x3 volume for a deleted
file
https://bugzilla.redhat.com/show_bug.cgi?id=1341650
[Bug 1341650] conservative merge happening on a x3 volume for a deleted
file
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list