[Bugs] [Bug 1344561] New: conservative merge happening on a x3 volume for a deleted file

bugzilla at redhat.com bugzilla at redhat.com
Fri Jun 10 04:21:58 UTC 2016


https://bugzilla.redhat.com/show_bug.cgi?id=1344561

            Bug ID: 1344561
           Summary: conservative merge happening on a x3 volume for a
                    deleted file
           Product: GlusterFS
           Version: 3.7.12
         Component: replicate
          Keywords: Triaged
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: ravishankar at redhat.com
                CC: bugs at gluster.org, nchilaka at redhat.com
        Depends On: 1341190, 1341650
            Blocks: 1344559



+++ This bug was initially created as a clone of Bug #1341650 +++

+++ This bug was initially created as a clone of Bug #1341190 +++

Description of problem:
=========================
When I set the object limit of a directory and bring down a brick and then
exceed quota object limits and then delete an existing file, I see that a
conservative merge happens for that file once the brick is brought online,
instead of deleting the file from the offline brick


Version-Release number of selected component (if applicable):
=====
3.7.9-6



Steps to Reproduce:
1.created a 2x3 volume
2.enabled quota, set the hard and soft timeouts to 0
3.fuse mounted the vol
4. created a dir /dir11
5. set the quota obj-limit to 10
6. created file f{1..5}--> f1 got created on subvol-1 and remaining on sbvol2
7. brought down brick6( the 3rd brick of 2nd subvol
8. created f{6..9} using touch -->passed
9. 10th file create failed as below
touch f10
touch: cannot touch ‘f10’: Disk quota exceeded
10. now did an rm of f5
[root at dhcp35-103 dir11]# rm f5
rm: remove regular empty file ‘f5’? y
==> that means now the quota level has not exceeded limit
11. more file creates   
[root at dhcp35-103 dir11]# touch f10
[root at dhcp35-103 dir11]# touch f11
touch: cannot touch ‘f11’: Disk quota exceeded

12. bring back down brick online using start force
[root at dhcp35-103 dir11]#  ####now going to bring bricks back###

12. checked that instead of f5 getting deleted from brick6, a conserveative
merge happend and f5 is created on all bricks.

==>also the quota limit has now crossed the limit from backend (heal) operation

                  Path                   Hard-limit   Soft-limit      Files    
  Dirs     Available  Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------

/dir11                                          10       80%(8)         10     
   1           0             Yes                  Yes






Getfattr details of subvol 2 :

brick4(1st of subvol2):
==>before bringing brick6 down
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

==>after bringing brick6 down and creating some files and quota limit exceeded
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

=-==>after delete of file f5
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.afr.distrepx3-client-5=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001

[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or
directory


=> after created f10(which could accomdate due to deelte of f5 whih passed
while f11 failed due to quota exceeded##
[root at nchilaka-node1 glusterfs]# 
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000002
trusted.afr.distrepx3-client-5=0x000000000000000000000002
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

===>after bringing brick6 online using force start
[root at nchilaka-node1 glusterfs]# ####now going to bring bricks back###
[root at nchilaka-node1 glusterfs]# 
[root at nchilaka-node1 glusterfs]# 
[root at nchilaka-node1 glusterfs]# ##brick is back online
[root at nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node1 glusterfs]# 
[root at nchilaka-node1 glusterfs]# 
[root at nchilaka-node1 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node1 glusterfs]# 
[root at nchilaka-node1 glusterfs]# 
[root at nchilaka-node1 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001

[root at nchilaka-node1 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.bit-rot.version=0x0200000000000000574d56f500040650
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001




=======>getfattr for brick5(2nd brick of subvol2)
[root at nchilaka-node2 glusterfs]# ################ dir11 start##############
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
[root at nchilaka-node2 glusterfs]# #getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root at nchilaka-node2 glusterfs]# #########bringing down brick
[root at nchilaka-node2 glusterfs]# ps -ef|grep 4308
root      4197 27604  0 17:22 pts/0    00:00:00 grep 4308
[root at nchilaka-node2 glusterfs]# kill -9 4308
-bash: kill: (4308) - No such process
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# ##deleted f5 and created some files and quota
limit exceeded#
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# #getfattr of deleted file #######;getfattr -d
-m . -e hex /rhs/brick2/dist*/dir11/f5
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root at nchilaka-node2 glusterfs]# ##sorry tried to delete f7 which was not even
part of filsystem during that time
[root at nchilaka-node2 glusterfs]# ##retrying delete of f5
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# ##deleted f5
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000001
trusted.afr.distrepx3-client-5=0x000000000000000000000001
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000030000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000030000000000000001

[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5getfattr: /rhs/brick2/dist*/dir11/f5: No such file or
directory
[root at nchilaka-node2 glusterfs]# ##created f10 whih passed while f11 failed due
to quota exceeded##
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000002
trusted.afr.distrepx3-client-5=0x000000000000000000000002
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root at nchilaka-node2 glusterfs]# ####now going to bring bricks back###
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# ##brick is back online
[root at nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# 
[root at nchilaka-node2 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.distrepx3-client-5=0x000000000000000000000000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001

[root at nchilaka-node2 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.bit-rot.version=0x0200000000000000574d56f70008f91f
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001


===>brick6 (3rd brick of subvol2) The brick which is brought down as part of
test


[root at nchilaka-node4 glusterfs]# ################ dir11 start##############
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
[root at nchilaka-node4 glusterfs]# #getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root at nchilaka-node4 glusterfs]# #########bringing down brick
[root at nchilaka-node4 glusterfs]# ps -ef|grep 4308
root      4308     1  0 17:18 ?        00:00:00 /usr/sbin/glusterfsd -s
10.70.42.123 --volfile-id distrepx3.10.70.42.123.rhs-brick2-distrepx3 -p
/var/lib/glusterd/vols/distrepx3/run/10.70.42.123-rhs-brick2-distrepx3.pid -S
/var/run/gluster/3e4dee7a2dcfb7735da718783949470e.socket --brick-name
/rhs/brick2/distrepx3 -l /var/log/glusterfs/bricks/rhs-brick2-distrepx3.log
--xlator-option *-posix.glusterd-uuid=51bc4ded-1ebb-4ec3-b5a3-2b5de0b025d1
--brick-port 49162 --xlator-option distrepx3-server.listen-port=49162
root      4404 27788  0 17:22 pts/0    00:00:00 grep 4308
[root at nchilaka-node4 glusterfs]# kill -9 4308
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# ##deleted f5 and created some files and quota
limit exceeded#
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# #getfattr of deleted file #######;getfattr -d
-m . -e hex /rhs/brick2/dist*/dir11/f5
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root at nchilaka-node4 glusterfs]# ##sorry tried to delete f7 which was not even
part of filsystem during that time
[root at nchilaka-node4 glusterfs]# ##retrying delete of f5
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# ##deleted f5
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5getfattr: Removing leading '/' from absolute path
names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root at nchilaka-node4 glusterfs]# ##created f10 whih passed while f11 failed due
to quota exceeded##
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000040000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000040000000000000001

[root at nchilaka-node4 glusterfs]# ####now going to bring bricks back###
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# ##brick is back online
[root at nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# ll /rhs/brick1/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f4
-rw-r--r--. 2 root root 0 May 31 17:04 f6
-rw-r--r--. 2 root root 0 May 31 17:04 f7
-rw-r--r--. 2 root root 0 May 31 17:04 f8
-rw-r--r--. 2 root root 0 May 31 17:04 f9
[root at nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# ll /rhs/brick2/dist*/dir11/
total 0
-rw-r--r--. 2 root root 0 May 31 17:02 f1
-rw-r--r--. 2 root root 0 May 31 17:07 f10
-rw-r--r--. 2 root root 0 May 31 17:02 f2
-rw-r--r--. 2 root root 0 May 31 17:02 f3
-rw-r--r--. 2 root root 0 May 31 17:02 f5
[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.gfid=0xff099d82666b4b51b79ce2de4cc233f8
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
trusted.glusterfs.quota.00000000-0000-0000-0000-000000000001.contri.1=0x000000000000000000000000000000050000000000000001
trusted.glusterfs.quota.dirty=0x3000
trusted.glusterfs.quota.limit-objects.1=0x000000000000000affffffffffffffff
trusted.glusterfs.quota.size.1=0x000000000000000000000000000000050000000000000001

[root at nchilaka-node4 glusterfs]# getfattr -d -m . -e hex
/rhs/brick2/dist*/dir11/f5
getfattr: Removing leading '/' from absolute path names
# file: rhs/brick2/distrepx3/dir11/f5
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a66696c655f743a733000
trusted.afr.distrepx3-client-3=0x000000000000000000000000
trusted.afr.distrepx3-client-4=0x000000000000000000000000
trusted.gfid=0x7279779380764113be45950ae7484d06
trusted.glusterfs.quota.ff099d82-666b-4b51-b79c-e2de4cc233f8.contri.1=0x00000000000000000000000000000001
trusted.pgfid.ff099d82-666b-4b51-b79c-e2de4cc233f8=0x00000001

[root at nchilaka-node4 glusterfs]# 
[root at nchilaka-node4 glusterfs]# #############creating new vol testnew
[root at nchilaka-node4 glusterfs]# rm -rf /rhs/brick1/testnew
[root at nchilaka-node4 glusterfs]# 




===>other cmd o/p
[root at nchilaka-node1 ~]# gluster v status distrepx3
Status of volume: distrepx3
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.43.22:/rhs/brick1/distrepx3     49160     0          Y       5130 
Brick 10.70.42.188:/rhs/brick1/distrepx3    49158     0          Y       31389
Brick 10.70.42.123:/rhs/brick1/distrepx3    49161     0          Y       2296 
Brick 10.70.43.22:/rhs/brick2/distrepx3     49161     0          Y       5149 
Brick 10.70.42.188:/rhs/brick2/distrepx3    49159     0          Y       31408
Brick 10.70.42.123:/rhs/brick2/distrepx3    49162     0          Y       4776 
NFS Server on localhost                     2049      0          Y       11839
Self-heal Daemon on localhost               N/A       N/A        Y       11847
Quota Daemon on localhost                   N/A       N/A        Y       11855
NFS Server on 10.70.42.123                  2049      0          Y       4796 
Self-heal Daemon on 10.70.42.123            N/A       N/A        Y       4804 
Quota Daemon on 10.70.42.123                N/A       N/A        Y       4813 
NFS Server on 10.70.42.204                  2049      0          Y       27323
Self-heal Daemon on 10.70.42.204            N/A       N/A        Y       27331
Quota Daemon on 10.70.42.204                N/A       N/A        Y       27339
NFS Server on 10.70.42.188                  2049      0          Y       4573 
Self-heal Daemon on 10.70.42.188            N/A       N/A        Y       4581 
Quota Daemon on 10.70.42.188                N/A       N/A        Y       4590 

Task Status of Volume distrepx3
------------------------------------------------------------------------------
There are no active volume tasks

[root at nchilaka-node1 ~]# gluster v quota distrepx3 list-obj
                  Path                   Hard-limit   Soft-limit      Files    
  Dirs     Available  Soft-limit exceeded? Hard-limit exceeded?
-----------------------------------------------------------------------------------------------------------------------------------------------
/dir1                                           35       80%(28)         90    
    1           0             Yes                  Yes
/dir2                                          200       80%(160)        300   
     1           0             Yes                  Yes
/dir3                                         1100       80%(880)       1210   
     1           0             Yes                  Yes
/dir4                                           50       80%(40)         40    
    1           9             Yes                   No
/dir5                                          100       80%(80)        199    
    1           0             Yes                  Yes
/dir6                                          100       80%(80)         99    
    1           0             Yes                  Yes
/dir7                                           10       80%(8)         10     
   1           0             Yes                  Yes
/dir8                                           10       80%(8)          9     
   1           0             Yes                  Yes
/dir10                                          10       80%(8)          9     
   1           0             Yes                  Yes
/dir11                                          10       80%(8)         10     
   1           0             Yes                  Yes
[root at nchilaka-node1 ~]# 
[root at nchilaka-node1 ~]# 
[root at nchilaka-node1 ~]# gluster v heal distrepx3 info
Brick 10.70.43.22:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.188:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.123:/rhs/brick1/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.43.22:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.188:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0

Brick 10.70.42.123:/rhs/brick2/distrepx3
Status: Connected
Number of entries: 0



Volume Name: distrepx3
Type: Distributed-Replicate
Volume ID: 353f80dc-aa40-468d-b6ae-8604b79e3397
Status: Started
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: 10.70.43.22:/rhs/brick1/distrepx3
Brick2: 10.70.42.188:/rhs/brick1/distrepx3
Brick3: 10.70.42.123:/rhs/brick1/distrepx3
Brick4: 10.70.43.22:/rhs/brick2/distrepx3
Brick5: 10.70.42.188:/rhs/brick2/distrepx3
Brick6: 10.70.42.123:/rhs/brick2/distrepx3
Options Reconfigured:
features.soft-timeout: 0
features.hard-timeout: 0
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on


--- Additional comment from Ravishankar N on 2016-06-01 05:30:31 EDT ---

RCA:
I was able to re-create the issue on a 1x3 plain replica (RHGS 3.1.3) with the
same steps given in the description. If one of the bricks of a replica is down,
AFR marks dirty xattrs for even entry transactions ('touch f10' in this case)
in the pre-op. In the post-op, since both the up bricks return EDQUOT, we
consider it a symmetric error and don't do the post-op, retaining the dirty
xattr for the parent directory. When self-heal is triggered when all bricks are
up, it considers directories with dirty xattr as a candidate for conservative
merge. Hence the deleted file is re created on the source bricks instead of
being deleted from the sink.

--- Additional comment from Vijay Bellur on 2016-06-01 08:42:44 EDT ---

REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as
symmetric errors) posted (#1) for review on master by Ravishankar N
(ravishankar at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-01 08:45:08 EDT ---

REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as
symmetric errors) posted (#2) for review on master by Ravishankar N
(ravishankar at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-06 07:22:48 EDT ---

REVIEW: http://review.gluster.org/14604 (afr: Consider ENOSPC and EDQUOT as
symmetric errors) posted (#3) for review on master by Ravishankar N
(ravishankar at redhat.com)


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1341190
[Bug 1341190] conservative merge happening on a x3 volume for a deleted
file
https://bugzilla.redhat.com/show_bug.cgi?id=1341650
[Bug 1341650] conservative merge happening on a x3 volume for a deleted
file
https://bugzilla.redhat.com/show_bug.cgi?id=1344559
[Bug 1344559] conservative merge happening on a x3 volume for a deleted
file
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list