[Bugs] [Bug 1266841] New: Data Tiering:Data Loss:File migrations(flushing of data) to cold tier fails on detach tier with quota limits reached

Mon Sep 28 07:30:29 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1266841

            Bug ID: 1266841
           Summary: Data Tiering:Data Loss:File migrations(flushing of
                    data) to cold tier fails on detach tier with quota
                    limits reached
           Product: GlusterFS
           Version: 3.7.4
         Component: tiering
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: nchilaka at redhat.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org

Description of problem:
======================
When quota limits are reached and then we try to do a detach tier.
The detach tier completes but fails to flush data to cold tier.
All the files are listed as failed

Version-Release number of selected component (if applicable):
==========================================================
[root at zod ~]# rpm -qa|grep gluster
glusterfs-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-api-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-client-xlators-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-server-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-cli-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-libs-3.7.4-0.43.gitf139283.el7.centos.x86_64
[root at zod ~]# gluster --version
glusterfs 3.7.4 built on Sep 19 2015 01:30:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
[root at zod ~]# 

Steps to Reproduce:
==================
1.create a tier volume
2.enable quota and set hardlimit of root volume / to say 10gb
3. Now enable ctr and set the demote freq to say 1500sec
3.now fill the hardlimit to full
4. Now issue a detach tier start

It can be seen that the detach tier completes but fails as all the files would
have failed to be flushed to cold tier. Only link files would have been created
in cold bricks but no actual data movement would have happened

Eg:
Cold brick:
/rhs/brick3/angola:
total 8
---------T. 2 root root    0 Sep 28 12:47 file.11
---------T. 2 root root    0 Sep 28 12:47 file.12
---------T. 2 root root    0 Sep 28 12:47 file.13
---------T. 2 root root    0 Sep 28 12:47 file.15
---------T. 2 root root    0 Sep 28 12:47 file.32
---------T. 2 root root    0 Sep 28 12:47 file.33
---------T. 2 root root    0 Sep 28 12:47 file.35
---------T. 2 root root    0 Sep 28 12:47 file.42
---------T. 2 root root    0 Sep 28 12:47 file.44
---------T. 2 root root    0 Sep 28 12:47 file.45
---------T. 2 root root    0 Sep 28 12:47 file.54
---------T. 2 root root    0 Sep 28 12:47 file.62
---------T. 2 root root    0 Sep 28 12:47 file.63
drwxr-xr-x. 2 root root 8192 Sep 28 12:46 hotdir1

hot brick:
/rhs/brick7/angola_hot:
total 1269588
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:40 file.31
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.38
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.41
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.43
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.47
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.49
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.52
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.57
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:44 file.6
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.64
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.66
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:42 file.68
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:44 file.8
drwxr-xr-x. 2 root root      8192 Sep 28 12:37 hotdir1

Mount point, after commit of detach tier:
========================================
[root at localhost angola]# du -sh *
0    file.1
0    file.10
0    file.11
0    file.12
0    file.13
0    file.14
0    file.15
0    file.2
0    file.3
0    file.31
0    file.32
0    file.33
0    file.34
0    file.35
0    file.36
0    file.37
0    file.38
0    file.39
0    file.4
0    file.40
0    file.41
0    file.42
0    file.43
0    file.44
0    file.45
0    file.46
0    file.47
0    file.48
0    file.49
0    file.5
0    file.50
0    file.51
0    file.52
0    file.53
0    file.54
0    file.55
0    file.56
0    file.57
0    file.58
0    file.59
0    file.6
0    file.60
0    file.61
0    file.62
0    file.63
0    file.64
0    file.65
0    file.66
0    file.67

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.