[Bugs] [Bug 1278419] New: Data Tiering:Data Loss:File migrations(flushing of data) to cold tier fails on detach tier with quota limits reached
bugzilla at redhat.com
bugzilla at redhat.com
Thu Nov 5 12:35:41 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1278419
Bug ID: 1278419
Summary: Data Tiering:Data Loss:File migrations(flushing of
data) to cold tier fails on detach tier with quota
limits reached
Product: Red Hat Gluster Storage
Version: 3.1
Component: glusterfs
Sub Component: tiering
Severity: urgent
Priority: urgent
Assignee: rhs-bugs at redhat.com
Reporter: nchilaka at redhat.com
QA Contact: nchilaka at redhat.com
CC: bugs at gluster.org, nchilaka at redhat.com,
vagarwal at redhat.com, vmallika at redhat.com
Depends On: 1265623, 1266841, 1267812
Blocks: 1260923
+++ This bug was initially created as a clone of Bug #1266841 +++
Description of problem:
======================
When quota limits are reached and then we try to do a detach tier.
The detach tier completes but fails to flush data to cold tier.
All the files are listed as failed
Version-Release number of selected component (if applicable):
==========================================================
[root at zod ~]# rpm -qa|grep gluster
glusterfs-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-api-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-client-xlators-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-server-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-cli-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-libs-3.7.4-0.43.gitf139283.el7.centos.x86_64
[root at zod ~]# gluster --version
glusterfs 3.7.4 built on Sep 19 2015 01:30:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
[root at zod ~]#
Steps to Reproduce:
==================
1.create a tier volume
2.enable quota and set hardlimit of root volume / to say 10gb
3. Now enable ctr and set the demote freq to say 1500sec
3.now fill the hardlimit to full
4. Now issue a detach tier start
It can be seen that the detach tier completes but fails as all the files would
have failed to be flushed to cold tier. Only link files would have been created
in cold bricks but no actual data movement would have happened
Eg:
Cold brick:
/rhs/brick3/angola:
total 8
---------T. 2 root root 0 Sep 28 12:47 file.11
---------T. 2 root root 0 Sep 28 12:47 file.12
---------T. 2 root root 0 Sep 28 12:47 file.13
---------T. 2 root root 0 Sep 28 12:47 file.15
---------T. 2 root root 0 Sep 28 12:47 file.32
---------T. 2 root root 0 Sep 28 12:47 file.33
---------T. 2 root root 0 Sep 28 12:47 file.35
---------T. 2 root root 0 Sep 28 12:47 file.42
---------T. 2 root root 0 Sep 28 12:47 file.44
---------T. 2 root root 0 Sep 28 12:47 file.45
---------T. 2 root root 0 Sep 28 12:47 file.54
---------T. 2 root root 0 Sep 28 12:47 file.62
---------T. 2 root root 0 Sep 28 12:47 file.63
drwxr-xr-x. 2 root root 8192 Sep 28 12:46 hotdir1
hot brick:
/rhs/brick7/angola_hot:
total 1269588
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:40 file.31
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.38
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.41
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.43
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.47
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.49
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.52
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.57
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:44 file.6
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.64
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:41 file.66
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:42 file.68
-rw-r-Sr-T. 2 root root 100000000 Sep 28 12:44 file.8
drwxr-xr-x. 2 root root 8192 Sep 28 12:37 hotdir1
Mount point, after commit of detach tier:
========================================
[root at localhost angola]# du -sh *
0 file.1
0 file.10
0 file.11
0 file.12
0 file.13
0 file.14
0 file.15
0 file.2
0 file.3
0 file.31
0 file.32
0 file.33
0 file.34
0 file.35
0 file.36
0 file.37
0 file.38
0 file.39
0 file.4
0 file.40
0 file.41
0 file.42
0 file.43
0 file.44
0 file.45
0 file.46
0 file.47
0 file.48
0 file.49
0 file.5
0 file.50
0 file.51
0 file.52
0 file.53
0 file.54
0 file.55
0 file.56
0 file.57
0 file.58
0 file.59
0 file.6
0 file.60
0 file.61
0 file.62
0 file.63
0 file.64
0 file.65
0 file.66
0 file.67
--- Additional comment from nchilaka on 2015-09-28 03:46:18 EDT ---
sosreports at rhsqe-repo.lab.eng.blr.redhat.com:/home/repo/sosreports/bug.1266841
--- Additional comment from Vijaikumar Mallikarjuna on 2015-09-28 05:55:29 EDT
---
Hi Nag Pavan,
What is the re-balance status? does it show any failure numbers. In this case
it is expected that files that have failed to migrate will can be lost when
commit is performed
--- Additional comment from nchilaka on 2015-09-28 07:17:19 EDT ---
Yes, there are failures under rebalance.
But given that detach tier must just move files b/w bricks and the end user
doesn't bother about how it is done. We must not be seeing any failures.
May be I should change the title.
In Short "quotas must not stop data being moved from hot to cold tier during a
detach tier"
--- Additional comment from Vijaikumar Mallikarjuna on 2015-10-01 02:29:18 EDT
---
Patch submitted: http://review.gluster.org/#/c/12266/
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1260923
[Bug 1260923] Tracker for tiering in 3.1.2
https://bugzilla.redhat.com/show_bug.cgi?id=1265623
[Bug 1265623] Data Tiering:Promotions and demotions fail after quota hard
limits are hit for a tier volume
https://bugzilla.redhat.com/show_bug.cgi?id=1266841
[Bug 1266841] Data Tiering:Data Loss:File migrations(flushing of data) to
cold tier fails on detach tier with quota limits reached
https://bugzilla.redhat.com/show_bug.cgi?id=1267812
[Bug 1267812] Data Tiering:Promotions and demotions fail after quota hard
limits are hit for a tier volume
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=uoPWj37p8I&a=cc_unsubscribe
More information about the Bugs
mailing list