[Bugs] [Bug 1267812] New: Data Tiering:Promotions and demotions fail after quota hard limits are hit for a tier volume

Thu Oct 1 06:15:26 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1267812

            Bug ID: 1267812
           Summary: Data Tiering:Promotions and demotions fail after quota
                    hard limits are hit for a tier volume
           Product: GlusterFS
           Version: mainline
         Component: tiering
          Severity: urgent
          Priority: urgent
          Assignee: bugs at gluster.org
          Reporter: vmallika at redhat.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org, dlambrig at redhat.com,
                    nbalacha at redhat.com, nchilaka at redhat.com
        Depends On: 1265623
            Blocks: 1260923, 1266841

+++ This bug was initially created as a clone of Bug #1265623 +++

Description of problem:
=====================
Once quota hard limits are hit for a tier volume, promotes and demotes are
failing.
Only link files are getting created, but files are not getting migrated.
As tiering migration is internal, it should do promos/demos even on quota
filled vols.

Version-Release number of selected component (if applicable):
==============================================================
[root at zod ~]# 
[root at zod ~]# 
[root at zod ~]# rpm -qa|grep gluster
glusterfs-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-fuse-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-debuginfo-3.7.4-0.33.git1d02d4b.el7.centos.x86_64
glusterfs-api-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-client-xlators-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-server-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-cli-3.7.4-0.43.gitf139283.el7.centos.x86_64
glusterfs-libs-3.7.4-0.43.gitf139283.el7.centos.x86_64
[root at zod ~]# gluster --version
glusterfs 3.7.4 built on Sep 19 2015 01:30:43
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
[root at zod ~]# 

Steps to Reproduce:
==================
1.create a tier volume
2.enable quota and set hardlimit to say 5gb
3.now fill the hardlimit and wait for demotes to happen

It can be seen that only link files are created but files are not really moved.

Errors:
=========

[2015-09-23 10:04:03.249877] E [MSGID: 109039]
[dht-helper.c:1176:dht_rebalance_inprogress_task] 0-taiwan-cold-dht: /h.125:
failed to get the 'linkto' xattr [No data available]
[2015-09-23 10:04:03.250197] E [MSGID: 109023]
[dht-rebalance.c:800:__dht_rebalance_open_src_file] 0-taiwan-tier-dht: failed
to set xattr on /h.125 in taiwan-cold-dht (Invalid argument)
[2015-09-23 10:04:03.250228] E [MSGID: 109023]
[dht-rebalance.c:1106:dht_migrate_file] 0-taiwan-tier-dht: Migrate file failed:
failed to open /h.125 on taiwan-cold-dht
[2015-09-23 10:04:03.250880] E [MSGID: 109037]
[tier.c:365:tier_migrate_using_query_file] 0-taiwan-tier-dht: ERROR -28 in
current migration h.125 /h.125

[root at zod ~]# gluster v rebal taiwan status
                                    Node Rebalanced-files          size      
scanned      failures       skipped               status   run time in secs
                               ---------      -----------   -----------  
-----------   -----------   -----------         ------------     --------------
                               localhost               12        0Bytes        
  216           204             0          in progress            3203.00
                                  yarrow              103        0Bytes        
  862           759             0          in progress            3203.00
volume rebalance: taiwan: success:

--- Additional comment from nchilaka on 2015-09-23 07:39:01 EDT ---

sosreports
[root at rhsqe-repo bug.1265623]# cd ..
[root at rhsqe-repo sosreports]# chmod -R 0777 bug.1265623
[root at rhsqe-repo sosreports]# pwd
/home/repo/sosreports
[root at rhsqe-repo sosreports]# hostname
rhsqe-repo.lab.eng.blr.redhat.com

 username:qe-admin

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1260923
[Bug 1260923] Tracker for tiering in 3.1.2
https://bugzilla.redhat.com/show_bug.cgi?id=1265623
[Bug 1265623] Data Tiering:Promotions and demotions fail after quota hard
limits are hit for a tier volume
https://bugzilla.redhat.com/show_bug.cgi?id=1266841
[Bug 1266841] Data Tiering:Data Loss:File migrations(flushing of data) to
cold tier fails on detach tier with quota limits reached
-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.