[Bugs] [Bug 1302012] New: [Tiering]: Values of watermarks, min free disk etc will be miscalculated with quota set on root directory of gluster volume

bugzilla at redhat.com bugzilla at redhat.com
Tue Jan 26 14:20:36 UTC 2016 

https://bugzilla.redhat.com/show_bug.cgi?id=1302012

            Bug ID: 1302012
           Summary: [Tiering]: Values of watermarks, min free disk etc
                    will be miscalculated with quota set on root directory
                    of gluster volume
           Product: GlusterFS
           Version: 3.7.7
         Component: tiering
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: nbalacha at redhat.com
        QA Contact: bugs at gluster.org
                CC: annair at redhat.com, bugs at gluster.org,
                    dlambrig at redhat.com, kramdoss at redhat.com,
                    nchilaka at redhat.com, vmallika at redhat.com
        Depends On: 1300246, 1301473



+++ This bug was initially created as a clone of Bug #1301473 +++

+++ This bug was initially created as a clone of Bug #1300246 +++

Description of problem:
Hot tier's disk space is calculated using statfs and when quota is enabled, 
limit usage is set on / directory, always the aggregate value of hot tier's
capacity is returned. i.e., the quota value set for / directory is returned as
hot tier's value. This will break the watermark levels, min free disk value etc
causing promotion/demotion to break.

In the following scenario,

cold tier - 2 TB
hot tier - 200 GB
limit usage on / - 1.5 TB

when disk capacity of hot tier is calculated to set watermark limits, hot
tier's capacity would be set to 1.5 TB. All watermark levels, min free disk 
etc would then be based on 1.5 TB. i.e., low watermark would be 1.1 TB, high
watermark at 1.3 TB, min free disk would be 1.3 TB. All these values are way
above the total hot tier capacity which means promotions/demotions would be
broken. 

Version-Release number of selected component (if applicable):
glusterfs-3.7.5-16.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1. create a tiered gluster volume and have legacy files on cold tier
2. Enable quota and set limit usage of / directory to a high value
3. Set low and high watermark to 20 and 30%
4. Promote all legacy files, all files would be promoted crossing watermark
levels
Actual results: No demotion would be seen

The reverse case would be,
1. create a tiered gluster volume and have legacy files on cold tier
cold tier - 2 TB, hot tier - 200GB
2. Enable quota and set limit usage of / directory to a low value
quota on / set to 50 GB
3. Set low and high watermark to 20 and 30%
4. Try to promote files from cold tier. 
Actual results: No promotion would be seen

Expected results:
Hot tier's capacity should be calculated appropriately

Additional info:

--- Additional comment from Vijaikumar Mallikarjuna on 2016-01-20 06:05:06 EST
---

This is because 'features.quota-deem-statfs' is enabled by default and quota
enforcer returns the 'limit value' on the directory and not the actual disk
space.

Could you please try with below workaround and see if it works?

gluster volume set volname features.quota-deem-statfs off

--- Additional comment from krishnaram Karthick on 2016-01-21 01:08:33 EST ---

On a newly created volume with features.quota-deem-statfs set to 'off', quota
limit set on root directory and attaching hot tier sets correct values of hot
tier and watermarks work as expected. [see test-1]

However, Turning 'features.quota-deem-statfs' 'on' and 'off' on the fly doesn't
seem to fetch appropriate values of hot tier. Hot tier's value seems to be set
to the initial value while tier is attached and doesn't change dynamically .
This means we should turn off features.quota-deem-statfs even before hot tier
is attached. Attaching hot tier and then turning off
'features.quota-deem-statfs' would again have incorrect value of hot tier. [see
test-2]

Test-1:
1) create a volume of size 2 TB
2) set quota on root directory to 1.5 TB
3) turn off 'features.quota-deem-statfs'
4) Write files on this volume - 40 x 2 GB files, 50 x 1GB files, 1000 x 100 MB
files
5) Attach a hot tier of size 50 GB
6) set watermarks at 10% and 20%, i.e., 5GB and 10GB 
7) Heat 20 files of size 2GB
8) Only 6 files (12 GB) were promoted and demotions started immediately,
demoting 3 files. This leaves hot tier with 6 GB of data which is perfect.

With features.quota-deem-statfs set to 'off' we seen the expected behavior.
i.e., quota was set to 1.5 TB, hot tier's actual capacity was taken into
account and watermarks were set accordingly. Files were promoted and demoted as
expected.

Test 2:
Following test-1, test-2 was done.

1) Turn on 'features.quota-deem-statfs' [Now, hot tier's capacity should be
incorrect value of 1.5 TB and watermarks should be at a very high value and
only promotions would be seen]
2) Heat 10 x 2 GB files
3) No files were promoted even after continuous heating of files
4) stopped and started the volume under test
5) Heated 10 x 2 files, all 10 files were heated. i.e., new values of hot tier
is in effect only after volume was restarted.

Hence, the converse case would also be true. ie., having quota set to 1.5 TB
and with hot tier attached, turning off 'features.quota-deem-statfs' would
still fetch hot tier's value as 1.5 TB which is not expected.

I'll update the bug if there are any more findings.

--- Additional comment from Vijay Bellur on 2016-01-25 03:00:27 EST ---

REVIEW: http://review.gluster.org/13288 (cluster/tier: Ignore quota-deem-statfs
for watermark calculation) posted (#1) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-25 11:06:10 EST ---

COMMIT: http://review.gluster.org/13288 committed in master by Dan Lambright
(dlambrig at redhat.com) 
------
commit 9ce86c63e349eaafbd923b0b541d75e929ecbae7
Author: N Balachandran <nbalacha at redhat.com>
Date:   Mon Jan 25 13:27:42 2016 +0530

    cluster/tier: Ignore quota-deem-statfs for watermark calculation

    The tier process watermark calculations were incorrect when the
    quota-deem-statfs option was enabled. We now ignore this while
    calculating hot tier usage to determine watermark levels.

    Change-Id: I308bc24432e2fa5ad1d5703e80fc391433538bbb
    BUG: 1301473
    Signed-off-by: N Balachandran <nbalacha at redhat.com>
    Reviewed-on: http://review.gluster.org/13288
    Smoke: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: mohammed rafi  kc <rkavunga at redhat.com>
    Tested-by: mohammed rafi  kc <rkavunga at redhat.com>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: Dan Lambright <dlambrig at redhat.com>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    Tested-by: Dan Lambright <dlambrig at redhat.com>


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1300246
[Bug 1300246] [Tiering]: Values of watermarks, min free disk etc will be
miscalculated with quota set on root directory of gluster volume
https://bugzilla.redhat.com/show_bug.cgi?id=1301473
[Bug 1301473] [Tiering]: Values of watermarks, min free disk etc will be
miscalculated with quota set on root directory of gluster volume
-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.

More information about the Bugs mailing list