[Gluster-users] Hot Tier exceeding watermark-hi

[David Brown]

Hey Everyone,

I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD cold
tier.
I recently ran into some problems when the hot tier became full with df-h
showing 100%.

I did not have a watermark-hi set, but it is my understanding that 90% is
the default. In an attempt to get the cluster to demote some files, I set
cluster.watermark-hi: 80 but it is still not demoting.


[root at Glus1 ~]# gluster volume info

Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
Options Reconfigured:
cluster.tier-promote-frequency: 120
cluster.tier-demote-frequency: 1800
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root at Glus1 ~]# gluster volume tier FFPrimary status
Node                 Promoted files       Demoted files        Status
         run time in h:m:s
---------            ---------            ---------            ---------
        ---------
localhost            49                   0                    in progress
        5151:30:45
Glus2 0                    0                    in progress
5151:30:45
Glus3 0                    2075                 in progress
5151:30:47
Tiering Migration Functionality: FFPrimary: success
[root at Glus1 ~]#

What can cause GlusterFS to stop demoting files and allow it to completely
fill the Hot Tier?

Thank you!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180929/e87edbcc/attachment.html>