[Gluster-users] Hot Tier exceeding watermark-hi

[Hari Gowtham]

Hi,

I can see that the demotion is happening from the status provided by you.
Do verify it.
I would recommend you to change the cluster.tier-demote-frequency to 120
and cluster.tier-promote-frequency to 1800 to increase the demotions until the
hot tier is emptied to a certain extent. Later you can use the values
existing now.
On Sat, Sep 29, 2018 at 5:39 PM David Brown <dbccemtp at gmail.com> wrote:
>
> Hey Everyone,
>
> I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD cold tier.
> I recently ran into some problems when the hot tier became full with df-h showing 100%.
>
> I did not have a watermark-hi set, but it is my understanding that 90% is the default. In an attempt to get the cluster to demote some files, I set cluster.watermark-hi: 80 but it is still not demoting.
>
>
> [root at Glus1 ~]# gluster volume info
>
> Volume Name: FFPrimary
> Type: Tier
> Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 9
> Transport-type: tcp
> Hot Tier :
> Hot Tier Type : Replicate
> Number of Bricks: 1 x 3 = 3
> Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
> Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
> Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
> Cold Tier:
> Cold Tier Type : Distributed-Replicate
> Number of Bricks: 2 x 3 = 6
> Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
> Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
> Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
> Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
> Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
> Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
> Options Reconfigured:
> cluster.tier-promote-frequency: 120
> cluster.tier-demote-frequency: 1800
> cluster.watermark-low: 60
> cluster.watermark-hi: 80
> performance.flush-behind: on
> performance.cache-max-file-size: 128MB
> performance.cache-size: 25GB
> diagnostics.count-fop-hits: off
> diagnostics.latency-measurement: off
> cluster.tier-mode: cache
> features.ctr-enabled: on
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
> [root at Glus1 ~]# gluster volume tier FFPrimary status
> Node                 Promoted files       Demoted files        Status               run time in h:m:s
> ---------            ---------            ---------            ---------            ---------
> localhost            49                   0                    in progress          5151:30:45
> Glus2 0                    0                    in progress          5151:30:45
> Glus3 0                    2075                 in progress          5151:30:47
> Tiering Migration Functionality: FFPrimary: success
> [root at Glus1 ~]#
>
> What can cause GlusterFS to stop demoting files and allow it to completely fill the Hot Tier?
>
> Thank you!
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users



-- 
Regards,
Hari Gowtham.