[Gluster-devel] tiering: emergency demotions

[Milind Changire]

Patch for review: http://review.gluster.org/15158

Milind

On 08/12/2016 07:27 PM, Milind Changire wrote:
> On 08/10/2016 12:06 PM, Milind Changire wrote:
>> Emergency demotions will be required whenever writes breach the
>> hi-watermark. Emergency demotions are required to avoid ENOSPC in case
>> of continuous writes that originate on the hot tier.
>>
>> There are two concerns in this area:
>>
>> 1. enforcing max-cycle-time during emergency demotions
>>    max-cycle-time is the time the tiering daemon spends in promotions or
>>    demotions
>>    I tend to think that the tiering daemon skip this check for the
>>    emergency situation and continue demotions until the watermark drops
>>    below the hi-watermark
>
> Update:
> To keep matters simple and manageable, it has been decided to *enforce*
> max-cycle-time to yield the worker threads to attend to impending tier
> management tasks if the need arises.
>
>>
>> 2. file demotion policy
>>    I tend to think that evicting the largest file with the most recent
>>    *write* should be chosen for eviction when write-freq-threshold is
>>    NON-ZERO.
>>    Choosing a least written file is just going to delay file migration
>>    of an active file which might consume hot tier disk space resulting
>>    in a ENOSPC, in the worst case.
>>    In cases where write-freq-threshold are ZERO, the most recently
>>    *written* file can be chosen for eviction.
>>    In the case of choosing the largest file within the
>>    write-freq-threshold, a stat() on the files would be required to
>>    calculate the number of files that need to be demoted to take the
>>    watermark below the hi-watermark. Finding the number of most recently
>>    written files to demote could also help make demotions in parallel
>>    rather than in the sequential manner currently in place.
>
> Update:
> The idea of choosing the files wrt file size has been dropped.
> Iteratively, the most recently written file will be chosen for eviction
> from the hot tier in case of a hi-watermark breach and until the
> watermark drops below hi-watermark.
> The idea of parallelizing multiple promotions/demotions has been
> deferred.
>
> -----
>
> Sustained writes creating larges files in the hot tier which
> cumulatively breach the hi-watermark does NOT seem to be a good
> workload for making use of tiering. The assumption is that, to make the
> most of of the hot tier, the hi-watermark would be closer to 100.
> In this case a sustained large file copy might easily breach the
> hi-watermark and may even consume the entire hot tier space, resulting
> in a ENOSPC.
>
> eg. an example of a sustained write
>
> # cp file1 /mnt/glustervol/dir
>
> Workloads that would seem to make the most of tiering are:
> 1. Many smaller files, which are created in small bursts of write
>    activity and then closed
> 2. Few large files where updates are in-place and the file size
>    does not grow beyond the hi-watermark eg. database, with frequent
>    in-line compaction/de-fragmentation policy enabled
> 3. Frequent reads of few large files, mostly static in size, which
>    cumulatively don't breach the hi-watermark. Frequently reading
>    a large number of smaller, mostly static, files would be good
>    tiering workload candidates as well.
>
>
>>
>> Comments are requested.
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel