[Gluster-devel] Throttling xlator on the bricks

Ravishankar N ravishankar at redhat.com
Mon Jan 25 05:36:26 UTC 2016


We are planning to introduce a throttling xlator on the server (brick)
process to regulate FOPS. The main motivation is to solve complaints about
AFR selfheal taking too much of CPU resources. (due to too many fops for 
self-heal, rchecksums for data self-heal etc.)

The throttling is achieved using the Token Bucket Filter algorithm 
is already used by bitrot's bitd signer (which is a client process) in
gluster to regulate the CPU intensive check-sum calculation. By putting the
logic on the brick side, multiple clients- selfheal, bitrot, rebalance or
even the mounts themselves can avail the benefits of throttling.

The TBF algorithm in a nutshell is as follows: There is a bucket which 
is filled
at a steady (configurable) rate with tokens. Each FOP will need a fixed 
of tokens to be processed. If the bucket has that many tokens, the FOP is
allowed and that many tokens are removed from the bucket. If not, the FOP is
queued until the bucket is filled.

The xlator will need to reside above io-threads and can have different 
one per client. There has to be a communication mechanism between the 
client and
the brick (IPC?) to tell what FOPS need to be regulated from it, and the 
no. of
tokens needed etc. These need to be re configurable via appropriate 
Each bucket will have a token filler thread which will fill the tokens 
in it.
The main thread will enqueue heals in a list in the bucket if there aren't
enough tokens. Once the token filler detects some FOPS can be serviced, 
it will
send a cond-broadcast to a dequeue thread which will process (stack 
wind) all
the FOPS that have the required no. of tokens from all buckets.

This is just a high level abstraction: requesting feedback on any aspect of
this feature. what kind of mechanism is best between the client/bricks for
tuning various parameters? What other requirements do you foresee?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160125/d56f0265/attachment.html>

More information about the Gluster-devel mailing list