[Gluster-devel] Re: [Gluster-users] I/O fair share to avoid I/O bottlenecks on small clsuters

[Jeff Darcy]

On 01/31/2010 09:06 AM, Ran wrote:
> You guys are talking about network IO im taking about the gluster server disk IO
> the idea to shape the trafic does make sence seens the virt machines
> server do use network to get to the disks(gluster)
> but what about if there are say 5 KVM servers(with VPS's) all on
> gluster what do you do then ? its not quite fair share seens every
> server has its own fair share and doesnt see the others .
> 
> Also there are other applications that uses gluster like mail etc..
> and i see that gluster IO is very high very often cousing the all
> storage not to work .
> Its very disturbing .


You bring up a good set of points.  Some of these problems can be
addressed at the hypervisor (i.e. GlusterFS client) level, some can be
addressed by GlusterFS itself, and some can be addressed only at the
level of the local-filesystem or block-device level on the GlusterFS
servers.  Unfortunately, I/O traffic shaping is still in its infancy
compared to what's available for networking - or perhaps even "infancy"
is too generous.  As far as the I/O stack is concerned, all of the
traffic is coming from the glusterfsd process(es) without
differentiation, so even if the functionality to apportion I/O amongst
tasks existed it wouldn't be usable without more information.  Maybe
some day...

What you can do now at the GlusterFS level, though, is make sure that
traffic is distributed across many servers and possibly across many
volumes per server to take advantage of multiple physical disks and/or
interconnects for one server.  That way, a single VM will only use a
small subset of the servers/volumes and will not starve other clients
that are using different servers/volumes (except for network bottlenecks
which are a separate issue).  That's what the "distribute" translator is
for, and it can be combined with replicate or stripe to provide those
functions as well.  Perhaps it would be useful to create and publish
some up-to-date recipes for these sorts of combinations.