[Gluster-users] GlusterFS performance

[Jeff Darcy]

On 02/27/2013 08:34 AM, Nux! wrote:
> To me that sounds just about right, it's the kind of speed I get as 
> well. If you think of it, what happens in the background is that the 1GB 
> file is written not only from you to one of the servers, but also from 
> this server to the other servers, so it gets properly 
> replicated/distributed.

Actually not quite.  The data has to go directly from the client to each
of the servers, so that client's outbound bandwidth gets divided by N
replicas.  The way you describe would actually perform better, because
then you get to use that first server's outbound bandwidth as well, so
for N=2 you'd be running at full speed (though at a slight cost in
latency if the writes are synchronous).  At N=4 you'd be at 1/3 speed,
because one copy has to go from client to first server while three have
to go from that first server to the others.  Some systems even do
"chain" replication where each server sends only to one other, giving an
even more extreme tradeoff between bandwidth and latency.

In the scenario that started this, the expected I/O rate is wire speed
divided by number of replicas.  If the measured network performance is
100MB/s (800Mb/s) then with two replicas one would expect no better than
50MB/s and with four replicas no better than 25MB/s.  Any better results
are likely to be the results of caching, so that the real bandwidth
isn't actually being measured.  Worse results are often the result of
contention, such as very large buffers resulting in memory starvation
for something else.  I don't know of any reason why results would be
worse with distribution, and have never seen that effect myself.