[Gluster-users] Very poor GlusterFS Volume performance (glusterfs 8.2)

[WK]

On 11/9/2020 12:59 PM, Marc Jakobs wrote:
> I have a GlusterFS Volume on three Linux Servers (Ubuntu 20.04LTS) which
> are connected via 1GBit/sec NIC with each other over a dedicated switch.
>
> Every server has a NVMe disk which is used for the GlusterFS Volume
> called "data".

So I assume you have a simple replica 3 setup.

Are you using sharding?


> I have mounted the Volume like this
>
> mount -t glusterfs -o direct-io-mode=disable 127.0.0.1:/data /mnt/test/
>
> so it does not even go over the local NIC but instead over the loopback
> device.

You are Network constrained.

Your mount is direct, but if you have replica 3 the data still has to 
travel to the other two gluster bricks and that is occurring over a 
single 1 Gbit/s ethernet port which would have a maximum throughput of 
125 MB/s.

Since you have two streams going out that is roughly 62+ MB/s assuming 
full replica 3.

My understanding is that gluster doesn't acknowledge a write until its 
been written to at least one of the replicas ( I am sure others will 
jump in and correct me).  So 60 MB/s under those circumstances is what I 
would expect to see.

You can improve things by using an arbiter and supposedly the new Thin 
Arbiter is even faster (though I haven't tried it), but you lose a 
little safety The arbiter node only receives the metadata so it can 
referee on split-brain decisions, freeing up more BW for the actually 
data replica node.

A huge improvement would be if you were to bond two or more Gbit/s 
ports. Round-Robin teamd is really easy to setup, or use the traditional 
bonding in its various flavors. You probably have some spare NIC cards 
lying around so its usually a 'freebie'

Of course best case  would be to make the jump to 10Gb/s kit.

-wk