[Gluster-users] Very poor GlusterFS Volume performance (glusterfs 8.2)
wkmail at bneit.com
Tue Nov 10 00:37:55 UTC 2020
On 11/9/2020 12:59 PM, Marc Jakobs wrote:
> I have a GlusterFS Volume on three Linux Servers (Ubuntu 20.04LTS) which
> are connected via 1GBit/sec NIC with each other over a dedicated switch.
> Every server has a NVMe disk which is used for the GlusterFS Volume
> called "data".
So I assume you have a simple replica 3 setup.
Are you using sharding?
> I have mounted the Volume like this
> mount -t glusterfs -o direct-io-mode=disable 127.0.0.1:/data /mnt/test/
> so it does not even go over the local NIC but instead over the loopback
You are Network constrained.
Your mount is direct, but if you have replica 3 the data still has to
travel to the other two gluster bricks and that is occurring over a
single 1 Gbit/s ethernet port which would have a maximum throughput of
Since you have two streams going out that is roughly 62+ MB/s assuming
full replica 3.
My understanding is that gluster doesn't acknowledge a write until its
been written to at least one of the replicas ( I am sure others will
jump in and correct me). So 60 MB/s under those circumstances is what I
would expect to see.
You can improve things by using an arbiter and supposedly the new Thin
Arbiter is even faster (though I haven't tried it), but you lose a
little safety The arbiter node only receives the metadata so it can
referee on split-brain decisions, freeing up more BW for the actually
data replica node.
A huge improvement would be if you were to bond two or more Gbit/s
ports. Round-Robin teamd is really easy to setup, or use the traditional
bonding in its various flavors. You probably have some spare NIC cards
lying around so its usually a 'freebie'
Of course best case would be to make the jump to 10Gb/s kit.
More information about the Gluster-users