[Gluster-users] Extremely low performance - am I doing somethingwrong?

[wkmail]

On 7/4/2019 2:28 AM, Vladimir Melnik wrote:
> So, the disk is OK and the network is OK, I'm 100% sure.
>
> Seems to be a GlusterFS-related issue. Either something needs to be
> tweaked or it's a normal performance for a replica-3 cluster.

There is more to it than Gluster on that particular test.

I have some addititional datapoints, since those numbers seemed low 
given the long time I have played with Gluster (first install was 3.3)

So I ran that exact test on some locally mounted hard drive sets (mdadm 
RAID1- spinning metal) on Cent7 (stock)  and U18(stock) and got the 
following:

No Gluster involved.

# for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 
oflag=sync; rm -f ./test.tmp; } done
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 1.0144 s, 10.3 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.791071 s, 13.3 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.832186 s, 12.6 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.80427 s, 13.0 MB/s
10+0 records in

That was reproducible over several machines with different CPUs that we 
have in production.

Performance is about 20%  better when 7200 rpm drives were involved or 
when no RAID was involved but never above 18 MB/s

Performance is also MUCH better when I use oflag=direct (roughly 2x)

However, on a U18 VM Host testbed machine that has a seperate SSD swap 
disk I get the following, even though I am writing the test.tmp file to 
the metal.

# for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 
oflag=sync; rm -f ./test.tmp; } done

10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0949153 s, 110 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0605883 s, 173 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0582863 s, 180 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0604369 s, 173 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.0598746 s, 175 MB/s

So something else is going on with that particular test. Clearly, 
buffers, elevators, cache etc count despite the oflag setting.

For the record on the Gluster Fuse Mount (2x+1arb Volume)  on that VM 
Host I do get reduced performance

part of that is due to the gluster network being 2x1G using teaming on 
that testbed, so there is a network bottleneck.

# for i in {1..5}; do { dd if=/dev/zero of=./test.tmp bs=1M count=10 
oflag=sync; rm -f ./test.tmp; } done
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.693351 s, 15.1 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.349881 s, 30.0 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.339699 s, 30.9 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.34202 s, 30.7 MB/s
10+0 records in
10+0 records out
10485760 bytes (10 MB, 10 MiB) copied, 0.337904 s, 31.0 MB/s

So the gluster fuse mount negates the advantage of that SSD swap disk 
along with the obvious network bottleneck.

But clearly we have to all agree on same valid test.

-wk