[Gluster-users] Slow write performance for small block sizes

[Andreas Mather]

Hi!

Thanks for the hint. In the meantime I found out that mounting the volume
on the client with direct-io-mode=enable helps (dd with 8K block size up to
55 MB/s). I will also try the 3.7.9 release...

Thanks,
Andreas






On Sat, Mar 26, 2016 at 9:51 PM, Serkan Çoban <cobanserkan at gmail.com> wrote:

> 3.7.8 had write performance problems. Can you try with 3.7.9?
>
> On Sat, Mar 26, 2016 at 7:52 PM, Andreas Mather <andreas at allaboutapps.at>
> wrote:
> > Hello!
> >
> > I experience very slow performance on gluster 3.7.8 on CentOS 7.2 when
> using
> > small block sizes with dd.
> >
> > I run 2 gluster servers with 2 bricks each (each located on separate
> SSDs)
> > and 2 gluster clients (which will run virtual machines). It's a
> completely
> > fresh setup, so no load on the systems. The systems are connected via 1
> GBit
> > network. iperf3 shows 940 MBit throughput, ping shows less than 1ms
> latency,
> > so network should be fine.
> >
> > The volume is a stripe 2 replica 2 volume. I've only set the volume
> options
> > for ping-timeout and owner-uid/gid (as required by qemu). The volume is
> > mounted via fuse client.
> >
> > He're my stats as seen from a client (370MB test file). 'storage1' is
> one of
> > the glusterfs servers which also has a fuse mount for these tests:
> >
> >
> > # read (always done echo 3 > /proc/sys/vm/drop_caches before running
> tests)
> >
> > $ dd if=/glustermount/testfile of=/localdisk/testfile bs=4K
> > 69.7 MB/s
> >
> > $ dd if=/glustermount/testfile of=/localdisk/testfile bs=8K
> > 78.8 MB/s
> >
> > $ scp storage1:/glustermount/testfile /localdisk/testfile
> > 83.3 MB/s
> >
> > $ ssh storage1 'dd if=/glustermount/testfile bs=8K' | dd
> > of=/localdisk/testfile bs=8K
> > 93.3 MB/s
> >
> >
> > # write
> >
> > $ dd if=/localdisk/testfile of=/glustermount/testfile bs=4K
> > 3.8 MB/s
> >
> > $ dd if=/localdisk/testfile of=/glustermount/testfile bs=8K
> > 6.9 MB/s
> >
> > $ dd if=/localdisk/testfile of=/glustermount/testfile bs=8M
> > 58.8 MB/s
> >
> > $ dd if=/localdisk/testfile of=/glustermount/testfile bs=64M
> > 58.8 MB/s
> >
> > $ scp /localdisk/testfile storage1:/localdisk/testfile
> > 94 MB/s
> >
> > $ scp /localdisk/testfile storage1:/glustermount/testfile
> > 980 KB/s
> >
> > # immediately ran after the previous one. what happened?!?!
> > $ scp /localdisk/testfile storage1:/glustermount/testfile
> > 75 MB/s
> >
> > $ dd if=/localdisk/testfile bs=8K | ssh storage1 'dd
> of=/localdisk/testfile
> > bs=8K'
> > 96.6 MB/s
> >
> > $ dd if=/localdisk/testfile bs=8K | ssh storage1 'dd
> > of=/glustermount/testfile bs=8K'
> > 225 KB/s
> >
> > Is this really as good as it gets with gluster? I've tried the following
> > setups in order to improve write speed for small block sizes:
> >
> > - tuning stripe-size. smaller (16KB) and larger (4MB) values
> > - 'replica 2' only setup (i.e. no striping)
> > - write-behind on/off
> > - larger write-behind-window-size
> > - larger cache-size
> > - more io-threads
> >
> > None of these helped or even changed the numbers much. I've also tried a
> few
> > sysctl tweaks, but still not much change.
> >
> > So I'd like to know:
> >
> > - is this normal speed/behaviour?
> > - what are the reasons for this?
> > - can anything be done to improve performance?
> > - is running VMs on top of gluster a supported/intended use case at all?
> >
> > Thanks a lot,
> > Andreas
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160327/01cd7ece/attachment.html>