<div dir="ltr"><div>Hi!</div><div><br></div><div>Thanks for the hint. In the meantime I found out that mounting the volume on the client with direct-io-mode=enable helps (dd with 8K block size up to 55 MB/s). I will also try the 3.7.9 release...</div><div><br></div><div>Thanks,</div><div>Andreas</div><div><br></div><div><br></div><div class="gmail_extra"><div><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-family:arial;font-size:small"><br></div><div style="font-family:arial;font-size:small"><br></div><p><br></p></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Sat, Mar 26, 2016 at 9:51 PM, Serkan Çoban <span dir="ltr"><<a href="mailto:cobanserkan@gmail.com" target="_blank">cobanserkan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">3.7.8 had write performance problems. Can you try with 3.7.9?<br>
<div><div><br>
On Sat, Mar 26, 2016 at 7:52 PM, Andreas Mather <<a href="mailto:andreas@allaboutapps.at" target="_blank">andreas@allaboutapps.at</a>> wrote:<br>
> Hello!<br>
><br>
> I experience very slow performance on gluster 3.7.8 on CentOS 7.2 when using<br>
> small block sizes with dd.<br>
><br>
> I run 2 gluster servers with 2 bricks each (each located on separate SSDs)<br>
> and 2 gluster clients (which will run virtual machines). It's a completely<br>
> fresh setup, so no load on the systems. The systems are connected via 1 GBit<br>
> network. iperf3 shows 940 MBit throughput, ping shows less than 1ms latency,<br>
> so network should be fine.<br>
><br>
> The volume is a stripe 2 replica 2 volume. I've only set the volume options<br>
> for ping-timeout and owner-uid/gid (as required by qemu). The volume is<br>
> mounted via fuse client.<br>
><br>
> He're my stats as seen from a client (370MB test file). 'storage1' is one of<br>
> the glusterfs servers which also has a fuse mount for these tests:<br>
><br>
><br>
> # read (always done echo 3 > /proc/sys/vm/drop_caches before running tests)<br>
><br>
> $ dd if=/glustermount/testfile of=/localdisk/testfile bs=4K<br>
> 69.7 MB/s<br>
><br>
> $ dd if=/glustermount/testfile of=/localdisk/testfile bs=8K<br>
> 78.8 MB/s<br>
><br>
> $ scp storage1:/glustermount/testfile /localdisk/testfile<br>
> 83.3 MB/s<br>
><br>
> $ ssh storage1 'dd if=/glustermount/testfile bs=8K' | dd<br>
> of=/localdisk/testfile bs=8K<br>
> 93.3 MB/s<br>
><br>
><br>
> # write<br>
><br>
> $ dd if=/localdisk/testfile of=/glustermount/testfile bs=4K<br>
> 3.8 MB/s<br>
><br>
> $ dd if=/localdisk/testfile of=/glustermount/testfile bs=8K<br>
> 6.9 MB/s<br>
><br>
> $ dd if=/localdisk/testfile of=/glustermount/testfile bs=8M<br>
> 58.8 MB/s<br>
><br>
> $ dd if=/localdisk/testfile of=/glustermount/testfile bs=64M<br>
> 58.8 MB/s<br>
><br>
> $ scp /localdisk/testfile storage1:/localdisk/testfile<br>
> 94 MB/s<br>
><br>
> $ scp /localdisk/testfile storage1:/glustermount/testfile<br>
> 980 KB/s<br>
><br>
> # immediately ran after the previous one. what happened?!?!<br>
> $ scp /localdisk/testfile storage1:/glustermount/testfile<br>
> 75 MB/s<br>
><br>
> $ dd if=/localdisk/testfile bs=8K | ssh storage1 'dd of=/localdisk/testfile<br>
> bs=8K'<br>
> 96.6 MB/s<br>
><br>
> $ dd if=/localdisk/testfile bs=8K | ssh storage1 'dd<br>
> of=/glustermount/testfile bs=8K'<br>
> 225 KB/s<br>
><br>
> Is this really as good as it gets with gluster? I've tried the following<br>
> setups in order to improve write speed for small block sizes:<br>
><br>
> - tuning stripe-size. smaller (16KB) and larger (4MB) values<br>
> - 'replica 2' only setup (i.e. no striping)<br>
> - write-behind on/off<br>
> - larger write-behind-window-size<br>
> - larger cache-size<br>
> - more io-threads<br>
><br>
> None of these helped or even changed the numbers much. I've also tried a few<br>
> sysctl tweaks, but still not much change.<br>
><br>
> So I'd like to know:<br>
><br>
> - is this normal speed/behaviour?<br>
> - what are the reasons for this?<br>
> - can anything be done to improve performance?<br>
> - is running VMs on top of gluster a supported/intended use case at all?<br>
><br>
> Thanks a lot,<br>
> Andreas<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://www.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div></div>