<div dir="ltr"><div><div><div><div>Hi Xavier,<br><br></div>Thanks a lot for your message, it really improved the performance.<br></div>I updated both the recordsize to 256K on all nodes, and thread-count to 8 and now the read performance got even better than write.<br><br></div>Regards,<br></div>Yann<br></div><div class="gmail_extra"><br><div class="gmail_quote">On 19 January 2017 at 13:09, Xavier Hernandez <span dir="ltr"><<a href="mailto:xhernandez@datalab.es" target="_blank">xhernandez@datalab.es</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Yann,<span class=""><br>
<br>
On 19/01/17 09:35, Yann MAUPU wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Hi everyone,<br>
<br>
I am currently working on a project for which I am using:<br>
<br></span>
* 3 storage nodes connected with Omnipath<br>
* 6 sata 750GB HDD per node (total of 18 disks)<span class=""><br>
<br>
I created a ZFS raidZ1 on each node (5 disks + 1) and used GlusterFS in<br>
raid5 mode between the 3 nodes.<br>
<br>
Unfortunately, with (very) big files, I experience a quite low<br>
read-performance, compared to write-perf (write=700MB/s while read=320<br>
MB/s).<br>
<br>
Do you know tuning/optimization parameters which could help me get<br></span>
*better read-performance* ?<br>
</blockquote>
<br>
You can try to set client.event-threads and server.event-threads to higher values. For example 4 or 8 (default is 2).<br>
<br>
gluster volume set <volname> client.event-threads 8<br>
gluster volume set <volname> server.event-threads 8<br>
<br>
Xavi<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
<br>
Here's more information on the configuration:<br>
<br></span>
_ZFS raidz1 on each node:_<br>
<br>
/# zpool list////NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT////tank 4,06T 10,5M 4,06T - 0% 0% 1.00x ONLINE -///<br>
<br>
/# zpool status//// pool: tank//// state: ONLINE//// scan: none requested////config://// NAME STATE READ WRITE CKSUM//// tank ONLINE 0 0 0//// raidz1-0 ONLINE 0 0 0//// sda ONLINE 0 0 0//// sdb ONLINE 0 0 0//// sdc ONLINE 0 0 0//// sdd ONLINE 0 0 0//// sde ONLINE 0 0 0//// sdf ONLINE 0 0 0////errors: No known data errors///<span class=""><br>
<br>
The command used to create the volume:<br>
<br>
# zpool create -f tank raidz sda sdb sdc sdd sde sdf<br>
<br></span>
When running IOR on each node, I get about /write perf = 460 MB/s/ and<br>
/read perf = 430 MB/s/ (writing 1024 GiB with zfer_size=16MiB).<br>
<br>
<br>
_GlusterFS raid5 through TCP (IPoIB) between the 3 nodes:_<span class=""><br>
<br>
# gluster volume create raid55 disperse 3 redundancy 1 sm01.opa:/tank/ec_point sm02.opa:/tank/ec_point sm03.opa:/tank/ec_point<br>
<br>
<br>
There is a big difference between the read-performance on each ZFS node<br>
and with Gluster.<br>
<br>
Thanks in advance :)<br>
<br>
Yann<br>
<br>
<br>
<br>
<br></span>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
<br>
</blockquote>
<br>
</blockquote></div><br></div>