<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi everyone,<br>
<br>
I am currently working on a project for which I am using:<br>
<ul>
<li>3 storage nodes connected with Omnipath</li>
<li>6 sata 750GB HDD per node (total of 18 disks)</li>
</ul>
<p>I created a ZFS raidZ1 on each node (5 disks + 1) and used
GlusterFS in raid5 mode between the 3 nodes.</p>
<p>Unfortunately, with (very) big files, I experience a quite low
read-performance, compared to write-perf (write=700MB/s while
read=320 MB/s).</p>
<p>Do you know tuning/optimization parameters which could help me
get <b>better read-performance</b> ?</p>
<p><br>
</p>
<p>Here's more information on the configuration:</p>
<p><u>ZFS raidz1 on each node:</u><br>
</p>
<pre><i># zpool list</i><i>
</i><i>NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT</i><i>
</i><i>tank 4,06T 10,5M 4,06T - 0% 0% 1.00x ONLINE -</i><i>
</i></pre>
<pre><i># zpool status</i><i>
</i><i> pool: tank</i><i>
</i><i> state: ONLINE</i><i>
</i><i> scan: none requested</i><i>
</i><i>config:</i><i>
</i><i> NAME STATE READ WRITE CKSUM</i><i>
</i><i> tank ONLINE 0 0 0</i><i>
</i><i> raidz1-0 ONLINE 0 0 0</i><i>
</i><i> sda ONLINE 0 0 0</i><i>
</i><i> sdb ONLINE 0 0 0</i><i>
</i><i> sdc ONLINE 0 0 0</i><i>
</i><i> sdd ONLINE 0 0 0</i><i>
</i><i> sde ONLINE 0 0 0</i><i>
</i><i> sdf ONLINE 0 0 0</i><i>
</i><i>errors: No known data errors</i><i>
</i></pre>
<p>The command used to create the volume:<br>
</p>
<pre># zpool create -f tank raidz sda sdb sdc sdd sde sdf
</pre>
<p>When running IOR on each node, I get about <i>write perf = 460
MB/s</i> and <i>read perf = 430 MB/s</i> (writing 1024 GiB with
zfer_size=16MiB).</p>
<p><br>
</p>
<p><u>GlusterFS raid5 through TCP (IPoIB) between the 3 nodes:</u></p>
<pre># gluster volume create raid55 disperse 3 redundancy 1 sm01.opa:/tank/ec_point sm02.opa:/tank/ec_point sm03.opa:/tank/ec_point</pre>
<br>
There is a big difference between the read-performance on each ZFS
node and with Gluster.<br>
<p>Thanks in advance :)</p>
<p>Yann<br>
<br>
</p>
<p><br>
</p>
</body>
</html>