<div dir="ltr"><div><div>>Can you please turn OFF client-io-threads as we have seen degradation of
performance with io-threads ON on sequential read/writes, random
read/writes.<br></div>May I ask which version is this degradation happened? I tested 3.10 vs 3.12 performance a while ago and saw 2-3x performance lost with 3.12. Is it because of client-io-threads?<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Oct 30, 2017 at 1:44 PM, Karan Sandha <span dir="ltr"><<a href="mailto:ksandha@redhat.com" target="_blank">ksandha@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Brandon,</div><div><br></div>Can you please turn OFF client-io-threads as we have seen degradation of performance with io-threads ON on sequential read/writes, random read/writes. Server event threads is 1 and client event threads are 2 by default. <div><br></div><div>Thanks & Regards</div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Fri, Oct 27, 2017 at 12:17 PM, Brandon Bates <span dir="ltr"><<a href="mailto:brandon@brandonbates.com" target="_blank">brandon@brandonbates.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><u></u>
<div>
<div>
<div><font face="Arial"><font size="2"><span class="m_3516616289438030001m_4291135282120425186468513406-27102017">Hi gluster
users,</span></font></font></div>
<div><font face="Arial"><font size="2"><span class="m_3516616289438030001m_4291135282120425186468513406-27102017"><span class="m_3516616289438030001m_4291135282120425186468513406-27102017">I've spent several months trying to get any kind
of high performance out of gluster. <span class="m_3516616289438030001m_4291135282120425186468513406-27102017">The
current XFS/samba array is used for video editing and 300-400MB/s for at least 4
clients is minimum (currently a single windows client gets at least 700/700 for
a single client over samba, peaking to 950 at times using blackmagic speed
test). Gluster has been getting me as low as 200MB/s when the server
can do well over 1000MB/s. </span></span></span>I have really been
counting on <span class="m_3516616289438030001m_4291135282120425186468513406-27102017">/ touting
</span>Gluster <span class="m_3516616289438030001m_4291135282120425186468513406-27102017">as </span>being the way of
the future for us<span class="m_3516616289438030001m_4291135282120425186468513406-27102017">. However</span> I
can't justify cutting our performance to a mere 13% of non-gluster<span class="m_3516616289438030001m_4291135282120425186468513406-27102017"> speeds</span>. I've started to reach a give up
point and really need some help/hope otherwise I'll just have to migrate the
data from server 1 to server 2 just like I've been doing for the last
decade.<span class="m_3516616289438030001m_4291135282120425186468513406-27102017"> </span><span class="m_3516616289438030001m_4291135282120425186468513406-27102017">:(</span></font></font></div>
<div><font face="Arial"><font size="2"><span class="m_3516616289438030001m_4291135282120425186468513406-27102017"></span></font></font> </div>
<div><font face="Arial"><font size="2"><span class="m_3516616289438030001m_4291135282120425186468513406-27102017"><span class="m_3516616289438030001m_4291135282120425186468513406-27102017">If anyone can please help me understand where I might
be going wrong it would be absolutely
wonderful!</span></span></font></font></div>
<div><font face="Arial"><font size="2"><span class="m_3516616289438030001m_4291135282120425186468513406-27102017"></span></font></font> </div></div>
<div><font face="Arial" size="2">Server 1:<br>Single E5-1620 v2<br>Ubuntu
14.04<br>glusterfs 3.10.5<br>16GB Ram<br>24 drive array on LSI raid<br>Sustained
>1.5GB/s to XFS (77TB)</font></div>
<div> </div>
<div><font face="Arial" size="2">Server 2:<br>Single E5-2620 v3<br>Ubuntu
16.04<br>glusterfs 3.10.5<br>32GB Ram<br>36 drive array on LSI raid<br>Sustained
>2.5GB/s to XFS (164TB)</font></div>
<div> </div>
<div><font face="Arial" size="2">Speed tests are done with local with single thread
(dd) or 4 threads (iozone) using my standard 64k io size to 20G or 5G files (20G
for local drives, 5G for gluster) files.</font></div>
<div><font face="Arial" size="2"></font> </div>
<div><font face="Arial" size="2">Servers have Intel X520-DA2 dual port 10Gbit NICS
bonded together with 802.11ad LAG to a Quanta LB6-M switch. Iperf
throughput numbers are single stream >9000Mbit/s</font></div>
<div> </div>
<div><font face="Arial" size="2">Here is my current gluster
performance:</font></div>
<div> </div>
<div><font face="Arial" size="2">Single brick on server 1 (server 2 was
similar):<br>Fuse mount:<br>1000MB/s write<br>325MB/s read</font></div>
<div> </div>
<div><font face="Arial" size="2">Distributed only servers 1+2:<br>Fuse mount on
server 1:<br>900MB/s write iozone 4 streams<br>320MB/s read iozone 4
streams<br>single stream read 91MB/s @64K, 141MB/s @1M<br>simultaneous iozone 4
stream 5G files<br>Server 1: 1200MB/s write, 200MB/s read<br>Server 2: 950MB/s
write, 310MB/s read</font></div>
<div><font face="Arial" size="2"></font> </div>
<div><font face="Arial" size="2">I did some earlier single brick tests with samba
VFS and 3 workstations and got up to 750MB/s write and 800MB/s read aggregate
but that's still not good.</font></div>
<div> </div>
<div><font face="Arial" size="2">These are the only volume settings tweaks I have
made <span class="m_3516616289438030001m_4291135282120425186468513406-27102017">(</span>after much single box
testing<span class="m_3516616289438030001m_4291135282120425186468513406-27102017"> </span>to find what actually made a
difference<span class="m_3516616289438030001m_4291135282120425186468513406-27102017">)</span>:<br>performance.cache-size
1GB (Default 23MB)<br>performance.client-io-threads
on<br>performance.io-thread-count
64<br>performance.read-ahead-page-co<wbr>unt
16<br>performance.stat-prefetch on<br>server.event-threads 8
(default?)<br>client.event-threads 8</font></div>
<div> </div>
<div><font face="Arial" size="2"><span class="m_3516616289438030001m_4291135282120425186468513406-27102017">Any help given is
appreciated!</span></font></div></div>
<br></div></div><span class="">______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></span></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="m_3516616289438030001gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>KARAN</span> <span>SANDHA</span></p><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>QUALITY ENGINEER</span><span style="color:rgb(170,170,170);margin:0px"></span></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>Bangalore</span></a></p><p style="font-family:overpass,sans-serif;margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)"><span style="margin:0px;padding:0px"><a href="mailto:ksandha@redhat.com" style="color:rgb(0,136,206);margin:0px" target="_blank">ksandha@redhat.com</a> </span> <span>M: <a href="tel:9888009555" style="color:rgb(0,136,206);font-size:11px;margin:0px" target="_blank">9888009555</a> </span> <span>IM: <span>Karan on @irc</span></span></p><table style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium" border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/files/brand/email/sig-redhat.png" height="auto" width="90"></a></td><td style="font-size:10px"><div><a href="https://redhat.com/trusted" style="color:rgb(204,0,0);font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></div></td></tr></tbody></table><div style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px"><div style="color:rgb(153,153,153)"><a href="https://twitter.com/redhatnews" title="twitter" style="background:url("https://www.redhat.com/files/brand/email/sm-twitter.png") 0px 50%/16px no-repeat transparent;height:20px;color:rgb(119,119,119);display:inline-block;line-height:20px;padding-left:16px" target="_blank">@redhatnews</a> <a href="https://www.linkedin.com/company/red-hat" title="LinkedIn" style="background:url("https://www.redhat.com/files/brand/email/sm-linkedin.png") 0px 50%/16px no-repeat transparent;height:20px;color:rgb(119,119,119);display:inline-block;line-height:20px;padding-left:16px" target="_blank">Red Hat</a> <a href="https://www.facebook.com/RedHatInc" title="Facebook" style="background:url("https://www.redhat.com/files/brand/email/sm-facebook.png") 0px 50%/16px no-repeat transparent;height:20px;color:rgb(119,119,119);display:inline-block;line-height:20px;padding-left:16px" target="_blank">Red Hat</a></div></div></div></div></div></div></div></div>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>