<div dir="ltr"><div>Hi Jorick,</div><div><br></div><div>Thank you for that, we will try direct-io-mode=disable.</div><div><br></div><div>Would anyone have any other advice on the cause of 114MBps of network traffic with GFS?</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 20 Dec 2019 at 03:00, Jorick Astrego <<a href="mailto:jorick@netbulae.eu">jorick@netbulae.eu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p>Hi David,</p>
<p>Did you try setting "direct-io-mode=disable" on the client
mounts? As it is mostly static content it would help to use the
kernel caching and read-ahead mechanisms.</p>
<p>I think the default is enabled.</p>
<p>Regards,</p>
<p>Jorick Astrego<br>
</p>
<div>On 12/19/19 1:28 AM, David Cunningham
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Hi Raghavendra and Strahil,</div>
<div><br>
</div>
<div>We are using GFS version 5.6-1.el7 from the CentOS
repository. Unfortunately we can't modify the application
and it expects to read and write from a normal filesystem.</div>
<div><br>
</div>
<div>There's around 25GB of data being written during a
business day, so over 10 hours that's around 0.7 MBps,
which has me mystified as to how it can generate 114MBps
of network traffic. Granted we have read traffic as well,
but still. The chart shows much more inbound traffic to
the GFS server than outbound, suggesting the problem is
with data writes.<br>
</div>
<div><br>
</div>
<div>Is it possible with GFS to not check with the other
nodes when reading? Our data is mostly static and we don't
require 100% guarantee that the data is up-to-date when
reading.</div>
<div><br>
</div>
<div>Thanks for any assistance.</div>
<div><br>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, 18 Dec 2019 at 16:39,
Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>What version of Glusterfs are you using? Though, not
sure what's the root cause of your problem, just wanted to
point out a bug with read-ahead which would cause
read-amplification over network [1][2], which should be
fixed in recent versions.<br>
</div>
<div><br>
</div>
<div>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1214489" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1214489</a></div>
<div>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1393419" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1393419</a></div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Dec 18, 2019 at
2:50 AM David Cunningham <<a href="mailto:dcunningham@voisonics.com" target="_blank">dcunningham@voisonics.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Hello,</div>
<div><br>
</div>
<div>We switched a production system to using GFS
instead of NFS at the weekend, however it didn't go
well on Monday when full load hit. The application
started crashing regularly and we had to revert to
NFS. It seems that the problem was high network
traffic used by GFS.<br>
</div>
<div><br>
</div>
<div>We've two GFS nodes plus one arbiter node, each
about 1.3ms latency from each other. Attached is a
chart of network traffic on one of the GFS nodes. We
see that it saturated the 1Gbps link before we
reverted to NFS at 15:10.</div>
<div><br>
</div>
<div>The question is, why does GFS use so much network
traffic and is there anything we can do about it? NFS
traffic doesn't exceed 4MBps, so 120MBps for GFS seems
awfully high.</div>
<div><br>
</div>
<div>It would also be good to have faster read
performance from GFS, but that's another issue.<br>
</div>
<div><br>
</div>
<div>Thanks in advance for any assistance.<br>
</div>
<div><br>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>David Cunningham, Voisonics
Limited<br>
<a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: +1 213 221 1092<br>
New Zealand: +64 (0)28 2558 3782</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>David Cunningham, Voisonics Limited<br>
<a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>
USA: +1 213 221 1092<br>
New Zealand: +64 (0)28 2558 3782</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<pre>________
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a>
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: <a href="https://bluejeans.com/441850968" target="_blank">https://bluejeans.com/441850968</a>
Gluster-users mailing list
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
<br>
<br>
<b style="color:rgb(96,76,120)"></b><br><br><span style="color:rgb(96,76,120)"><font color="000000"><span lang="NL">Met vriendelijke groet, With kind regards,<br><br>Jorick Astrego<br></span></font></span><b style="color:rgb(96,76,120)"><br>Netbulae Virtualization Experts </b><br><hr style="border-color:rgb(204,204,204) currentcolor currentcolor;border-style:solid none none;border-width:1px medium medium"><table style="width:522px"><tbody><tr><td style="width:130px;font-size:10px">Tel: 053 20 30 270</td> <td style="width:130px;font-size:10px"><a href="mailto:info@netbulae.eu" target="_blank">info@netbulae.eu</a></td> <td style="width:130px;font-size:10px">Staalsteden 4-3A</td> <td style="width:130px;font-size:10px">KvK 08198180</td></tr><tr> <td style="width:130px;font-size:10px">Fax: 053 20 30 271</td> <td style="width:130px;font-size:10px"><a href="http://www.netbulae.eu" target="_blank">www.netbulae.eu</a></td> <td style="width:130px;font-size:10px">7547 TA Enschede</td> <td style="width:130px;font-size:10px">BTW NL821234584B01</td></tr></tbody></table><br><hr style="border-color:rgb(204,204,204) currentcolor currentcolor;border-style:solid none none;border-width:1px medium medium"><br>
</div>
________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>David Cunningham, Voisonics Limited<br><a href="http://voisonics.com/" target="_blank">http://voisonics.com/</a><br>USA: +1 213 221 1092<br>New Zealand: +64 (0)28 2558 3782</div></div></div></div></div></div></div></div></div></div></div>