<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>I have some good news -- upgrading to gluster 5.3 resolved the
issue :-)</p>
<p>Regards,</p>
<p> -Kartik<br>
</p>
<div class="moz-cite-prefix">On 2/21/2019 11:34 AM, Kartik Subbarao
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:13d40e22-4fd2-0a8d-9606-d536afb12d52@computer.org">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<div class="moz-cite-prefix">Here are three profile reports from
60-second intervals:</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Ubuntu 18.04 system with low load:</div>
<div class="moz-cite-prefix"><a class="moz-txt-link-freetext"
href="https://pastebin.com/XzgmjeuJ" moz-do-not-send="true">https://pastebin.com/XzgmjeuJ</a></div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Ubuntu 14.04 system with low load:</div>
<div class="moz-cite-prefix"><a class="moz-txt-link-freetext"
href="https://pastebin.com/5BEHDFwq" moz-do-not-send="true">https://pastebin.com/5BEHDFwq</a></div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Ubuntu 14.04 system with high load:</div>
<div class="moz-cite-prefix"><a class="moz-txt-link-freetext"
href="https://pastebin.com/CFSWW4qn" moz-do-not-send="true">https://pastebin.com/CFSWW4qn</a></div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Each of these systems is "gluster1"
in the report. In each cluster, there are two bricks,
gluster1:/md3/gluster and gluster2:/md3/gluster. The systems are
identical hardware-wise (I noticed this morning that the 18.04
upgrade applied a powersave governor to the CPU. I changed it to
the performance governor before running the profile, but that
doesn't seem to have changed the iowait behavior or the profile
report appreciably).</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">What jumps out at me for the 18.04
systems is:</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">1) The excessively high average
latency of the FINODELK operations on the *local* brick (i.e.
gluster1:/md3/gluster). The latency is far lower for these
FINODELK operations against the other node's brick
(gluster2:/md3/gluster). This is puzzling to me.</div>
<div class="moz-cite-prefix">2) Almost double higher average
latency for FSYNC operations against both the gluster1 and
gluster2 bricks.<br>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On the 14.04 systems, the number of
FINODELK operations performed during the 60-second interval is
much lower (even on the highload system). And the latencies are
lower.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Regards,</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix"> -Kartik</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">On 2/21/19 12:18 AM, Amar Tumballi
Suryanarayan wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAHxyDdNRUWqW+S9vJEz7wVLGJcoKqsVNXQBnAnxtXgVUvjDV3g@mail.gmail.com">
<meta http-equiv="content-type" content="text/html;
charset=UTF-8">
<div dir="ltr">If you have both systems to get some idea, can
you get the `gluster profile info' output? That helps a bit to
understand the issue.
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Thu, Feb 21, 2019 at 8:20
AM Kartik Subbarao <<a
href="mailto:subbarao@computer.org" moz-do-not-send="true">subbarao@computer.org</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">We're running gluster on
two hypervisors running Ubuntu. When we <br>
upgraded from Ubuntu 14.04 to 18.04, it upgraded gluster
from 3.4.2 to <br>
3.13.2. As soon as we upgraded and since then, we've been
seeing <br>
substantially higher iowait on the system, as measured by
top and iotop, <br>
and iotop indicates that glusterfsd is the culprit. For some
reason, <br>
glusterfsd is doing more disk reads and/or those reads are
being held up <br>
up at a greater rate. The guest VMs are also seeing more
iowait -- their <br>
images are hosted on the gluster volume. This is causing
inconsistent <br>
responsiveness from the services hosted on the VMs.<br>
<br>
I'm looking for any recommendations on how to troubleshoot
and/or <br>
resolve this problem. We have other sites that are still
running 14.04, <br>
so I can compare/contrast any configuration parameters and
performance.<br>
<br>
The block scheduler on 14.04 was set to deadline and 18.04
was set to <br>
cfq. But changing the 18.04 scheduler to deadline didn't
make any <br>
difference.<br>
<br>
I was wondering whether glusterfsd on 18.04 isn't caching as
much as it <br>
should. We tried increasing performance.cache-size
substantially but <br>
that didn't make any difference.<br>
<br>
Another option we're considering but haven't tried yet is
upgrading to <br>
gluster 5.3 by back-porting the package from Ubuntu 19.04 to
18.04. Does <br>
anyone think this might help?<br>
<br>
Is there any particular debug logging we could set up or
other commands <br>
we could run to troubleshoot this better? Any thoughts,
suggestions, <br>
ideas would be greatly appreciated.<br>
<br>
Thanks,<br>
<br>
-Kartik<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank"
moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>Amar Tumballi (amarts)<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
</blockquote>
</body>
</html>