<div dir="ltr">If you have both systems to get some idea, can you get the `gluster profile info' output? That helps a bit to understand the issue.<div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Feb 21, 2019 at 8:20 AM Kartik Subbarao <<a href="mailto:subbarao@computer.org">subbarao@computer.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">We're running gluster on two hypervisors running Ubuntu. When we <br>
upgraded from Ubuntu 14.04 to 18.04, it upgraded gluster from 3.4.2 to <br>
3.13.2. As soon as we upgraded and since then, we've been seeing <br>
substantially higher iowait on the system, as measured by top and iotop, <br>
and iotop indicates that glusterfsd is the culprit. For some reason, <br>
glusterfsd is doing more disk reads and/or those reads are being held up <br>
up at a greater rate. The guest VMs are also seeing more iowait -- their <br>
images are hosted on the gluster volume. This is causing inconsistent <br>
responsiveness from the services hosted on the VMs.<br>
<br>
I'm looking for any recommendations on how to troubleshoot and/or <br>
resolve this problem. We have other sites that are still running 14.04, <br>
so I can compare/contrast any configuration parameters and performance.<br>
<br>
The block scheduler on 14.04 was set to deadline and 18.04 was set to <br>
cfq. But changing the 18.04 scheduler to deadline didn't make any <br>
difference.<br>
<br>
I was wondering whether glusterfsd on 18.04 isn't caching as much as it <br>
should. We tried increasing performance.cache-size substantially but <br>
that didn't make any difference.<br>
<br>
Another option we're considering but haven't tried yet is upgrading to <br>
gluster 5.3 by back-porting the package from Ubuntu 19.04 to 18.04. Does <br>
anyone think this might help?<br>
<br>
Is there any particular debug logging we could set up or other commands <br>
we could run to troubleshoot this better? Any thoughts, suggestions, <br>
ideas would be greatly appreciated.<br>
<br>
Thanks,<br>
<br>
-Kartik<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>