[Gluster-users] glusterfsd Ubuntu 18.04 high iowait issues

[Kartik Subbarao]

We're running gluster on two hypervisors running Ubuntu. When we 
upgraded from Ubuntu 14.04 to 18.04, it upgraded gluster from 3.4.2 to 
3.13.2. As soon as we upgraded and since then, we've been seeing 
substantially higher iowait on the system, as measured by top and iotop, 
and iotop indicates that glusterfsd is the culprit. For some reason, 
glusterfsd is doing more disk reads and/or those reads are being held up 
up at a greater rate. The guest VMs are also seeing more iowait -- their 
images are hosted on the gluster volume. This is causing inconsistent 
responsiveness from the services hosted on the VMs.

I'm looking for any recommendations on how to troubleshoot and/or 
resolve this problem. We have other sites that are still running 14.04, 
so I can compare/contrast any configuration parameters and performance.

The block scheduler on 14.04 was set to deadline and 18.04 was set to 
cfq. But changing the 18.04 scheduler to deadline didn't make any 
difference.

I was wondering whether glusterfsd on 18.04 isn't caching as much as it 
should. We tried increasing performance.cache-size substantially but 
that didn't make any difference.

Another option we're considering but haven't tried yet is upgrading to 
gluster 5.3 by back-porting the package from Ubuntu 19.04 to 18.04. Does 
anyone think this might help?

Is there any particular debug logging we could set up or other commands 
we could run to troubleshoot this better? Any thoughts, suggestions, 
ideas would be greatly appreciated.

Thanks,

     -Kartik