[Gluster-users] Gluster High CPU/Clients Hanging on Heavy Writes

[Xavi Hernandez]

Hi Yuhao,

On Mon, 6 Aug 2018, 15:26 Yuhao Zhang, <zzyzxd at gmail.com> wrote:

> Hello,
>
> I just experienced another hanging one hour ago and the server was not
> even under heavy IO.
>
> Atin, I attached the process monitoring results and another statedump.
>
> Xavi, ZFS was fine, during the hanging, I can still write directly to the
> ZFS volume. My ZFS version: ZFS: Loaded module v0.6.5.6-0ubuntu16, ZFS pool
> version 5000, ZFS filesystem version 5
>

I highly recommend you to upgrade to version 0.6.5.8 at least. It fixes a
kernel panic that can happen when used with gluster. However this is not
your current problem.

Top statistics show low available memory and high CPU utilization of kswapd
process (along with one of the gluster processes). I've seen frequent
memory management problems with ZFS. Have you configured any ZFS
parameters? It's highly recommendable to tweak some memory limits.

If that were the problem, there's one thing that should alleviate it (and
see if it could be related):

echo 3 >/proc/sys/vm/drop_caches

This should be done on all bricks from time to time. You can wait until the
problem appears, but in this case the recovery time can be larger.

I think this should fix the high CPU usage of kswapd. If so, we'll need to
tweak some ZFS parameters.

I'm not sure if the high CPU usage of gluster could be related to this or
not.

Xavi

>
> Thank you,
> Yuhao
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180807/ee0b70fc/attachment.html>