[Gluster-users] Gluster High CPU/Clients Hanging on Heavy Writes

[Yuhao Zhang]

Hi,

I am running into a situation that heavy write causes Gluster server went into zombie with many high CPU processes and all clients hangs, it is almost 100% reproducible on my machine. Hope someone can help.

I started to observe this issue when running rsync to copy files from another server and I thought it might be because Gluster doesn't like rsync's delta transfer with a lot of small writes. However, I was able to reproduce this with "rsync --whole-file --inplace", or even with cp or scp. It usually appears after starting the transfer for a few hours, but sometimes can happen within several minutes.

Since this is a single node Gluster distributed volume, I tried to transfer files directly onto the server bypassing Gluster clients, but it still caused the same issue.

It is running on top of a ZFS RAIDZ2 dataset. Options are attached. Also, I attached the statedump generated when my clients hung, and volume options.

- Ubuntu 16.04 x86_64 / 4.4.0-116-generic
- GlusterFS 3.12.8

Thank you,
Yuhao

-------------- next part --------------
A non-text attachment was scrubbed...
Name: zfs_options
Type: application/octet-stream
Size: 4220 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180805/77c0c46e/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: volume_options
Type: application/octet-stream
Size: 14895 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180805/77c0c46e/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zfs-my_vol_name.6856.dump.1533387154
Type: application/octet-stream
Size: 3681718 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180805/77c0c46e/attachment-0005.obj>