[Gluster-users] Write operations failing on clients

[Alex]

Hello.

We've been using glusterfs for five months without any problems until
yesterday: suddenly all clients who tried to write something began to hang
with "D" status (waiting for disk). Also at the same time gluster nodes
began to consume very high CPU which never happened before. "htop" command
suggested that it was mostly kernel load via gluster processes.

I tried so stop rebalance task, restarted volume and even upgraded from
3.5.2 to 3.5.3 but it did'nt help. Currently we only reading from cluster
and writing to other place. And CPU usage is still on high level.

Gluster is running in 3 x 2, distributed-replicate volume on top of ext4.
We're keeping about 5 TB of small files with high amount of hard links.
Binaries deployed from source (via gentoo portage). Kernel version is 3.8.13.

There are many debug information in logs but to me it looks vague and I
don't know where to start. I can provide it if needed.

I'm lost here. Any help appreciated.

Thanks,
Alex