[Gluster-users] rmtab file is a bottleneck when lot of clients are accessing a volume through NFS

[PEPONNET, Cyril N (Cyril)]

Folks,

This feature:

http://review.gluster.org/#/c/4430/

Create bottleneck when several clients are accessing a nfs volume.

On our setup:

Gluster 3.5.2 under centos7.

Hardware:

dual Xeon® CPU E5-2640
64GB RAM
SSD for rootfs
10Gb NIC

Context:

Around 700 nfs clients for small file or vm images.

The issue:

NFS client got intermittent hang (every minutes / for 10s each time). Even an “rpcinfo -t server nfs 3" will hang.

Gluster nfs process literally eat the CPU of the server

The cause:

the rmtab file located in /var/lib/glusterd/nfs/ is flushed from memory to  /var/lib/glusterd/nfs/rmtab.tmp. During this time, NFS server hang literraly.

[cid:2BF1DF8D-4144-48FF-A63E-F5D2F61285CA at eng.timetra.com]

[cid:D11F71BF-A4FD-4B96-B7A1-17AB2F7114A3 at eng.timetra.com]


Workaroud:

Move the file to memory for faster I/O using this option:

set nfs.mount-rmtab: /dev/shm/glusterfs.rmtab

Result:

We still have some hang but for ~300ms now, the load average of the server is WAY better.

Personal thought:

This feature is not usable and should be disabled by default.

I raised Bug 1166862<https://bugzilla.redhat.com/show_bug.cgi?id=1166862>.
--
Cyril Peponnet

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141121/b404808c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: loadaverage.png
Type: image/png
Size: 116283 bytes
Desc: loadaverage.png
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141121/b404808c/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: diskusage.png
Type: image/png
Size: 120872 bytes
Desc: diskusage.png
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141121/b404808c/attachment-0001.png>