[Bugs] [Bug 1215385] New: rmtab file is a bottleneck when lot of clients are accessing a volume through NFS
bugzilla at redhat.com
bugzilla at redhat.com
Sun Apr 26 08:41:48 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1215385
Bug ID: 1215385
Summary: rmtab file is a bottleneck when lot of clients are
accessing a volume through NFS
Product: GlusterFS
Version: 3.7.0
Component: nfs
Keywords: Triaged
Severity: urgent
Assignee: ndevos at redhat.com
Reporter: ndevos at redhat.com
CC: bugs at gluster.org, gluster-bugs at redhat.com
Depends On: 1169317
Blocks: 1199352 (glusterfs-3.7.0)
+++ This bug was initially created as a clone of Bug #1169317 +++
+++ This bug was initially created as a clone of Bug #1166862 +++
Description of problem:
This feature: http://review.gluster.org/#/c/4430/
Create bottleneck when several clients are accessing a nfs volume.
On our setup:
Gluster 3.5.2 under centos7.
Hardware:
dual Xeon® CPU E5-2640
64GB RAM
SSD for rootfs
10Gb NIC
Context:
Around 700 nfs clients for small file or vm images.
Version-Release number of selected component (if applicable):
3.5.2
How reproducible:
Always as long as you have enough NFS clients
Steps to Reproduce:
1. Create a volume accessible through gluster nfs
2. Make it accessible for 700 clients
3. See how it hangs
Actual results:
NFS client got intermittent hang (every minutes / for 10s each time). Even an
“rpcinfo -t server nfs 3" will hang.
Gluster nfs process literally eat the CPU of the server
Expected results:
No hanging
Additional info:
The cause:
the rmtab file located in /var/lib/glusterd/nfs/ is flushed from memory to
/var/lib/glusterd/nfs/rmtab.tmp. During this time, NFS server hang literraly.
Workaroud:
Move the file to memory for faster I/O using this option:
set nfs.mount-rmtab: /dev/shm/glusterfs.rmtab
Result:
We still have some hang but for ~300ms now, the load average of the server is
WAY better.
Personal thought:
This feature is not usable and should be disabled by default.
You can find attached load average and Disk usage before and after using SHM
for rmtab.
--- Additional comment from Anand Avati on 2014-12-01 11:32:14 CET ---
REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable
nfs.mount-rmtab) posted (#2) for review on master by Niels de Vos
(ndevos at redhat.com)
--- Additional comment from Anand Avati on 2014-12-01 15:18:03 CET ---
REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable
nfs.mount-rmtab) posted (#4) for review on master by Niels de Vos
(ndevos at redhat.com)
--- Additional comment from Anand Avati on 2014-12-02 12:14:50 CET ---
REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable
nfs.mount-rmtab) posted (#5) for review on master by Niels de Vos
(ndevos at redhat.com)
--- Additional comment from Anand Avati on 2014-12-05 22:30:27 CET ---
REVIEW: http://review.gluster.org/9223 (nfs: make it possible to disable
nfs.mount-rmtab) posted (#6) for review on master by Niels de Vos
(ndevos at redhat.com)
--- Additional comment from Anand Avati on 2015-04-26 10:40:25 CEST ---
COMMIT: http://review.gluster.org/9223 committed in master by Niels de Vos
(ndevos at redhat.com)
------
commit 331ef6e1a86bfc0a93f8a9dec6ad35c417873849
Author: Niels de Vos <ndevos at redhat.com>
Date: Tue Dec 2 10:54:53 2014 +0100
nfs: make it possible to disable nfs.mount-rmtab
When there are many NFS-clients doing very often mount/unmount actions,
the updating of the 'rmtab' can become a bottleneck and cause delays. In
these situations, the output of 'showmount' may be less important than
the responsiveness of the (un)mounting.
By setting 'nfs.mount-rmtab' to the value "/-", the cache file is not
updated anymore, and the entries are only kept in memory.
BUG: 1169317
Change-Id: I40c4d8d754932f86fb2b1b2588843390464c773d
Reported-by: Cyril Peponnet <cyril at peponnet.fr>
Signed-off-by: Niels de Vos <ndevos at redhat.com>
Reviewed-on: http://review.gluster.org/9223
Tested-by: Gluster Build System <jenkins at build.gluster.com>
Reviewed-by: soumya k <skoduri at redhat.com>
Reviewed-by: jiffin tony Thottan <jthottan at redhat.com>
Reviewed-by: Kaleb KEITHLEY <kkeithle at redhat.com>
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1169317
[Bug 1169317] rmtab file is a bottleneck when lot of clients are accessing
a volume through NFS
https://bugzilla.redhat.com/show_bug.cgi?id=1199352
[Bug 1199352] GlusterFS 3.7.0 tracker
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=53mUZjBRhO&a=cc_unsubscribe
More information about the Bugs
mailing list