[Gluster-users] Slow NFS Mounts on one node in a replicated cluster

[Mark Selby]

I have two servers running Ubuntu 14.04 and Gluster 3.7.6

Each volume is a made up of two bricks replicated between the two servers.

I have this very strange situation whereby NFS mounts against ONE of the 
servers incurs a 5 second wait during the mount phase. NFS performance 
is fine after the mount.

FUSE mounts are fine on both server.

This slowness in NFS mounting occurs across ALL volumes on only one of 
the server

There are some messages in nfs.log on the server that has the slow 
mounts that are not on the server where the mounts are instant.

I have done some Google searching but nothing definitive comes up.

Has anyone ever seen this?

Any pointers on where else to look or how to further debug?

I see that the glusterfs process is responsible for mounting and am not 
sure how to ask it to display debug info.

I have done a tcpdump and see nothing other than the repsonse to the 
mount request takes 5 seconds to complete on one node in the set.

root at dc1strg001 /root 632# rpcinfo -p
    program vers proto   port  service
     100000    4   tcp    111  portmapper
     100000    3   tcp    111  portmapper
     100000    2   tcp    111  portmapper
     100000    4   udp    111  portmapper
     100000    3   udp    111  portmapper
     100000    2   udp    111  portmapper
     100005    3   tcp  38465  mountd
     100005    1   tcp  38466  mountd
     100003    3   tcp   2049  nfs
     100021    4   tcp  38468  nlockmgr
     100024    1   udp  32765  status
     100024    1   tcp  32765  status
     100227    3   tcp   2049
     100021    1   udp    643  nlockmgr
     100021    1   tcp    645  nlockmgr

root at dc1strg001 /root 635# gluster volume status home
Status of volume: home
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick dc1strg001:/zfspool/glusterfs/home/da
ta                                          49159 0          Y       32185
Brick dc1strg002:/zfspool/glusterfs/home/da
ta                                          49159 0          Y       31317
NFS Server on localhost                     2049 0          Y       466
Self-heal Daemon on localhost               N/A N/A        Y       482
NFS Server on dc1strg002                    2049 0          Y       31950
Self-heal Daemon on dc1strg002              N/A N/A        Y       31958

Task Status of Volume home
------------------------------------------------------------------------------
There are no active volume tasks

root at dc1strg001 /var/log/glusterfs 646# gluster volume get home all | 
grep nfs
performance.nfs.flush-behind on
performance.nfs.write-behind-window-size1MB
performance.nfs.strict-o-direct off
performance.nfs.strict-write-ordering off
performance.nfs.write-behind on
performance.nfs.read-ahead off
performance.nfs.io-cache off
performance.nfs.quick-read off
performance.nfs.stat-prefetch off
performance.nfs.io-threads off
nfs.enable-ino32 no
nfs.mem-factor 15
nfs.export-dirs on
nfs.export-volumes on
nfs.addr-namelookup off
nfs.dynamic-volumes off
nfs.register-with-portmap on
nfs.outstanding-rpc-limit 16
nfs.port 2049
nfs.rpc-auth-unix on
nfs.rpc-auth-null on
nfs.rpc-auth-allow all
nfs.rpc-auth-reject none
nfs.ports-insecure off
nfs.trusted-sync off
nfs.trusted-write off
nfs.volume-access read-write
nfs.export-dir
nfs.disable false
nfs.nlm on
nfs.acl on
nfs.mount-udp off
nfs.mount-rmtab /var/lib/glusterd/nfs/rmtab
nfs.rpc-statd /sbin/rpc.statd
nfs.server-aux-gids off
nfs.drc off
nfs.drc-size 0x20000
nfs.read-size                           (1 * 1048576ULL)
nfs.write-size                          (1 * 1048576ULL)
nfs.readdir-size                        (1 * 1048576ULL)
nfs.exports-auth-enable (null)
nfs.auth-refresh-interval-sec (null)
nfs.auth-cache-ttl-sec (null)

[2016-03-23 15:29:48.640220] W [rpcsvc.c:278:rpcsvc_program_actor] 
0-rpc-service: RPC program version not available (req 100003 2) for 
10.123.14.14:42552
[2016-03-23 15:29:48.640293] E 
[rpcsvc.c:565:rpcsvc_check_and_reply_error] 0-rpcsvc: rpc actor failed 
to complete successfully
[2016-03-23 15:30:28.488880] I [socket.c:3382:socket_submit_reply] 
0-socket.nfs-server: not connected (priv->connected = -1)
[2016-03-23 15:30:28.488953] E [rpcsvc.c:1314:rpcsvc_submit_generic] 
0-rpc-service: failed to submit message (XID: 0x18363d6c, Program: 
MOUNT3, ProgVers: 3, Proc: 3) to rpc-transport (socket.nfs-server)