[Gluster-devel] NFS

Fri Jun 12 15:44:44 UTC 2015

On Fri, Jun 12, 2015 at 03:17:08PM +0000, David Robinson wrote:
> My NFS stopped working after upgrading to 3.6.3.  When I do a gluster volume
> status homegfs_bkp, it shows as N/A and I cannot mount the volume using NFS
> instead of FUSE.
> Any suggestions for how to fix?
> 
> [root at gfs01bkp glusterfs]# gluster volume status homegfs_bkp
> Status of volume: homegfs_bkp
> Gluster process                                         Port    Online  Pid
> ------------------------------------------------------------------------------
> Brick gfsib01bkp.corvidtec.com:/data/brick01bkp/homegfs
> _bkp                                                    49152   Y       4311
> Brick gfsib01bkp.corvidtec.com:/data/brick02bkp/homegfs
> _bkp                                                    49155   Y       4318
> NFS Server on localhost                                 N/A     N       N/A
> 
> Task Status of Volume homegfs_bkp
> ------------------------------------------------------------------------------
> Task                 : Rebalance
> ID                   : 6d4c6c4e-16da-48c9-9019-dccb7d2cfd66
> Status               : completed
> 
> 
> From the /var/log/glusterfs/nfs.log file:
> 
> st_atim.tv_nsec 1
> package-string: glusterfs 3.6.3
> [2015-06-12 15:05:09.184368] W [glusterfsd.c:1194:cleanup_and_exit] (--> 0-:
> received signum (0), shutting down
> [2015-06-12 15:09:40.589589] I [MSGID: 100030] [glusterfsd.c:2018:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.6.3
> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p
> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S
> /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket)
> [2015-06-12 15:09:40.622811] I
> [rpcsvc.c:2142:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured
> rpc.outstanding-rpc-limit with value 16
> [2015-06-12 15:09:40.683881] E
> [rpcsvc.c:1303:rpcsvc_program_register_portmap] 0-rpc-service: Could not
> register with portmap 100021 4 38468
> [2015-06-12 15:09:40.683907] E [nfs.c:331:nfs_init_versions] 0-nfs: Program
> NLM4 registration failed
> [2015-06-12 15:09:40.683926] E [nfs.c:1342:init] 0-nfs: Failed to initialize
> protocols

These messages suggest that the portmapper/rpcbind already has a NLM
service registered. Gluster/NFS tries to be friendly to other services
and does not kick them out. Sometimes it happens that there is a stale
registration left behind... One cause was a change in the rpcbind
systemd unit, see https://bugzilla.redhat.com/1181779 for more details.

If you are on RHEL-7 or CentOS-7, this solution is known to work:

    # sed "s/ -w//" /usr/lib/systemd/system/rpcbind.service > \
                    /etc/systemd/system/rpcbind.service
    # systemctl daemon-reload
    # systemctl restart rpcbind
    # systemctl restart glusterd

Alternatively you can check with 'rpcinfo -p' which ports are registered
at the portmapper, and check with 'netstat' or 'ss' if there is a
process listening on these ports. You can manually unregister the
(stale) ports with 'rpcinfo -d 100021 4' (NLM, version 4).

HTH,
Niels

> [2015-06-12 15:09:40.683933] E [xlator.c:425:xlator_init] 0-nfs-server:
> Initialization of volume 'nfs-server' failed, review your volfile again
> [2015-06-12 15:09:40.683939] E [graph.c:322:glusterfs_graph_init]
> 0-nfs-server: initializing translator failed
> [2015-06-12 15:09:40.683945] E [graph.c:525:glusterfs_graph_activate]
> 0-graph: init failed
> pending frames:
> frame : type(0) op(0)
> patchset: git://git.gluster.com/glusterfs.git
> signal received: 11
> time of crash:
> 2015-06-12 15:09:40
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 3.6.3
> 
> Gluster version:
> [root at gfs01bkp glusterfs]# rpm -qa | grep gluster
> glusterfs-server-3.6.3-1.el6.x86_64
> glusterfs-3.6.3-1.el6.x86_64
> glusterfs-rdma-3.6.3-1.el6.x86_64
> glusterfs-api-3.6.3-1.el6.x86_64
> glusterfs-devel-3.6.3-1.el6.x86_64
> glusterfs-api-devel-3.6.3-1.el6.x86_64
> glusterfs-libs-3.6.3-1.el6.x86_64
> glusterfs-extra-xlators-3.6.3-1.el6.x86_64
> glusterfs-geo-replication-3.6.3-1.el6.x86_64
> samba-vfs-glusterfs-4.1.11-2.el6.x86_64
> glusterfs-fuse-3.6.3-1.el6.x86_64
> glusterfs-debuginfo-3.6.3-1.el6.x86_64
> glusterfs-cli-3.6.3-1.el6.x86_64
> 
> 
> [root at gfs01bkp glusterfs]# gluster volume info homegfs_bkp
> 
> Volume Name: homegfs_bkp
> Type: Distribute
> Volume ID: 96de8872-d957-4205-bf5a-076e3f35b294
> Status: Started
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: gfsib01bkp.corvidtec.com:/data/brick01bkp/homegfs_bkp
> Brick2: gfsib01bkp.corvidtec.com:/data/brick02bkp/homegfs_bkp
> 
> 
> ========================
> 
> 
> 
> David F. Robinson, Ph.D.
> 
> President - Corvid Technologies
> 
> 145 Overhill Drive
> 
> Mooresville, NC 28117
> 
> 704.799.6944 x101   [Office]
> 
> 704.252.1310           [Cell]
> 
> 704.799.7974           [Fax]
> 
> david.robinson at corvidtec.com
> 
> http://www.corvidtec.com
> 

> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel