[Gluster-users] [Gluster-devel] v3.6.2
David F. Robinson
david.robinson at corvidtec.com
Tue Jan 27 16:12:10 UTC 2015
In my /etc/fstab, I have the following:
gfsib01bkp.corvidtec.com:/homegfs_bkp /backup/homegfs
glusterfs transport=tcp,_netdev 0 0
gfsib01bkp.corvidtec.com:/Software_bkp /backup/Software
glusterfs transport=tcp,_netdev 0 0
gfsib01bkp.corvidtec.com:/Source_bkp /backup/Source
glusterfs transport=tcp,_netdev 0 0
#... Setup NFS mounts as well
gfsib01bkp.corvidtec.com:/homegfs_bkp /backup_nfs/homegfs nfs
vers=3,intr,bg,rsize=32768,wsize=32768 0 0
It looks like it is trying to start the nfs mount before gluster has
finished coming up and that this is hanging the nfs ports. I have
_netdev in the glusterfs mount point to make sure the network has come
up (including infiniband) prior to starting gluster. Shouldn't the
gluster init scripts check for gluster startup prior to starting the nfs
mount? It doesn't look like this is working properly.
David
------ Original Message ------
From: "Xavier Hernandez" <xhernandez at datalab.es>
To: "David F. Robinson" <david.robinson at corvidtec.com>; "Kaushal M"
<kshlmster at gmail.com>
Cc: "Gluster Users" <gluster-users at gluster.org>; "Gluster Devel"
<gluster-devel at gluster.org>
Sent: 1/27/2015 10:02:31 AM
Subject: Re: [Gluster-devel] [Gluster-users] v3.6.2
>Hi,
>
>I had a similar problem once. It happened after doing some unrelated
>tests with NFS. I thought it was a problem I generated doing weird
>things, so I didn't investigate the cause further.
>
>To see if this is the same case, try this:
>
>* Unmount all NFS mounts and stop all gluster volumes
>* Check that there are no gluster processes running (ps ax | grep
>gluster), specially any glusterfs. glusterd is ok.
>* Check that there are no NFS processes running (ps ax | grep nfs)
>* Check with 'rpcinfo -p' that there's no nfs service registered
>
>The output should be similar to this:
>
> program vers proto port service
> 100000 4 tcp 111 portmapper
> 100000 3 tcp 111 portmapper
> 100000 2 tcp 111 portmapper
> 100000 4 udp 111 portmapper
> 100000 3 udp 111 portmapper
> 100000 2 udp 111 portmapper
> 100024 1 udp 33482 status
> 100024 1 tcp 37034 status
>
>If there are more services registered, you can directly delete them or
>check if they correspond to an active process. For example, if the
>output is this:
>
> program vers proto port service
> 100000 4 tcp 111 portmapper
> 100000 3 tcp 111 portmapper
> 100000 2 tcp 111 portmapper
> 100000 4 udp 111 portmapper
> 100000 3 udp 111 portmapper
> 100000 2 udp 111 portmapper
> 100021 3 udp 39618 nlockmgr
> 100021 3 tcp 41067 nlockmgr
> 100024 1 udp 33482 status
> 100024 1 tcp 37034 status
>
>You can do a "netstat -anp | grep 39618" to see if there is some
>process really listening at the nlockmgr port. You can repeat this for
>port 41067. If there is some process, you should stop it. If there is
>no process listening on that port, you should remove it with a command
>like this:
>
> rpcinfo -d 100021 3
>
>You must execute this command for all stale ports for any services
>other than portmapper and status. Once done you should get the output
>shown before.
>
>After that, you can try to start your volume and see if everything is
>registered (rpcinfo -p) and if gluster has started the nfs server
>(gluster volume status).
>
>If everything is ok, you should be able to mount the volume using NFS.
>
>Xavi
>
>On 01/27/2015 03:18 PM, David F. Robinson wrote:
>>Turning off nfslock did not help. Also, still getting these messages
>>every 3-seconds:
>>
>>[2015-01-27 14:16:12.921880] W [socket.c:611:__socket_rwv]
>>0-management:
>>readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
>>(Invalid argument)
>>[2015-01-27 14:16:15.922431] W [socket.c:611:__socket_rwv]
>>0-management:
>>readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
>>(Invalid argument)
>>[2015-01-27 14:16:18.923080] W [socket.c:611:__socket_rwv]
>>0-management:
>>readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
>>(Invalid argument)
>>[2015-01-27 14:16:21.923748] W [socket.c:611:__socket_rwv]
>>0-management:
>>readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
>>(Invalid argument)
>>[2015-01-27 14:16:24.924472] W [socket.c:611:__socket_rwv]
>>0-management:
>>readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
>>(Invalid argument)
>>[2015-01-27 14:16:27.925192] W [socket.c:611:__socket_rwv]
>>0-management:
>>readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
>>(Invalid argument)
>>[2015-01-27 14:16:30.925895] W [socket.c:611:__socket_rwv]
>>0-management:
>>readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
>>(Invalid argument)
>>[2015-01-27 14:16:33.926563] W [socket.c:611:__socket_rwv]
>>0-management:
>>readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
>>(Invalid argument)
>>[2015-01-27 14:16:36.927248] W [socket.c:611:__socket_rwv]
>>0-management:
>>readv on /var/run/1f0cee5a2d074e39b32ee5a81c70e68c.socket failed
>>(Invalid argument)
>>------ Original Message ------
>>From: "Kaushal M" <kshlmster at gmail.com <mailto:kshlmster at gmail.com>>
>>To: "David F. Robinson" <david.robinson at corvidtec.com
>><mailto:david.robinson at corvidtec.com>>
>>Cc: "Joe Julian" <joe at julianfamily.org <mailto:joe at julianfamily.org>>;
>>"Gluster Users" <gluster-users at gluster.org
>><mailto:gluster-users at gluster.org>>; "Gluster Devel"
>><gluster-devel at gluster.org <mailto:gluster-devel at gluster.org>>
>>Sent: 1/27/2015 1:49:56 AM
>>Subject: Re: Re[2]: [Gluster-devel] [Gluster-users] v3.6.2
>>>Your nfs.log file has the following lines,
>>>```
>>>[2015-01-26 20:06:58.298078] E
>>>[rpcsvc.c:1303:rpcsvc_program_register_portmap] 0-rpc-service: Could
>>>not register with portmap 100021 4 38468
>>>[2015-01-26 20:06:58.298108] E [nfs.c:331:nfs_init_versions] 0-nfs:
>>>Program NLM4 registration failed
>>>```
>>>
>>>The Gluster NFS server has it's own NLM (nlockmgr) implementation.
>>>You
>>>said that you have the nfslock service on. Can you turn that off and
>>>try again?
>>>
>>>~kaushal
>>>
>>>On Tue, Jan 27, 2015 at 11:21 AM, David F. Robinson
>>><david.robinson at corvidtec.com <mailto:david.robinson at corvidtec.com>>
>>>wrote:
>>>
>>> On a different system where gluster-NFS (not kernel-nfs) is
>>> working properly shows the following:
>>> [root at gfs01a glusterfs]# rpcinfo -p
>>> program vers proto port service
>>> 100000 4 tcp 111 portmapper
>>> 100000 3 tcp 111 portmapper
>>> 100000 2 tcp 111 portmapper
>>> 100000 4 udp 111 portmapper
>>> 100000 3 udp 111 portmapper
>>> 100000 2 udp 111 portmapper
>>> 100005 3 tcp 38465 mountd
>>> 100005 1 tcp 38466 mountd
>>> 100003 3 tcp 2049 nfs
>>> 100024 1 udp 42413 status
>>> 100024 1 tcp 35424 status
>>> 100021 4 tcp 38468 nlockmgr
>>> 100021 1 udp 801 nlockmgr
>>> 100227 3 tcp 2049 nfs_acl
>>> 100021 1 tcp 804 nlockmgr
>>>
>>> [root at gfs01a glusterfs]# /etc/init.d/nfs status
>>> rpc.svcgssd is stopped
>>> rpc.mountd is stopped
>>> nfsd is stopped
>>> rpc.rquotad is stopped
>>> ------ Original Message ------
>>> From: "Joe Julian" <joe at julianfamily.org
>>> <mailto:joe at julianfamily.org>>
>>> To: "Kaushal M" <kshlmster at gmail.com
>>> <mailto:kshlmster at gmail.com>>; "David F. Robinson"
>>> <david.robinson at corvidtec.com
>>><mailto:david.robinson at corvidtec.com>>
>>> Cc: "Gluster Users" <gluster-users at gluster.org
>>> <mailto:gluster-users at gluster.org>>; "Gluster Devel"
>>> <gluster-devel at gluster.org <mailto:gluster-devel at gluster.org>>
>>> Sent: 1/27/2015 12:48:49 AM
>>> Subject: Re: [Gluster-devel] [Gluster-users] v3.6.2
>>>> If that was true, wouldn't it not "Connection refused" because
>>>> the kernel nfs is listening?
>>>>
>>>> On January 26, 2015 9:43:34 PM PST, Kaushal M
>>>> <kshlmster at gmail.com <mailto:kshlmster at gmail.com>> wrote:
>>>>
>>>> Seems like you have the kernel NFS server running. The
>>>> `rpcinfo -p` output you provided shows that there are other
>>>> mountd, nfs and nlockmgr services running your system.
>>>> Gluster NFS server requires that the kernel nfs services be
>>>> disabled and not running.
>>>>
>>>> ~kaushal
>>>>
>>>> On Tue, Jan 27, 2015 at 10:56 AM, David F. Robinson
>>>> <david.robinson at corvidtec.com
>>>> <mailto:david.robinson at corvidtec.com>> wrote:
>>>>
>>>> [root at gfs01bkp ~]# gluster volume status homegfs_bkp
>>>> Status of volume: homegfs_bkp
>>>> Gluster process
>>>> Port Online Pid
>>>>
>>>>------------------------------__------------------------------__------------------
>>>> Brick
>>>>gfsib01bkp.corvidtec.com:/__data/brick01bkp/homegfs
>>>> _bkp
>>>> 49152 Y 4087
>>>> Brick
>>>>gfsib01bkp.corvidtec.com:/__data/brick02bkp/homegfs
>>>> _bkp
>>>> 49155 Y 4092
>>>> NFS Server on localhost
>>>> N/A N N/A
>>>>
>>>> Task Status of Volume homegfs_bkp
>>>>
>>>>------------------------------__------------------------------__------------------
>>>> Task : Rebalance
>>>> ID : 6d4c6c4e-16da-48c9-9019-__dccb7d2cfd66
>>>> Status : completed
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ------ Original Message ------
>>>> From: "Atin Mukherjee" <amukherj at redhat.com
>>>> <mailto:amukherj at redhat.com>>
>>>> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com
>>>> <mailto:pkarampu at redhat.com>>; "Justin Clift"
>>>> <justin at gluster.org <mailto:justin at gluster.org>>; "David
>>>> F. Robinson" <david.robinson at corvidtec.com
>>>> <mailto:david.robinson at corvidtec.com>>
>>>> Cc: "Gluster Users" <gluster-users at gluster.org
>>>> <mailto:gluster-users at gluster.org>>; "Gluster Devel"
>>>> <gluster-devel at gluster.org
>>>> <mailto:gluster-devel at gluster.org>>
>>>> Sent: 1/26/2015 11:51:13 PM
>>>> Subject: Re: [Gluster-devel] v3.6.2
>>>>
>>>>
>>>>
>>>> On 01/27/2015 07:33 AM, Pranith Kumar Karampuri
>>>>wrote:
>>>>
>>>>
>>>> On 01/26/2015 09:41 PM, Justin Clift wrote:
>>>>
>>>> On 26 Jan 2015, at 14:50, David F. Robinson
>>>> <david.robinson at corvidtec.com
>>>> <mailto:david.robinson at corvidtec.com>>
>>>>wrote:
>>>>
>>>> I have a server with v3.6.2 from which
>>>>I
>>>> cannot mount using NFS. The
>>>> FUSE mount works, however, I cannot get
>>>> the NFS mount to work. From
>>>> /var/log/message:
>>>> Jan 26 09:27:28 gfs01bkp mount[2810]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> Jan 26 09:27:53 gfs01bkp mount[4456]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> Jan 26 09:29:28 gfs01bkp mount[2810]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> Jan 26 09:29:53 gfs01bkp mount[4456]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> Jan 26 09:31:28 gfs01bkp mount[2810]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> Jan 26 09:31:53 gfs01bkp mount[4456]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> Jan 26 09:33:28 gfs01bkp mount[2810]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> Jan 26 09:33:53 gfs01bkp mount[4456]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> Jan 26 09:35:28 gfs01bkp mount[2810]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> Jan 26 09:35:53 gfs01bkp mount[4456]:
>>>> mount to NFS server
>>>> 'gfsib01bkp.corvidtec.com
>>>> <http://gfsib01bkp.corvidtec.com/>'
>>>> failed: Connection refused, retrying
>>>> I also am continually getting the
>>>> following errors in
>>>> /var/log/glusterfs:
>>>> [root at gfs01bkp glusterfs]# tail -f
>>>> etc-glusterfs-glusterd.vol.log
>>>> [2015-01-26 14:41:51.260827] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:41:54.261240] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:41:57.261642] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:42:00.262073] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:42:03.262504] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:42:06.262935] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:42:09.263334] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:42:12.263761] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:42:15.264177] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:42:18.264623] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:42:21.265053] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>> [2015-01-26 14:42:24.265504] W
>>>> [socket.c:611:__socket_rwv]
>>>> 0-management: readv on
>>>>
>>>>/var/run/__1f0cee5a2d074e39b32ee5a81c70e6__8c.socket
>>>> failed (Invalid
>>>> argument)
>>>>
>>>> I believe this error message comes when the
>>>> socket file is not present.
>>>> I see the following commit which changed the
>>>> location of the sockets.
>>>> May be Atin may know more. about this: +Atin.
>>>>
>>>> Can we get the output of 'gluster volume status' for
>>>> the volume which
>>>> you are trying to mount?
>>>>
>>>> ~Atin
>>>>
>>>> Pranith
>>>>
>>>> ^C
>>>> Also, when I try to NFS mount my
>>>> gluster volume, I am getting
>>>>
>>>> Any chance there's a network or host based
>>>> firewall stopping some of
>>>> the ports?
>>>>
>>>> + Justin
>>>>
>>>> --
>>>> GlusterFS - http://www.gluster.org/
>>>>
>>>> An open source, distributed file system
>>>> scaling to several
>>>> petabytes, and handling thousands of
>>>>clients.
>>>>
>>>> My personal twitter:
>>>> twitter.com/realjustinclift
>>>> <http://twitter.com/realjustinclift>
>>>>
>>>>
>>>>_________________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> <mailto:Gluster-devel at gluster.org>
>>>>
>>>>http://www.gluster.org/__mailman/listinfo/gluster-devel
>>>>
>>>><http://www.gluster.org/mailman/listinfo/gluster-devel>
>>>>
>>>>
>>>>
>>>> _________________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>><mailto:Gluster-users at gluster.org>
>>>> http://www.gluster.org/__mailman/listinfo/gluster-users
>>>> <http://www.gluster.org/mailman/listinfo/gluster-users>
>>>>
>>>>
>>>>
>>>>------------------------------------------------------------------------
>>>>
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>>
>>>> --
>>>> Sent from my Android device with K-9 Mail. Please excuse my
>>>>brevity.
>>>
>>>
>>
>>
>>_______________________________________________
>>Gluster-devel mailing list
>>Gluster-devel at gluster.org
>>http://www.gluster.org/mailman/listinfo/gluster-devel
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150127/84373c97/attachment.html>
More information about the Gluster-users
mailing list