[Gluster-users] [Nfs-ganesha-devel] Questions on ganesha HA and shared storage size

[Malahal Naineni]

The logs indicate that ganesha was started successfully without any
exports.  gstack output seemed normal as well -- threads were waiting to
serve requests.

Assuming that you are running "showmount -e" on the same system, there
shouldn't be any firewall coming into the picture. If you are running
"showmount" from some other system, make sure there is no firewall
dropping the packets.

I think you need tcpdump trace to figure out the problem. My wireshark
trace showed two requests from the client to complete the "showmount -e"
command:

1. Client sent "GETPORT" call to port 111 (rpcbind) to get the port number
   of MOUNT.
2. Then it sent "EXPORT" call to mountd port (port it got in response to #1).

What does "rpcinfo -p <server-ip>" show?

Do you have selinux enabled? I am not sure if that is playing any role
here...

Regards, Malahal.

Alessandro De Salvo [Alessandro.DeSalvo at roma1.infn.it] wrote:
> Hi,
> this was an extract from the old logs, before Soumya's suggestion of
> changing the rquota port in the conf file. The new logs are attached
> (ganesha-20150611.log.gz) as well as the gstack of the ganesha process
> while I was executing the hanging showmount
> (ganesha-20150611.gstack.gz).
> Thanks,
> 
> 	Alessandro
> 
> 
> 
> On Thu, 2015-06-11 at 11:37 -0500, Malahal Naineni wrote:
> > Soumya Koduri [skoduri at redhat.com] wrote:
> > > CCin ganesha-devel to get more inputs.
> > > 
> > > In case of ipv6 enabled, only v6 interfaces are used by NFS-Ganesha.
> > 
> > I am not a network expert but I have seen IPv4 traffic over IPv6
> > interface while fixing few things before. This may be normal.
> > 
> > > commit - git show 'd7e8f255' , which got added in v2.2 has more details.
> > > 
> > >  > # netstat -ltaupn | grep 2049
> > >  > tcp6       4      0 :::2049                 :::*
> > >  > LISTEN      32080/ganesha.nfsd
> > >  > tcp6       1      0 x.x.x.2:2049      x.x.x.2:33285     CLOSE_WAIT
> > >  > -
> > >  > tcp6       1      0 127.0.0.1:2049          127.0.0.1:39555
> > >  > CLOSE_WAIT  -
> > >  > udp6       0      0 :::2049                 :::*
> > >  > 32080/ganesha.nfsd
> > >  >
> > > 
> > > >>> I have enabled the full debug already, but I see nothing special. Before exporting any volume the log shows no error, even when I do a showmount (the log is attached, ganesha.log.gz). If I do the same after exporting a volume nfs-ganesha does not even start, complaining for not being able to bind the IPv6 ruota socket, but in fact there is nothing listening on IPv6, so it should not happen:
> > > >>>
> > > >>> tcp6       0      0 :::111                  :::*                    LISTEN      7433/rpcbind
> > > >>> tcp6       0      0 :::2224                 :::*                    LISTEN      9054/ruby
> > > >>> tcp6       0      0 :::22                   :::*                    LISTEN      1248/sshd
> > > >>> udp6       0      0 :::111                  :::*                                7433/rpcbind
> > > >>> udp6       0      0 fe80::8c2:27ff:fef2:123 :::*                                31238/ntpd
> > > >>> udp6       0      0 fe80::230:48ff:fed2:123 :::*                                31238/ntpd
> > > >>> udp6       0      0 fe80::230:48ff:fed2:123 :::*                                31238/ntpd
> > > >>> udp6       0      0 fe80::230:48ff:fed2:123 :::*                                31238/ntpd
> > > >>> udp6       0      0 ::1:123                 :::*                                31238/ntpd
> > > >>> udp6       0      0 fe80::5484:7aff:fef:123 :::*                                31238/ntpd
> > > >>> udp6       0      0 :::123                  :::*                                31238/ntpd
> > > >>> udp6       0      0 :::824                  :::*                                7433/rpcbind
> > > >>>
> > > >>> The error, as shown in the attached ganesha-after-export.log.gz logfile, is the following:
> > > >>>
> > > >>>
> > > >>> 10/06/2015 02:07:47 : epoch 55777fb5 : node2 : ganesha.nfsd-26195[main] Bind_sockets_V6 :DISP :WARN :Cannot bind RQUOTA tcp6 socket, error 98 (Address already in use)
> > > >>> 10/06/2015 02:07:47 : epoch 55777fb5 : node2 : ganesha.nfsd-26195[main] Bind_sockets :DISP :FATAL :Error binding to V6 interface. Cannot continue.
> > > >>> 10/06/2015 02:07:48 : epoch 55777fb5 : node2 : ganesha.nfsd-26195[main] glusterfs_unload :FSAL :DEBUG :FSAL Gluster unloaded
> > > >>>
> > 
> > The above messages indicate that someone tried to restart ganesha. But
> > ganesha failed to come up because RQUOTA port (default is 875) is
> > already in use by an old ganesha instance or some other program holding
> > it. The new instance of ganesha will die, but if you are using systemd,
> > it will try to restart automatically. We have disabled systemd auto
> > restart in our environment as it was causing issues for debugging.
> > 
> > What version of ganesha is this?
> > 
> > Regards, Malahal.
> > 
>