[Gluster-devel] mount.glusterfs health check troubles - help appreciated

[Niels de Vos]

On Fri, May 06, 2016 at 04:04:02PM +0530, Kaushal M wrote:
> I'm currently trying to straighten out the encrypted transport
> (SSL/TLS socket) code, and make it more robust, and work well with
> IPv6 in particular [1]. When testing the changes, the mount.glusterfs
> script cause some troubles.
> 
> The mount script tries to check if the mount is online by performing a
> stat on the mount point after the glusterfs command returns, and
> umounts if the stat fails. This is a check is racey and doesn't always
> do the right thing.
> 
> The check is racey because it could be run before the client
> translators have been able to connect to the bricks. The following
> sequence of events happen when the mount happens, which help explain
> the race.
> 
> - mount script runs the glusterfs command
> - mount process fetches the volfile
> - mount process initalizes the graph. The client xlator is also
> initialized now, but the connections aren't started.
> - mount process sends a PARENT_UP event to the graph. The client now
> begins the connection process (portmap first, followed by connecting
> to the brick). It's not guaranteed yet if the connection happened.
> - mount process returns
> - mount script does a stat on mount point to check health
> 
> In an environment (like the on I'm testing in) the connection couldn't
> be completed by the time the health check is done. In my environment,
> the client connection sequence is as follows,
> - the portmap connection is started
>  - the first address returned for the hostname is a IPv6 address. With
> the IPv6 change that was merged recently name lookups are done with
> AF_UNSPEC, which return IPv6. My envrionment returns v6 addresses
> first for getaddrinfo calls (which I think is the default for a lot of
> environments)
>  - the connection fails as glusterd doesn't listen on IPv6 addresses
> (it listens on 0.0.0.0 which v4 only)
>  - a reconnection is made with the next address. This takes a while
> because of the encrypted transports.
>  - portmap query is done after connection is established and the port
> is obtained
> - the client xlator now reconnects to the obtained port.
>  - (same above cycle of connection/reconnection happens)
> - once connection is established, handshakes are done
> - CHILD_UP event is sent
> 
> After this point the client xlator becomes usable.
> 
> But this is not reached before the mount script does the health check
> in my environment. So the mount ends up being terminated.
> 
> Now the simplest solution would be to sleep for some time before doing
> the check to give the xlators time to get ready. But this is
> non-deterministic and isn't something I'm very fond of.
> 
> This turning out to be problematic in my very simple environment, and
> I think it's gonna be a bigger problem in larger more complex
> environments. My environment is,
> - single node
> - single brick volume
> - client is the same node
> - IO transport encryption is on
> - Management transport encryption is on
> - IPv6 enabled in kernel, no actual IPv6 network is in place
> (disabling IPv6 in kernel causes the problem to stop, but I want to
> test with IPv6)
> 
> Does anyone else have ideas on how to fix this? (For now I've disabled
> this check in the script).

I would argue that CHILD_UP is not sufficient for the glusterfs binary
to daemonize/exit. Instead of doing the stat in the mount.glusterfs
script, it would probably be better to move this to the fuse part of the
client xlator-stack. Once the lookup on the mountpoint is done, a valid
return value can be given. This can then be success/fail, and the
mount.glusterfs script can easily check that.

The --verbose mount option (http://review.gluster.org/11469) that
Prasanna is adding would be able to use this as well. Withe --verbose
the process (like mount.nfs) should only output the messages on the
console until the mounting is complete (until fuse-bridge finished the
lookup).

To me, the current 'stat' in the mount.glusterfs script looks more like
a hack than a serious solution.

HTH,
Niels
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160507/ca5cbf97/attachment.sig>