[Gluster-users] The continuing story ...

[Anand Avati]

> For me, it does not clear after 3 mins or 3 hours. I restarted the machines
> at midnight, and the first time I tried again was around 1pm the next day
> (13 hours). I easily recognize the symptoms as the /bin/mount remains in the
> process tree. I can't get a strace -p on the /bin/mount process since it is
> frozen. The glusterfsd process is not frozen - the glusterfs process seems
> to be waiting on /bin/mount to complete. The only way to unfreeze the mount
> seems to be to kill -9 /bin/mount (regular kill does not work), at which the
> mount point goes into the disconnected state, and it is recovered using
> unmount / remount. I tried to track down the problem before, but became
> confused, because glusterfs seems to do it's own FUSE mount management
> rather than using the standard (for Linux anyways?) FUSE user space
> libraries. If my memory is correct - it seems like the process is: I run
> mount, the mount runs /sbin/mount.glusterfs, which runs glusterfs, which
> runs /bin/mount with the full options?

This looks like a different issue from what I previously described. If
you are certain that the /bin/mount which was hung was the one which
glusterfs had spawned, then the issue might be something else. The way
fuse based filesystems mount is 2-fold. The first 'mount -t glusterfs'
starts /bin/mount which in turn calls /sbin/mount.glusterfs. This
starts the glusterfs binary, which at the time of initializing the
fuse xlator results in a call to fuse_mount() call of libfuse. libfuse
in-turn does the second phase of mounting by calling mount -t fuse an
in turn /sbin/mount.fuse. I'm trying to think how the three machines
rebooting together can be correlated to the second phase fuse mount to
hang.

> This is where I discovered the other issue where the 'mount
> /gluster/mountpoint' can return before the mount point is completely set up,
> introducing a race where a user can access the mount point and see an error
> or an empty directory before seeing the actual contents. I don't know if
> these are related or separate issues.

Your are right. A fix for this is already under way.

Avati