[Gluster-devel] Problem with autofs configuration - sometimes mount does not complete fast enough?

[Geoff Kassel]

Hi Mark,
   This sounds a lot like the fast first access bug (Bug 220) that's been an 
issue since at least 2.0.0rc1.

   You might want to add your observations into the comments for this bug - 
http://bugs.gluster.com/cgi-bin/bugzilla3/show_bug.cgi?id=220 

   (The devs seem to prefer handling bugs this way.)

Geoff.

On Mon, 7 Sep 2009, Mark Mielke wrote:
> On 09/06/2009 11:19 PM, Mark Mielke wrote:
> > On 09/06/2009 10:42 PM, Mark Mielke wrote:
> >> This seems to happen about 50% of the time:
> >>
> >> [root at wcarh035 ~]# ls /gluster/data
> >> ls: cannot open directory /gluster/data: No such file or directory
> >> [root at wcarh035 ~]# ls /gluster/data
> >> 00      06.fun  15      23.fun  32      40.fun  47      55.fun  64
> >> 00.fun  07      15.fun  24      32.fun  41      47.fun  56      64.fun
> >>
> >> My current guess is that GlusterFS is saying the mount is complete to
> >> AutoFS before the actual mount operation takes effect. 50% of the
> >> time GlusterFS is able to complete the mount before AutoFS let's the
> >> user continue, and all is well. The other 50% of the time, GlusterFS
> >> does not quite finish the mount, and AutoFS gives the user a broken
> >> directory.
> >>
> >> I might try and prove this by adding a sleep 5 to
> >> /sbin/mount.glusterfs, although I do not consider this a valid
> >> solution, as it just reduces the effect of the race - it does not
> >> eliminate the race.
> >
> > Uhh... Hmm... It already has a "sleep 3", and changing it to "sleep 5"
> > does not reduce the frequency of the problem. Changing it to "sleep
> > 10" also has no effect.
> >
> > Why does it sometimes work and sometimes not?
>
> I note that the fusermount from the FUSE libraries does not seem to have
> the same problem:
>
> $ /stage/linux/fuse-2.7.4/example/fusexmp_fh /tmp/t ; ls /tmp/t
> backup/   boot/  etc/   lib64/       media/  pccyber/  sbin/     stage/
> usr/
> backup2/  db/    home/  lost+found/  mnt/    proc/     selinux/  sys/
> var/
> bin/      dev/   lib/   mail/        opt/    root/     srv/      tmp/
> www/
>
> It works immediately. Compare this to:
>
> [root at wcarh033]~# echo hi >/tmp/t/hi
> [root at wcarh033]~# time /opt/glusterfs/sbin/glusterfs
> --volfile=/etc/glusterfs/gluster-data.vol /tmp/t ; ls /tmp/t ; sleep 1 ;
> ls /tmp/t
> /opt/glusterfs/sbin/glusterfs --volfile=/etc/glusterfs/gluster-data.vol
> /tmp/  0.00s user 0.00s system 113% cpu 0.003 total
> hi
> 00      06.fun  15      23.fun  32      40.fun  47      55.fun  64
> 00.fun  07      15.fun  24      32.fun  41      47.fun  56      64.fun
> 01      07.fun  16      24.fun  33      41.fun  50      56.fun  65
> 01.fun  10      16.fun  25      33.fun  42      50.fun  57      65.fun
> 02      10.fun  17      25.fun  34      42.fun  51      57.fun  66
> 02.fun  11      17.fun  26      34.fun  43      51.fun  60      66.fun
> 03      11.fun  20      26.fun  35      43.fun  52      60.fun  67
> 03.fun  12      20.fun  27      35.fun  44      52.fun  61      67.fun
> 04      12.fun  21      27.fun  36      44.fun  53      61.fun  lost+found
> 04.fun  13      21.fun  30      36.fun  45      53.fun  62
> 05      13.fun  22      30.fun  37      45.fun  54      62.fun
> 05.fun  14      22.fun  31      37.fun  46      54.fun  63
> 06      14.fun  23      31.fun  40      46.fun  55      63.fun
>
> Note that the first 'ls' returns 'hi', and a second later, 'ls' returns
> the glusterfs content.
>
> For fusexmp, it appears to complete the mount before it returns. For
> glusterfs, it seems to complete the mount a short time after it completes.
>
> I think this is where autofs is getting confused, and serving the handle
> to the directory to the client too early. It thinks glusterfs is done
> mounting, and gives the handle to the client, but this handle is broken
> and fails. Glusterfs completes the mount, and a short time later the
> lookups succeed. Adding 'sleep' in mount.glusterfs do not seem to be
> good enough - as 'sleep 1' and 'sleep 20' do not change the frequency.
> The existing 'sleep 3' in /sbin/mount.glusterfs should be completely
> unnecessary. Instead, we should figure out why GlusterFS cannot ensure
> the mount is in place before it returns?
>
> I'm worn out investigating for today - hopefully somebody can help me? :-)
>
> Cheers,
> mark