[Gluster-devel] 2.0.1

[Brent A Nelson]

On Fri, 15 May 2009, Gordan Bobic wrote:

> Brent A Nelson wrote:
>> On Thu, 14 May 2009, Gordan Bobic wrote:
>> 
>>> Gordan Bobic wrote:
>>>> SQLite is still unstable with 2.0.1. Bookmarks in firefox still don't 
>>>> work when /home is on GlusterFS.
>>> 
>>> First access immediately after mounting still fails, too.
>>> 
>> 
>> FYI, I haven't seen the first-access-on-mounting problem since one of the 
>> earlier 2.0.0RCs.  I don't need to first "ls" my filesystems, they just 
>> work from the start.
>
> It still seems to occur, at least I use a script that does something along 
> the following lines:
>
> mount /etc/glusterfs/foo.vol /mnt/foo
> mount --bind /mnt/foo/bar/x /mnt/foo/baz
>
> If it is being done by a script so there is no delay between the two, the 
> mount --bind will fail the first time at least when foo is an AFR volume with 
> the current machine being the only one that is up from the AFR cluster.
>
> The delay required may have reduced since earlier versions, but it sounds 
> like there is still a race somewhere.

Makes sense.  With the old bug, delay didn't matter; first access would 
fail unless you ls'ed the mount.  That bug seems to be fixed, but perhaps 
the new situation is that mount is returning slightly before everything is 
quite ready for use.

>
>> I hope to try a GlusterFS as my home directory soon and will be very 
>> interested to see if I hit any trouble with FireFox bookmarks.
>
> Both this and the first access problem are 100% reproducible on all my 
> systems.
>
>> 2) File writes to a kernel-NFS-reexported GlusterFS are marked dirty by 
>> replicate, such that self-heal is triggered when stat'ing the directory. 
>> The unfs3 user-mode NFS server does not trigger this issue, nor do non-NFS 
>> writes.
>
> Can you elaborate on this? If a file is written via NFS, then it should be 
> synced to other nodes since it wouldn't have been written to all the nodes.
>

The files are written correctly to both halves of a replicate mirror, 
except the extended attributes are marked in such a way as to trigger 
self-heal.  When the self-heal occurs, it messes up the mtimes.  I think 
I've seen a few cases now of this happening without kernel-nfs reexport, 
although it is quite rare; with kernel-nfs, it's extremely common, or 
maybe always occurs for files.

>> 3) replicate self-heal fails to preserve mtime.
>
> Doesn't this completely kill the concept of self-heal? Surely, the primary 
> ways to detect that a file needs healing is to check it's mtime?
>

As noted by one of the developers, versioning status is tracked with 
extended attributes, not mtimes.  This bug just messes up the mtimes on 
part of a replicate mirror, causing incorrect mtimes to be displayed at 
random.  It's annoying, of course, and sometimes mtimes are important. 
It's certainly annoying when you're trying to replicate data from 
elsewhere, and mtimes never quite match up 100%...

>> 4) GlusterFS seems to have trouble creating or renaming files that match 
>> the pattern '..*.*.*'; this is noticed when rsyncing directories containing 
>> dot-files.
>
> Interesting, I hadn't noticed that. Thanks for pointing it out.

Yep, that's a weird one.  The files do seem to get created in the 
underlying bricks, but GlusterFS doesn't seem to see them and can't work 
with or rename them, and the permissions are "---------T".

Thanks,

Brent