[Gluster-devel] Re: Experience with kernel NFS reexport

[Brent A Nelson]

Hopefully, this will be enough of a hint for the developers to be able to 
tell what's going on with the mtime self-heal issue.  I'm guessing that 
Gluster writes via NFS are marked as requiring self-healing, even if 
they're fine.  Self-heal gets triggered and messes up the mtime in 
the process.  So, that may be two different bugs...

Before self-heal, a file has attributes as follows (on both halves of the 
mirror):
# file: disk1/glusterfs/software2/xwin32.bat
trusted.afr.brick4-1a=0sAAAAAQAAAAAAAAAA
trusted.afr.brick4-1b=0sAAAAAQAAAAAAAAAA

The directory has:
trusted.afr.brick4-1a=0sAAAAAAAAAAAAAAAA
trusted.afr.brick4-1b=0sAAAAAAAAAAAAAAAA
trusted.glusterfs.dht=0sAAAAAQAAAAB//////////w==

After the self-heal, there's no change in directory attributes, but the 
file attributes change on both nodes to:

# file: disk1/glusterfs/software2/xwin32.bat
trusted.afr.brick4-1a=0sAAAAAAAAAAAAAAAA
trusted.afr.brick4-1b=0sAAAAAAAAAAAAAAAA

Thanks,

Brent

On Wed, 13 May 2009, Brent A Nelson wrote:

> Further insight: the initial mtimes are correct (and probably are for rsync, 
> as well); the mtimes are getting modified by self-heal.  I'm not sure why 
> self-heal did anything at all, as the mirrors should have been in sync, but 
> it triggers on ls and quickly changes the mtimes on the second half of the 
> mirror.
>
> Thanks,
>
> Brent
>
> On Wed, 13 May 2009, Brent A Nelson wrote:
>
>> I take that back; it doesn't seem to help, at least for the initial rsync. 
>> Shouldn't it, though? If a file on 1/2 of the mirror has a different mtime 
>> than the file on the other half, shouldn't self-heal fix it?
>> 
>> The early indication was that it did seem to help in the case that I 
>> manually removed a file from the second half of the mirror.  With the 
>> metadata-change-log option on, the file self-healed with the correct mtime; 
>> without the metadata-change-log option, it did not.  I only tried it once, 
>> though, so it might have just been coincidence.
>> 
>> I also saw the mtime issue with "cp -a".  It appeared to occur in one brief 
>> burst, and this burst spanned multiple mirrors (a few files created at 
>> 4:46pm, all in the same specific directory, for two different mirrors and 
>> different server nodes, have bad mtimes on the second half of each mirror). 
>> This was far less common than rsync, however.
>> 
>> Thanks,
>> 
>> Brent
>> 
>> On Wed, 13 May 2009, Brent A Nelson wrote:
>> 
>>> Early indications are that setting "option metadata-change-log on" for 
>>> cluster/replicate is a likely workaround for the mtime issue.  I'll start 
>>> over, and see if the issue is truly gone with this option in place...
>>> 
>>> It might be worth considering defaulting this option to "on".
>>> 
>>> Thanks,
>>> 
>>> Brent
>>> 
>>> On Wed, 13 May 2009, Brent A Nelson wrote:
>>> 
>>>> On Wed, 13 May 2009, Brent A Nelson wrote:
>>>> 
>>>>> With regards to the incorrect modification time appearing on some files, 
>>>>> I note the following:
>>>>> 
>>>>> ls -l on Node1 in a mirror:
>>>>> 
>>>>> -r--r--r-- 1 root root 40280 2008-03-17 12:03 
>>>>> /disk1/glusterfs/tftpboot/hardy64/pool/main/t/tasksel/tasksel-data_2.70ubuntu4_all.deb
>>>>> 
>>>>> Node 2 in the same mirror:
>>>>> 
>>>>> -r--r--r-- 1 root root 40280 2009-05-12 19:40 
>>>>> /disk1/glusterfs/tftpboot/hardy64/pool/main/t/tasksel/tasksel-data_2.70ubuntu4_all.deb
>>>>> 
>>>>> It appears that 1/2 of the mirror set the modification time correctly, 
>>>>> but the other half did not.
>>>>> 
>>>> 
>>>> Just a bit of additional info.  It appears that the first half of the 
>>>> mirror has correct mtimes.  The second half of the mirror has wrong 
>>>> mtimes on all files, but directory mtimes are fine.  When you go to view 
>>>> the mtimes on the GlusterFS, sometimes you will get the mtime from one 
>>>> node, sometimes the other, hence the seeming randomness.
>>>> 
>>>> Also, I see that zero-length files have correct mtimes on both halves of 
>>>> a mirror.
>>>> 
>>>> Thanks,
>>>> 
>>>> Brent
>>>> 
>>> 
>> 
>