[Gluster-devel] Replication Initialization from Existing Directory

[gordan at bobich.net]

On Thu, 24 Apr 2008, Krishna Srinivas wrote:

>>>>  I'm trying to move a large volume of data from local disk to GlusterFS.
>> I
>>>> could just copy it, but copying ~ 1TB of data is slow. So, what I've
>> tried
>>>> to do (with some randomly generated data for a test case) is to specify
>> the
>>>> directory already containing the data as the data source for the
>> underlying
>>>> storage brick.
>>>>
>>>>  I then fire up glusterfsd and glusterfs on the same machine, and I can
>> see
>>>> all the data via the mountpoint.
>>>>
>>>>  On another node, I start glusterfsd and glusterfs, and I can see and
>> read
>>>> the data. But, the data doesn't appear on the underlying data brick on
>> the
>>>> 2nd node after I have done cat * > /dev/null in the mounted directory.
>>>>
>>>>  So it looks like GluserFS isn't causing the data to get copied on reads
>> in
>>>> this scenario.
>>>>
>>>>  Can anyone hazard a guess as to why this might be? I am guessing that
>> it's
>>>> to do with the fact that the xattrs/metaddata have not been initialized
>> by
>>>> glusterfs because the files were added "underneath" rather than via the
>>>> mountpoint. Is there a workaround for this, e.g. by manually setting
>> some
>>>> xattrs on the files (in a hope that this might be faster than copying
>> the
>>>> whole volume)?
>>>>
>>>
>>> Your guess is right, just set xattr "trusted.glusterfs.version" to 3 to
>> the
>>> entire tree structure files/dirs (including the exported directory) and
>> try
>>> find + cat, it should work
>>>
>>
>>  Thanks for that, much appreciated. Would I be correct in assuming that
>> setting this attribute to "3" is actually meant to mean "set it to a higher
>> value on the server than it might be set on the client"?
>
> version of a version less file will be assumed to be "1". So we should set it
> to atleast "2", setting it to "3" is just being paranoid :)

Sure, in a completely uninitialized network of nodes (all nodes without 
xattrs set on ANY file), I can see how that works.

>>  Another question -  if the value of the trusted.glusterfs.version is higher
>> on node2 (for whatever reason, e.g. reusing an old store dump directory)
>> than on node1, where a fresh storage brick is initialized, would that lead
>> to the empty directory on node2 clobbering all the data on node1, because
>> the empty directory has a higher version number? I can see how that would
>> make sense if it is the case, but that means that there are major dangers
>> involved in manually initializing the store. The problem is similar to what
>> happens in a split-brain situation (half of the files simply get silently
>> dropped/replaced) if the same file was modified on both sides.
>
> glusterfs uses two xattrs to decide on which is latest copy, i.e
> trusted.glusterfs.createtime and trusted.glusterfs.version. But
> the split brain situation is still a problem.

I was thinking more along the lines of the case I was testing:
node1 is the one with source files to be replicated:
node1:/gluster-store, trusted.glusterfs.version=3
node1:/gluster-store/*, trusted.glusterfs.version=3

node2 is empty, but it's store was used before:
node2:/gluster-store, trusted.glusterfs.version=18

Would this cause node2 to sync node1 by getting all the files on 
node1 purged?

Or would it merely cause node2 to not get any files sent to it (neither 
metadata for ls, nor (implicitly from lack of metadata) the files 
themselves)?

In other words, is deletion replication implicit, or does it have to be 
explicit? i.e. if a file gets removed while a node is down, and that node 
rejoins, does the file get replicated back to the other nodes, or does the 
deletion get replicated back? Or to put it another way, does the "latest 
wins" strategy apply in the same way to directories regarding content, or 
is there a special case for this to limit possible damage when there is 
node churn?

Thanks.

Gordan