[Gluster-devel] AFR Replication

[gordan at bobich.net]

On Fri, 18 Apr 2008, Christopher Hawkins wrote:

> See:   http://www.gluster.org/docs/index.php/Understanding_AFR_Translator
>
> At the bottom of the page are examples to initiate the sync. To clarify on
> this point and some of your other questions in the splitbrain thread:
>
> Automatic re-sync due to one server being down is not available yet, but is
> coming in the next release with an HA translator. For now you can do a total
> re-sync manually by the method listed above, or allow the cluster re-sync
> itself over time because accessing a file for a read or write will cause
> that file to be synced.

I'm aware of the "read 1 byte/line from a file to sync it" approach. The 
problem I am seeing, however, is that I cannot see files that were created 
on node2 before node1 was started. If I cannot see them, I cannot read 
them to sync them.

I am assuming that this is due to the fact that my underlying FS (Reiser4) 
is having issues with xattrs.

> You don't have to AFR from the client side, but you can. You can also do it
> on the server side, or even both. Part of the beauty of glusterfs is the
> simple building blocks - you can set it up any number of ways. Personally I
> don't think n-fold increases in client bandwidth for mirroring is all that
> bad. How many "mirrors" do you really need??  :-)

Fair. How do I configure server-server mirroring?

> The server AFR's to other servers, then unifies the AFR'd volumes, then
> exports them. The clients mount the export from any given server using round
> robin dns or something similar (probably will be deprecated once the HA
> translator is available).

You mean, have servers AFR as clients, then re-export the AFR-ed volume 
again? GlusterFS on top of GlusterFS?

> That way the client needs only N*1 bandwidth (but
> the servers need N* (num of AFR's)). So if you only need to keep 2x copies
> of your data, you never need more than 2x the bandwidth. And no matter what
> cluster filesystem you use, I can't think of a way to get 2x the files
> without 2x the writes.

Sure, I accept that. I was just asking if there was a way to make the 
additional writes server-side, because servers are few and clients are 
many, so n* the server bandwdth will generally be smaller than 
server*client bandwidth.

> There is no fencing and no quorum - the cluster is essentially stateless,
> which is really great because if you build it right then you can't really
> have a situation where split brain is possible (ok, VERY, very unlikely).

I can see that it's less of an issue than block-level split-brain, because 
this would at most lead to the odd file getting corrupted, whereas 
block-level split-brain would destroy the entire FS very quickly.

> All clients connect on the same port, so if you AFR on the client side, say,
> then it's tough to imagine how one client would be able to write to a server
> while another client would think it was down, and yet would still have
> access to another server on the same network and could write to it. Of
> course if you don't consider these issues at build time, it is possible to
> set yourself up for disaster in certain situations. But that's the case with
> anything cluster related... All in all I think it's a tremendous filesystem
> tool.

I agree, but I don't think split-brain conditions are as few or as 
preventtable as you are implying. Whenever there is more than 1 server, 
split-brain is possible. Especially if, for example, you want 2 mirrored 
servers and each is the client of the mirrored cluster pair. If the 
connection between the servers fails, each server would continue being 
able to see it's own mirrored copy and continue working, thus causing a 
split-brain. File-systems like GFS implement quorum and fencing to prevent 
this situation.

So, although a split-brain is less terminal than with GFS (file corruption 
rather than file system corruption), it is still possible.

Gordan