[Gluster-devel] Re; Load balancing ...

[Gordan Bobic]

Martin Fick wrote:

> May I suggest an alternate approach?  The rsync model
> seems like a nice one when you have no idea what the
> changes are, but with the glusterfs AFR it is possible
> to keep track of the changes.  What about adding a
> journaling volume option to the AFR translator?

Sounds like you are effectively describing an extent based volume, very 
similar to what DRBD does to limit the amount of sync required.

> So if changes cannot be written to Sub B they would
> be recorded in Journal A.  When B comes back up and
> AFR notices a mismatch between a file on Sub A and Sub
> B and would normally query Sub A for the file
> contents, it could query Journal A first to see if the
> changes to the file are stored there.  If so, Journal
> A could reply with just the changes instead of the
> whole file and AFR can then apply the changes to Sub
> B.

Splitbrain handling of this would be impossible, and one version would 
always have to win. But other than that, I can see that would work.

> The journal volume would not actually be required and
> would be space limited, it would simply drop changes
> that it can no longer keep track of.  If the journal
> does not have the change logged, everything would
> proceed as it does today, the subvolume would be
> queried for the whole file.  This would be a little
> like the DRBD model, but more inline with the gluster
> way of doing things.  It would be better than what
> DRBD does since it would be more granular.  When space
> for changes runs out, whole files might have to be
> synced, but not necessarily the whole filessytem!

I think having an rsync type syncing algorithm that can operate on the 
whole file would be more flexible and potentially provide enough of an 
improvement to make the complication of adding journals/extents not 
worthwhile.

> I realize that this a major enhancement, and would be
> a lot of work, but then again, so probably would the
> rsync model implementation, would it not?

I haven't looked at the GlusterFS code (yet), but I would imagine that 
implementing rsync-like file sync would be _much_ less work than 
implementing extents/journals/undo logs.

> The
> advantage here is that consistency would be assured.

That is arguably fairly academic. Just use the rolling hash for rsync 
that is big enough that the probability of a false negative in the 
hashed block is around the same as the probability of a media error.

> The tradeoff between the journal and the rsync model
> is one of disk space for the journal versus CPU time
> for the rsync model.  Certainly both could be
> implemented, the journal could be queried first, and
> if that fails, use the rsync method!
> Thoughts?

In the ideal world - yes. In practice, I think that just adding rsync 
capability for partial syncs would give most of the benefits for 
relatively little effort in terms of implementation.

Gordan