[Gluster-users] GlusterFS performance questions for Amazon EC2 deployment

[Jeff Darcy]

On 06/30/2010 10:22 AM, Craig Box wrote:
> OK, so this brings me to Plan B.  (Feel free to suggest a plan C if you can.)
> 
> I want to have six nodes, three in each availability zone, replicate a
> Mercurial repository.  Here's some art:
> 
> [gluster c/s] [gluster c/s] | [gluster c/s] [gluster c/s]
>                             |
>            [gluster s]      |      [gluster s]
>               [OCFS 2]      |      [OCFS 2]
>               [ DRBD ] ----------- [ DRBD ]
> 
> DRBD doing the cross-AZ replication, and a three node GlusterFS
> cluster inside each AZ.  That way, any one machine going down should
> still mean all the rest of the nodes can access the files.
> 
> Sound believable?

OCFS2 is a shared-disk filesystem, and in EC2 neither ephemeral storage
nor EBS can be mounted on more than one instance simultaneously.
Therefore, you'd need something to provide a shared-disk abstraction
within an AZ.  DRBD mode can do this, and I think it's even reentrant so
that the devices created this way can themselves be used as components
for the inter-AZ-replication devices, but active/active mode isn't
recommended and I don't think you can connect more than two nodes this
way.  What's really needed, and I'm slightly surprised doesn't already
exist, is a DRBD proxy that can be connected as a destination by several
local DRBD sources, and then preserve request order even across devices
as it becomes a DRBD source and ships those requests to another proxy in
another AZ.  Linbit's proxy doesn't seem to be designed for that
particular purpose.  The considerations for dm-replicator are
essentially the same BTW.

An async/long-distance replication translator has certainly been a
frequent topic of discussion between me, the Gluster folks, and others.
 I have plans to shoot for full N-way active/active replication, but
with that ambition comes complexity and we'll probably see simpler forms
(e.g. two-way active/passive) much earlier.