[Gluster-users] configuration questions & advice

[Anand Babu Periasamy]

On Mon, Oct 10, 2011 at 8:26 PM, Miles Fidelman
<mfidelman at meetinghouse.net> wrote:
> Hi,
>
> I have a small cluster that I use to host a collection of Xen virtual
> machines.  I just expanded from 2 nodes to 4 nodes and am looking for some
> advice re. configuring a storage subsystem.
>
> The current (2-node) configuration is simple:
> - 4 disks per node
> - md-based raid
> - lvm
> - DRBD to replicate (some) volumes between the two nodes
> - (some) VMs set up for auto-failover on node failure (pacemaker, etc.)
>
> In moving to 4 nodes, I'd like to add some flexibility to move VMs across
> all 4 nodes, but... that requires using something other than DRBD to
> replicate volumes.  I'm thinking of something with the following
> characteristics:
>
> - 4-node storage cluster (4 drives per node, total of 16 drives in the
> storage pool)
> - 4-node VM cluster
> - using the SAME 4 nodes for both
> - note: I've got 4 gigE ports to play with on each box (plan on using 2 for
> outside access, 2 for storage/heartbeat networking)
>
> GlusterFS stands out as the package that seems most capable of supporting
> this (if we were using KVM, I'd probably look at Sheepdog as well).
>
> So... a few questions:
>
> - it looks like running replicated volumes, across 4 nodes, will provide for
> redundancy and support migration/failover (am I right in this? or should I
> be looking at running RAID on the individual nodes as well?)

If you create a volume with replica count = 2, it creates a
distributed replicated volume. (Imagine intelligent RAID-10).
You may choose to use disk level RAID too as second level of
protection. It is a small investment for added reliability.

> - what kind of performance hit is involved in replicated volumes?

Synchronous replication does take a hit on performance. It treats
writes as a transaction across N nodes. Hit depends on application to
application.

> - is there anything more efficient in disk use (i.e., mirroring 4 copies
> eats up lots of disk, is there anything equivalent to RAID 5/6 that is a
> little more efficient while maintaining redundancy?)

Just create distributed-mirror with replica count = 2.  You can also
write a script to automatically replace-brick to a spare disk space in
case of node failures. (This way, system will re-build itself, if a
mirror member does not come back on time).

> - am I missing anything (either re. GlusterFS or other alternatives)

Ceph (in development), Sheepdog (KVM specific) are two other projects.

> Thanks very much for any suggestions and advice.
>
> Miles Fidelman
>
> --
> In theory, there is no difference between theory and practice.
> In<fnord>  practice, there is.   .... Yogi Berra
>

-- 
Anand Babu Periasamy
Blog [ http://www.unlocksmith.org ]
Twitter [ http://twitter.com/abperiasamy ]

Imagination is more important than knowledge --Albert Einstein