[Gluster-users] Questions on Replication and Design

[Brian Smith]

Hi, all,

We're looking to replace our DRBD/Ext4/NFS file storage configuration,
using RHEL cluster w/ GlusterFS 3.2.2 (or greater, depending on the
timeline).  Currently, our configuration includes 2 four node clusters
at different sites with

1. 8 @ 1.5TB LVM LVs on top of an HP-P2000 storage array
2. Each LV is mirrored to an identical LV on a cluster at a remote site
using DRBD
3. Each LV/DRBD is an Ext4 volume and an NFS mount-point
4. DRBD is started as primary in one site and secondary in the remote
site and can be switched easily in the event of a failure.
5. RHEL cluster w/ some patches runs DRBD, floating IP, ext4, NFS as a
service for each of the 8 mounts.

We went this route because it was the only way to get POSIX-ACL support
and a working Quota implementation w/ replication of a large-ish volume
without spending incredible amounts of money.  With GlusterFS 3.2.2,
these are both supported features.  My proposed layout for the new
configuration would look like so:

1. 8 @ 1.5TB LVM LVs on top of an HP-P2000 storage array
2. Each LV is an Ext4 FS
3. RHEL cluster runs a glusterd instance, floating IP and, ext4 mount
for each of the 8 LVs.
4. Each of the 8 LVs is configured with a replicated pair in our remote
site while they distribute across the local site.  For instance:

site1-node1:                     site2-node1:
  gluster1:/glusterfs   <------>   gluster9:/glusterfs
  gluster2:/glusterfs   <------>   gluster10:/glusterfs
site1-node2:
  gluster3:/glusterfs   <------>   gluster11:/glusterfs
  gluster4:/glusterfs   <------>   gluster12:/glusterfs
site1-node3:
  gluster5:/glusterfs   <------>   gluster13:/glusterfs
  gluster6:/glusterfs   <------>   gluster14:/glusterfs
site1-node4:
  gluster7:/glusterfs   <------>   gluster15:/glusterfs
  gluster8:/glusterfs   <------>   gluster16:/glusterfs
                        ||||||||
                        ||||||||
                       Distributed
                        ||||||||
                        ||||||||
                  ----------------------
                 |       |       |      |
              client  client   client  client

We will still use RHEL cluster to facilitate HA and failover of the
glusterd/ip/fs instances on each cluster site.

What say the experts about this approach and what caveats/issues should
I be looking out for?  I'll be building a test environment, but was
wondering, before I start, whether this is a supportable configuration
in the event we decide to get support, etc.

Many thanks in advance!

-Brian

-- 
Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB308
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu