[Gluster-users] Questions on Replication and Design

Brian Smith brs at usf.edu
Mon Jul 18 19:21:16 UTC 2011

Hi, all,

We're looking to replace our DRBD/Ext4/NFS file storage configuration,
using RHEL cluster w/ GlusterFS 3.2.2 (or greater, depending on the
timeline).  Currently, our configuration includes 2 four node clusters
at different sites with

1. 8 @ 1.5TB LVM LVs on top of an HP-P2000 storage array
2. Each LV is mirrored to an identical LV on a cluster at a remote site
using DRBD
3. Each LV/DRBD is an Ext4 volume and an NFS mount-point
4. DRBD is started as primary in one site and secondary in the remote
site and can be switched easily in the event of a failure.
5. RHEL cluster w/ some patches runs DRBD, floating IP, ext4, NFS as a
service for each of the 8 mounts.

We went this route because it was the only way to get POSIX-ACL support
and a working Quota implementation w/ replication of a large-ish volume
without spending incredible amounts of money.  With GlusterFS 3.2.2,
these are both supported features.  My proposed layout for the new
configuration would look like so:

1. 8 @ 1.5TB LVM LVs on top of an HP-P2000 storage array
2. Each LV is an Ext4 FS
3. RHEL cluster runs a glusterd instance, floating IP and, ext4 mount
for each of the 8 LVs.
4. Each of the 8 LVs is configured with a replicated pair in our remote
site while they distribute across the local site.  For instance:

site1-node1:                     site2-node1:
  gluster1:/glusterfs   <------>   gluster9:/glusterfs
  gluster2:/glusterfs   <------>   gluster10:/glusterfs
  gluster3:/glusterfs   <------>   gluster11:/glusterfs
  gluster4:/glusterfs   <------>   gluster12:/glusterfs
  gluster5:/glusterfs   <------>   gluster13:/glusterfs
  gluster6:/glusterfs   <------>   gluster14:/glusterfs
  gluster7:/glusterfs   <------>   gluster15:/glusterfs
  gluster8:/glusterfs   <------>   gluster16:/glusterfs
                 |       |       |      |
              client  client   client  client

We will still use RHEL cluster to facilitate HA and failover of the
glusterd/ip/fs instances on each cluster site.

What say the experts about this approach and what caveats/issues should
I be looking out for?  I'll be building a test environment, but was
wondering, before I start, whether this is a supportable configuration
in the event we decide to get support, etc.

Many thanks in advance!


Brian Smith
Senior Systems Administrator
IT Research Computing, University of South Florida
4202 E. Fowler Ave. ENB308
Office Phone: +1 813 974-1467
Organization URL: http://rc.usf.edu

More information about the Gluster-users mailing list