[Gluster-users] design of gluster cluster

[Dave Sherohman]

On Tue, Jun 12, 2018 at 03:04:14PM +1200, Thing wrote:
> What I would like to do I think is a,
> 
> *Distributed-Replicated volume*
> 
> a) have 1 and 2 as raid1
> b) have 4 and 5 as raid1
> c) have 3 and 6 as a raid1
> d) join this as concatenation 2+2+2tb

You probably don't actually want to do that because quorum is handled
separately for each subvolume (bricks 1/2, 4/5, or 3/6), not a single
quorum for the volume as a whole.  (Consider if bricks 1 and 2 both went
down.  You'd still have 4 of 6 bricks running, so whole-volume quorum
would still be met, but the volume can't continue to run normally since
the first subvolume is completely missing.)

In the specific case of replica 2, gluster treats the first brick in
each subvolume as "slightly more than one", so you'd be able to continue
normally if brick 2, 5, or 6 went down, but, if brick 1, 4, or 3 went
down, all files on that subvolume would become read-only.

> I tried to do this and failed as it kept asking for an arbiter, which the
> docs simply dont mention how to do.

https://gluster.readthedocs.io/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/

> So say we have,
> 
> a) have 1 and 2 as raid1 with 3 as the arbiter?
> b) have 4 and 5 as raid 1 with 6 as the arbiter
> c) 3 and 6 as a raid 1 with 5 as the arbiter
> d) join this as concatenation 2+2+2tb

I would recommend finding one or more other servers with small amounts
of unused space and allocating the arbiter bricks there, or carving a
gig or two out of your current bricks for that purpose.  Arbiters only
need about 4k of disk space per file in the subvolume, regardless of the
actual file size (the arbiter only stores metadata), so TB-sized
arbiters would be a huge waste of space, especially if you're only
putting a few very large files (such as VM disk images) on the volume.

As a real-world data point, I'm using basically the setup you're aiming
for - six data bricks plus three arbiters, used to store VM disk images.
My data bricks are 11T each, while my arbiters are 98G.  Disk usage for
the volume is currently at 19%, but all arbiters are under 1% usage (the
largest has 370M used).  Assuming my usage patterns don't change, I
could completely fill my 11T subvolumes and only need about 1.5G in the
corresponding arbiters.

> if so what is the command used to build this?

# gluster volume create my-volume replica 3 arbiter 1 host1:/path/to/brick host2:/path/to/brick arb-host1:/path/to/brick host4:/path/to/brick host5:/path/to/brick arb-host2:/path/to/brick host3:/path/to/brick host6:/path/to/brick arb-host3:/path/to/brick

-- 
Dave Sherohman