[Gluster-users] Understanding Gluster Replication/Distribute

[Dan Mons]

Replies inline:

On 7 February 2014 10:11, Scott Dungan <scott at gps.caltech.edu> wrote:
> I am new to Gluster and I am having a hard time grasping how Gluster
> functions in distribute mode vs. distribute+replication. I am planning on
> having 5 servers, with each server hosting a raid6-backed 36TB brick. For
> simplicity, lets just pretend this is a 40TB brick. Here are my questions:
>
> 1. If I do a distribute configuration only, usable capacity of the Gluster
> volume will be 5x40TB or 200TB?

Using "40TB" as a round number per brick:

distribute (no replicate) would be a single ~200TB GlusterFS volume.

> 2. In this configuration, what would clients see if one of the servers were
> to fail?

Lots of errors.  Typically, every fifth file or directory would be
missing, and you'd see lots of question marks in your "ls -l" output.

> 3. When the server comes back up, what steps would need to be taken to make
> the Gluster volume consistent again?

In a distribute-only setup, there's no redundancy.  So there's no
"consistency" so to speak.  When the missing volume came online, the
files it holds would be available again.

> 4. if I do a distributed replicated (2) volume, will my usable capacity
> become 160TB or 100TB, or perhaps something else entirely?

5 servers is an uneven amount of bricks.  You'd end up with 120TB, but
40TB of that wouldn't be replicated.  A 6th brick would solve that
problem, and you'd have ~120TB in full distribute+replicate(2).

> 5. In this configuration, one server may be removed for maintenance and the
> file system stays consistent?

Theoretically yes.  I try to keep my replicated brick downtime to a
minimum though.  Similar to the ideas behind a RAID mirror, I don't
like running in production on only one copy of something for too long.

-Dan