[Gluster-users] Understanding Gluster Replication/Distribute

Scott Dungan scott at gps.caltech.edu
Fri Feb 7 17:19:29 UTC 2014

Thanks Dan. I think I get it now. One more question:

The size of the Gluster volume we want to create is 150TB. We are either 
going to do a distribute only with 4 nodes or a distribute+repl2 with 8 
nodes (depends on budget). Considering this, do you have any server ram 
recommendations. The starting point is going to be 32GB, but should we 
be thinking of 64 or 128?


On 2/6/2014 7:07 PM, Dan Mons wrote:
> Replies inline:
> On 7 February 2014 10:11, Scott Dungan <scott at gps.caltech.edu> wrote:
>> I am new to Gluster and I am having a hard time grasping how Gluster
>> functions in distribute mode vs. distribute+replication. I am planning on
>> having 5 servers, with each server hosting a raid6-backed 36TB brick. For
>> simplicity, lets just pretend this is a 40TB brick. Here are my questions:
>> 1. If I do a distribute configuration only, usable capacity of the Gluster
>> volume will be 5x40TB or 200TB?
> Using "40TB" as a round number per brick:
> distribute (no replicate) would be a single ~200TB GlusterFS volume.
>> 2. In this configuration, what would clients see if one of the servers were
>> to fail?
> Lots of errors.  Typically, every fifth file or directory would be
> missing, and you'd see lots of question marks in your "ls -l" output.
>> 3. When the server comes back up, what steps would need to be taken to make
>> the Gluster volume consistent again?
> In a distribute-only setup, there's no redundancy.  So there's no
> "consistency" so to speak.  When the missing volume came online, the
> files it holds would be available again.
>> 4. if I do a distributed replicated (2) volume, will my usable capacity
>> become 160TB or 100TB, or perhaps something else entirely?
> 5 servers is an uneven amount of bricks.  You'd end up with 120TB, but
> 40TB of that wouldn't be replicated.  A 6th brick would solve that
> problem, and you'd have ~120TB in full distribute+replicate(2).
>> 5. In this configuration, one server may be removed for maintenance and the
>> file system stays consistent?
> Theoretically yes.  I try to keep my replicated brick downtime to a
> minimum though.  Similar to the ideas behind a RAID mirror, I don't
> like running in production on only one copy of something for too long.
> -Dan


Scott A Dungan
Senior Systems Administrator
Geological and Planetary Sciences
California Institute of Technology
Office: (626) 395-3170
Cell:   (626) 993-4932

More information about the Gluster-users mailing list