[Gluster-users] Fwd: New GlusterFS deployment, doubts on 1 brick per host vs 1 brick per drive.

[Miguel Mascarenhas Filipe]

Hello all,

I'm setting up GlusterFS on 2 hw w/ same configuration, 8  hdds. This
deployment will grow later on.

I'm undecided between these different configurations and am seeing comments
or advice from more experienced users of GlusterFS.

Here is the summary of 3 options:
1. 1 brick per host, Gluster "distributed" volumes, internal redundancy at
brick level
2. 1 brick per drive, Gluster "distributed replicated" volumes, no internal
redundancy
3. 1 brick per host, Gluster "distributed replicated" volumes, no internal
redundancy

I don't know how the performance of these different configurations would
compare. The workload is HPC and ML-training, data-heavy, metadata-light.


# 1 brick per host, simplified cluster management, higher blast-radius

having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS
raid10 of 8 hdd.
Pros:
* I know ZFS raid10 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster in "distributed" mode, no replication (is this a pro?)
* don't need to worry about GlusterFS performance with "distributed
replicated"
* future hosts can have raidz2 + hot spares or whatnot
Cons:
* large blast radius, if a zfs volume goes bad or node goes bad, I lose
data.
* not using "distributed replicated" (is this a con?)
* I can't use hosts without internal redundancy later on?


# 1 brick per hard disk, fine grained device management on Gluster, smaller
blast-radius.

Having 1 brick per drive (/data/bricks/hddN for 1 to X drives on box), each
brick would still use ZFS.
Pros:
* 1 drive blast radius, the ideal.
* GlusterFS w/ distributed replicated
* no complicated host-fault management or runbook, I can use hosts with low
availability
Cons:
* distributed replicated performance vs zfs raid10
* managing on gluster at the disk level can be more time consuming
* managing disk spares and replacements w/ gluster


# 1 brick per host, coarse grained brick management on Gluster, Gluster for
replication.

having 1 brick per host (/data/bricks/hdd0) where each brick is a ZFS raid0
of 8 hdd.

Pros:
* I know ZFS raid0 performs very well.
* simpler management of Gluster at the Host-brick level.
* using Gluster for redundancy (is this a pro?)
* system can use lower redundancy hosts
Cons:
* large blast radius, replicating a whole host takes longer.

Any comments or advice is welcome


Thanks in advance,

-- Miguel Filipe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200909/225c99b3/attachment.html>