[Gluster-users] Advice for setup: SW RAID 6 vs JBOD

[Jim Kinney]

I have about 200TB in a gluster replicate only 3-node setup. We stopped
using hardware RAID6 after the third drive failed on one array at the
same time we replaced the other two and before recovery could complete.
200TB is a mess to resync.
So now each hard drive is a single entity. We add 1 drive to each node
as it's own PV in gluster (with LUKS encryption). Each brick is mounted
into the final tree on the client end.
This way our recover is usually just a single drive to sync. With
replica 3, we keep quorum if one brick fails. No RAID cards. Just big,
multipath SAS JBOD arrays. The server head on each array is pretty
beefy (24 cores, 128GB RAM, 40G IB, 40G Ethernet).
On Thu, 2019-06-06 at 20:46 +0200, Michael Metz-Martini wrote:
> Hi
> Am 06.06.19 um 18:48 schrieb Eduardo Mayoral:
> > Your comment actually helps me more than you think, one of the
> > maindoubts I have is whether I go for JOBD with replica 3 or SW
> > RAID 6 withreplica2 + arbitrer. Before reading your email I was
> > leaning moretowards JOBD, as reconstruction of a moderately big
> > RAID 6 with mdadmcan be painful too. Now I see a reconstruct is
> > going to be painfuleither way...
> > For the record, the workload I am going to migrate is
> > currently18,314,445 MB and 34,752,784 inodes (which is not exactly
> > the same asfiles, but let's use that for a rough estimate), for an
> > average filesize of about 539 KB per file.
> > Thanks a lot for your time and insights!
> Currently we're hosting ~200 TB split into about 3.500.000.000 files
> ona Distributed-Replicate-2-gluster volume with each brick running on
> ahw-raid6 of 8 x 8 TB disks. As we never had a failed drive 'till now
> Ican't tell you something about recovery times but rebalance is damn
> slowwith such high number of small files (so should recovery onjbod-
> bricks). I think raid-recovery from local disks will be much faster.
> As our files are nearly 100% readonly and split-brain-issues could
> beresolevd more or less "easily" we decided against replica 3 in
> favor ofhardware raid6 redundancy.
-- 
James P. Kinney III

Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain

http://heretothereideas.blogspot.com/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190606/68b8a143/attachment.html>