<html dir="ltr"><head></head><body style="text-align:left; direction:ltr;"><div>I have about 200TB in a gluster replicate only 3-node setup. We stopped using hardware RAID6 after the third drive failed on one array at the same time we replaced the other two and before recovery could complete. 200TB is a mess to resync.</div><div><br></div><div>So now each hard drive is a single entity. We add 1 drive to each node as it's own PV in gluster (with LUKS encryption). Each brick is mounted into the final tree on the client end.</div><div><br></div><div>This way our recover is usually just a single drive to sync. With replica 3, we keep quorum if one brick fails. No RAID cards. Just big, multipath SAS JBOD arrays. The server head on each array is pretty beefy (24 cores, 128GB RAM, 40G IB, 40G Ethernet).</div><div><br></div><div>On Thu, 2019-06-06 at 20:46 +0200, Michael Metz-Martini wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Hi</pre><pre><br></pre><pre>Am 06.06.19 um 18:48 schrieb Eduardo Mayoral:</pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><pre>Your comment actually helps me more than you think, one of the main</pre><pre>doubts I have is whether I go for JOBD with replica 3 or SW RAID 6 with</pre><pre>replica2 + arbitrer. Before reading your email I was leaning more</pre><pre>towards JOBD, as reconstruction of a moderately big RAID 6 with mdadm</pre><pre>can be painful too. Now I see a reconstruct is going to be painful</pre><pre>either way...</pre><pre><br></pre><pre>For the record, the workload I am going to migrate is currently</pre><pre>18,314,445 MB and 34,752,784 inodes (which is not exactly the same as</pre><pre>files, but let's use that for a rough estimate), for an average file</pre><pre>size of about 539 KB per file.</pre><pre><br></pre><pre>Thanks a lot for your time and insights!</pre></blockquote><pre>Currently we're hosting ~200 TB split into about 3.500.000.000 files on</pre><pre>a Distributed-Replicate-2-gluster volume with each brick running on a</pre><pre>hw-raid6 of 8 x 8 TB disks. As we never had a failed drive 'till now I</pre><pre>can't tell you something about recovery times but rebalance is damn slow</pre><pre>with such high number of small files (so should recovery on</pre><pre>jbod-bricks). I think raid-recovery from local disks will be much faster.</pre><pre><br></pre><pre>As our files are nearly 100% readonly and split-brain-issues could be</pre><pre>resolevd more or less "easily" we decided against replica 3 in favor of</pre><pre>hardware raid6 redundancy.</pre><pre><br></pre></blockquote><div><span><pre><pre>-- <br></pre>James P. Kinney III
Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain
http://heretothereideas.blogspot.com/
</pre></span></div></body></html>