<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="text-align:left; direction:ltr;">
<div>Hi,</div>
<div><br>
</div>
<div>The new cluster is set up with two physical servers with HDDs and a VM backed by an all-flash stretched vSAN.</div>
<div>The old cluster will be set up the same way.</div>
<div><br>
</div>
<div>The main volume that I'm concerned about usually takes about 20-30 minutes to finish the self-heal, the network is 10Gbps.</div>
<div><span>
<pre><br></pre>
<br>
Best regards<br>
-- <br>
<b>THORGEIR MARTHINUSSEN</b><br>
<div>Senior Systems Consultant</div>
<b>BASEFARM</b></span></div>
<div><br>
</div>
<div>-----Original Message-----</div>
<div><b>From</b>: Strahil <<a href="mailto:Strahil%20%3chunter86_bg@yahoo.com%3e">hunter86_bg@yahoo.com</a>></div>
<div><b>To</b>: Thorgeir <<a href="mailto:Thorgeir%20%3cthorgeir.marthinussen@basefarm.com%3e">thorgeir.marthinussen@basefarm.com</a>>, gluster-users <<a href="mailto:gluster-users%20%3cgluster-users@gluster.org%3e">gluster-users@gluster.org</a>></div>
<div><b>Subject</b>: Re: [Gluster-users] Adding arbiter on a large existing replica 2 set</div>
<div><b>Date</b>: Wed, 16 Oct 2019 21:04:50 +0300</div>
<div><br>
</div>
<!-- text/html -->
<p dir="ltr">Hi Thorgeir,</p>
<p dir="ltr">Did you try adding an arbiter with SSD brick/bricks ?</p>
<p dir="ltr">SSD/NVMe is the best type of storage for an arbiter - yes , it's more expensive but you will need less disks than a data brick .</p>
<p dir="ltr">Of course , arbiter is only one side of the equasion and the time to heal might depend on your data bricks' IOPS.</p>
<p dir="ltr">How much time does a node in the cluster need to heal after being reboot ?</p>
<p dir="ltr">Best Regards,<br>
Strahil Nikolov</p>
<div class="quote">On Oct 16, 2019 16:37, Thorgeir Marthinussen <thorgeir.marthinussen@basefarm.com> wrote:<br type="attribution">
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div style="text-align:left;direction:ltr">
<div>Hi,</div>
<div><br>
</div>
<div>We have an old Gluster cluster setup, running a replica 2 across two datacenters, and currently on version 4.1.5</div>
<div><br>
</div>
<div>I need to add an arbiter to this setup, but I'm concerned about the performance impact of this on the volumes.</div>
<div><br>
</div>
<div>I recently set up a new cluster, for a different purpose, and decided to test adding an arbiter to the volume after adding in some data.</div>
<div>Had a volume with ~435,000 files totaling about 12TB.</div>
<div>Adding the arbiter initiated a heal-operation that took almost 3 hours.</div>
<div><br>
</div>
<div>The older cluster, one of the volumes is about 14TB, but ~45,5 million files.</div>
<div><br>
</div>
<div>Since arbiter is only concerned about metadata and checksums, I'm concerned about the fact that we have 100 times the amount of files, i.e. 100 times the amount of I/O operations to execute during healing, and possibly 100 times the time which would mean
about 12,5 days.</div>
<div><br>
</div>
<div>Another "issue" is that the 'gluster volume heal <vol-name> info summary' command seems to "count" all the files, so the command can take a very long time to complete.</div>
<div>The metrics-scraping script I created for us, with a timeout of 110seconds, fails to complete when a volume has over ~800-900 files unsynced (which happens regularily when taking one cluster-node down for patching).</div>
<div><br>
</div>
<div><br>
</div>
<div>Does anyone have any experience with adding arbiter afterwards, performance impact, time to heal, etc.</div>
<div>Also other ways to get the status on healing.</div>
<div><br>
</div>
<div>Any advice would be appreciated.</div>
<div><br>
<br>
Best regards<br>
-- <br>
<b>THORGEIR MARTHINUSSEN</b><br>
<div>Senior Systems Consultant</div>
<b>BASEFARM</b></div>
</div>
</blockquote>
</div>
</body>
</html>