[Gluster-users] Adding arbiter on a large existing replica 2 set

[Strahil]

Then,

You should expect the arbiter to be setup in less time , but this depends to the amount of files - if you have few large files , sync is faster than 10 mil small files.

Best Regards,
Strahil NikolovOn Oct 21, 2019 16:23, Thorgeir Marthinussen <thorgeir.marthinussen at basefarm.com> wrote:
>
> Hi,
>
> The new cluster is set up with two physical servers with HDDs and a VM backed by an all-flash stretched vSAN.
> The old cluster will be set up the same way.
>
> The main volume that I'm concerned about usually takes about 20-30 minutes to finish the self-heal, the network is 10Gbps.
>
>
>
> Best regards
> -- 
> THORGEIR MARTHINUSSEN
> Senior Systems Consultant
> BASEFARM
>
> -----Original Message-----
> From: Strahil <hunter86_bg at yahoo.com>
> To: Thorgeir <thorgeir.marthinussen at basefarm.com>, gluster-users <gluster-users at gluster.org>
> Subject: Re: [Gluster-users] Adding arbiter on a large existing replica 2 set
> Date: Wed, 16 Oct 2019 21:04:50 +0300
>
> Hi Thorgeir,
>
> Did you try adding an arbiter with SSD brick/bricks ?
>
> SSD/NVMe is the best type of storage for an arbiter - yes , it's more expensive but you will need less disks than a data brick .
>
> Of course , arbiter is only one side of the equasion and the time to heal might depend on your data bricks' IOPS.
>
> How much time does a node in the cluster need to heal after being reboot ?
>
> Best Regards,
> Strahil Nikolov
>
> On Oct 16, 2019 16:37, Thorgeir Marthinussen <thorgeir.marthinussen at basefarm.com> wrote:
>>
>> Hi,
>>
>> We have an old Gluster cluster setup, running a replica 2 across two datacenters, and currently on version 4.1.5
>>
>> I need to add an arbiter to this setup, but I'm concerned about the performance impact of this on the volumes.
>>
>> I recently set up a new cluster, for a different purpose, and decided to test adding an arbiter to the volume after adding in some data.
>> Had a volume with ~435,000 files totaling about 12TB.
>> Adding the arbiter initiated a heal-operation that took almost 3 hours.
>>
>> The older cluster, one of the volumes is about 14TB, but ~45,5 million files.
>>
>> Since arbiter is only concerned about metadata and checksums, I'm concerned about the fact that we have 100 times the amount of files, i.e. 100 times the amount of I/O operations to execute during healing, and possibly 100 times the time which would mean about 12,5 days.
>>
>> Another "issue" is that the 'gluster volume heal <vol-name> info summary' command seems to "count" all the files, so the command can take a very long time to complete.
>> The metrics-scraping script I created for us, with a timeout of 110seconds, fails to complete when a volume has over ~800-900 files unsynced (which happens regularily when taking one cluster-node down for patching).
>>
>>
>> Does anyone have any experience with adding arbiter afterwards, perf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191021/6e07d9eb/attachment.html>