[Gluster-users] Brick Reboot => VMs slowdown, client crashes

[Carl Sirotic]

However,

I must have misunderstood the whole concept of gluster.

In a replica 3, for me, it's completely unacceptable, regardless of the 
options, that all my VMs go down when I reboot one node.

The whole purpose of having a full 3 copy of my data on the fly is 
suposed to be this.

I am in the process of sharding every file.

But even if the healing time would be longer, I would still expect a 
non-sharded replica 3 brick with vm boot disk, to not go down if I 
reboot one of its copy.


I am not very impressed by gluster so far.

Carl

On 2019-08-19 4:15 p.m., Darrell Budic wrote:
> /var/lib/glusterd/groups/virt is a good start for ideas, notably some 
> thread settings and choose-local=off to improve read performance. If 
> you don’t have at least 10 cores on your servers, you may want to 
> lower the recommended shd-max-threads=8 to no more than half your CPU 
> cores to keep healing from swamping out regular work.
>
> It’s also starting to depend on what your backing store and networking 
> setup are, so you’re going to want to test changes and find what works 
> best for your setup.
>
> In addition to the virt group settings, I use these on most of my 
> volumes, SSD or HDD backed, with the default 64M shard size:
>
> performance.io <http://performance.io>-thread-count: 32# seemed good 
> for my system, particularly a ZFS backed volume with lots of spindles
> client.event-threads: 8
> cluster.data-self-heal-algorithm: full# 10G networking, uses more 
> net/less cpu to heal. probably don’t use this for 1G networking?
> performance.stat-prefetch: on
> cluster.read-hash-mode: 3# distribute reads to least loaded server (by 
> read queue depth)
>
> and these two only on my HDD backed volume:
>
> performance.cache-size: 1G
> performance.write-behind-window-size: 64MB
>
> but I suspect these two need another round or six of tuning to tell if 
> they are making a difference.
>
> I use the throughput-performance tuned profile on my servers, so you 
> should be in good shape there.
>
>> On Aug 19, 2019, at 12:22 PM, Guy Boisvert 
>> <guy.boisvert at ingtegration.com 
>> <mailto:guy.boisvert at ingtegration.com>> wrote:
>>
>> On 2019-08-19 12:08 p.m., Darrell Budic wrote:
>>> You also need to make sure your volume is setup properly for best 
>>> performance. Did you apply the gluster virt group to your volumes, 
>>> or at least features.shard = on on your VM volume?
>>
>> That's what we did here:
>>
>>
>> gluster volume set W2K16_Rhenium cluster.quorum-type auto
>> gluster volume set W2K16_Rhenium network.ping-timeout 10
>> gluster volume set W2K16_Rhenium auth.allow \*
>> gluster volume set W2K16_Rhenium group virt
>> gluster volume set W2K16_Rhenium storage.owner-uid 36
>> gluster volume set W2K16_Rhenium storage.owner-gid 36
>> gluster volume set W2K16_Rhenium features.shard on
>> gluster volume set W2K16_Rhenium features.shard-block-size 256MB
>> gluster volume set W2K16_Rhenium cluster.data-self-heal-algorithm full
>> gluster volume set W2K16_Rhenium performance.low-prio-threads 32
>>
>> tuned-adm profile random-io        (a profile i added in CentOS 7)
>>
>>
>> cat /usr/lib/tuned/random-io/tuned.conf
>> ===========================================
>> [main]
>> summary=Optimize for Gluster virtual machine storage
>> include=throughput-performance
>>
>> [sysctl]
>>
>> vm.dirty_ratio = 5
>> vm.dirty_background_ratio = 2
>>
>>
>> Any more optimization to add to this?
>>
>>
>> Guy
>>
>> -- 
>> Guy Boisvert, ing.
>> IngTegration inc.
>> http://www.ingtegration.com
>> https://www.linkedin.com/in/guy-boisvert-8990487
>>
>> AVIS DE CONFIDENTIALITE : ce message peut contenir des
>> renseignements confidentiels appartenant exclusivement a
>> IngTegration Inc. ou a ses filiales. Si vous n'etes pas
>> le destinataire indique ou prevu dans ce  message (ou
>> responsable de livrer ce message a la personne indiquee ou
>> prevue) ou si vous pensez que ce message vous a ete adresse
>> par erreur, vous ne pouvez pas utiliser ou reproduire ce
>> message, ni le livrer a quelqu'un d'autre. Dans ce cas, vous
>> devez le detruire et vous etes prie d'avertir l'expediteur
>> en repondant au courriel.
>>
>> CONFIDENTIALITY NOTICE : Proprietary/Confidential Information
>> belonging to IngTegration Inc. and its affiliates may be
>> contained in this message. If you are not a recipient
>> indicated or intended in this message (or responsible for
>> delivery of this message to such person), or you think for
>> any reason that this message may have been addressed to you
>> in error, you may not use or copy or deliver this message to
>> anyone else. In such case, you should destroy this message
>> and are asked to notify the sender by reply email.
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190823/23e6d6b8/attachment.html>