<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">/var/lib/glusterd/groups/virt is a good start for ideas, notably some thread settings and choose-local=off to improve read performance. If you don’t have at least 10 cores on your servers, you may want to lower the recommended shd-max-threads=8 to no more than half your CPU cores to keep healing from swamping out regular work.<div class=""><br class=""></div><div class="">It’s also starting to depend on what your backing store and networking setup are, so you’re going to want to test changes and find what works best for your setup.</div><div class=""><br class=""></div><div class="">In addition to the virt group settings, I use these on most of my volumes, SSD or HDD backed, with the default 64M shard size:</div><div class=""><br class=""></div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""><a href="http://performance.io" class="">performance.io</a>-thread-count: 32<span class="Apple-tab-span" style="white-space:pre"> </span># seemed good for my system, particularly a ZFS backed volume with lots of spindles</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">client.event-threads: 8<span class="Apple-tab-span" style="white-space:pre"> </span></span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">cluster.data-self-heal-algorithm: full<span class="Apple-tab-span" style="white-space:pre"> </span># 10G networking, uses more net/less cpu to heal. probably don’t use this for 1G networking?</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">performance.stat-prefetch: on</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">cluster.read-hash-mode: 3<span class="Apple-tab-span" style="white-space:pre"> </span># distribute reads to least loaded server (by read queue depth)</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">and these two only on my HDD backed volume:</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">performance.cache-size: 1G</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">performance.write-behind-window-size: 64MB</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">but I suspect these two need another round or six of tuning to tell if they are making a difference.</span></div></span></div></span></div></span></div></span></div><div class=""><br class=""></div><div class="">I use the throughput-performance tuned profile on my servers, so you should be in good shape there.</div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 19, 2019, at 12:22 PM, Guy Boisvert <<a href="mailto:guy.boisvert@ingtegration.com" class="">guy.boisvert@ingtegration.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="">On 2019-08-19 12:08 p.m., Darrell Budic wrote:<br class=""><blockquote type="cite" class="">You also need to make sure your volume is setup properly for best performance. Did you apply the gluster virt group to your volumes, or at least features.shard = on on your VM volume?<br class=""></blockquote><br class="">That's what we did here:<br class=""><br class=""><br class="">gluster volume set W2K16_Rhenium cluster.quorum-type auto<br class="">gluster volume set W2K16_Rhenium network.ping-timeout 10<br class="">gluster volume set W2K16_Rhenium auth.allow \*<br class="">gluster volume set W2K16_Rhenium group virt<br class="">gluster volume set W2K16_Rhenium storage.owner-uid 36<br class="">gluster volume set W2K16_Rhenium storage.owner-gid 36<br class="">gluster volume set W2K16_Rhenium features.shard on<br class="">gluster volume set W2K16_Rhenium features.shard-block-size 256MB<br class="">gluster volume set W2K16_Rhenium cluster.data-self-heal-algorithm full<br class="">gluster volume set W2K16_Rhenium performance.low-prio-threads 32<br class=""><br class="">tuned-adm profile random-io (a profile i added in CentOS 7)<br class=""><br class=""><br class="">cat /usr/lib/tuned/random-io/tuned.conf<br class="">===========================================<br class="">[main]<br class="">summary=Optimize for Gluster virtual machine storage<br class="">include=throughput-performance<br class=""><br class="">[sysctl]<br class=""><br class="">vm.dirty_ratio = 5<br class="">vm.dirty_background_ratio = 2<br class=""><br class=""><br class="">Any more optimization to add to this?<br class=""><br class=""><br class="">Guy<br class=""><br class="">-- <br class="">Guy Boisvert, ing.<br class="">IngTegration inc.<br class=""><a href="http://www.ingtegration.com" class="">http://www.ingtegration.com</a><br class="">https://www.linkedin.com/in/guy-boisvert-8990487<br class=""><br class="">AVIS DE CONFIDENTIALITE : ce message peut contenir des<br class="">renseignements confidentiels appartenant exclusivement a<br class="">IngTegration Inc. ou a ses filiales. Si vous n'etes pas<br class="">le destinataire indique ou prevu dans ce message (ou<br class="">responsable de livrer ce message a la personne indiquee ou<br class="">prevue) ou si vous pensez que ce message vous a ete adresse<br class="">par erreur, vous ne pouvez pas utiliser ou reproduire ce<br class="">message, ni le livrer a quelqu'un d'autre. Dans ce cas, vous<br class="">devez le detruire et vous etes prie d'avertir l'expediteur<br class="">en repondant au courriel.<br class=""><br class="">CONFIDENTIALITY NOTICE : Proprietary/Confidential Information<br class="">belonging to IngTegration Inc. and its affiliates may be<br class="">contained in this message. If you are not a recipient<br class="">indicated or intended in this message (or responsible for<br class="">delivery of this message to such person), or you think for<br class="">any reason that this message may have been addressed to you<br class="">in error, you may not use or copy or deliver this message to<br class="">anyone else. In such case, you should destroy this message<br class="">and are asked to notify the sender by reply email.<br class=""><br class=""></div></div></blockquote></div><br class=""></div></body></html>