<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>However,</p>
<p>I must have misunderstood the whole concept of gluster.</p>
<p>In a replica 3, for me, it's completely unacceptable, regardless
of the options, that all my VMs go down when I reboot one node.</p>
<p>The whole purpose of having a full 3 copy of my data on the fly
is suposed to be this.</p>
<p>I am in the process of sharding every file.</p>
<p>But even if the healing time would be longer, I would still
expect a non-sharded replica 3 brick with vm boot disk, to not go
down if I reboot one of its copy.</p>
<p><br>
</p>
<p>I am not very impressed by gluster so far.<br>
</p>
<p>Carl<br>
</p>
<div class="moz-cite-prefix">On 2019-08-19 4:15 p.m., Darrell Budic
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:WM!70b2a24c324753176289e0b250d790e7f5ffa931f81e9072ffcc23c4f6fc1a7199617ab90bbd5e0e5170f02e1339ca54!@filter4.lastspam.com">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
/var/lib/glusterd/groups/virt is a good start for ideas, notably
some thread settings and choose-local=off to improve read
performance. If you don’t have at least 10 cores on your servers,
you may want to lower the recommended shd-max-threads=8 to no more
than half your CPU cores to keep healing from swamping out regular
work.
<div class=""><br class="">
</div>
<div class="">It’s also starting to depend on what your backing
store and networking setup are, so you’re going to want to test
changes and find what works best for your setup.</div>
<div class=""><br class="">
</div>
<div class="">In addition to the virt group settings, I use these
on most of my volumes, SSD or HDD backed, with the default 64M
shard size:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal; line-height:
normal;" class=""><span style="font-variant-ligatures:
no-common-ligatures; background-color: rgba(255, 255, 255,
0);" class=""><a href="http://performance.io" class=""
moz-do-not-send="true">performance.io</a>-thread-count: 32<span class="Apple-tab-span" style="white-space:pre"> </span>#
seemed good for my system, particularly a ZFS backed volume
with lots of spindles</span></div>
<div style="margin: 0px; font-stretch: normal; line-height:
normal;" class=""><span style="font-variant-ligatures:
no-common-ligatures; background-color: rgba(255, 255, 255,
0);" class="">client.event-threads: 8<span class="Apple-tab-span" style="white-space:pre"> </span></span></div>
</div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal; line-height:
normal;" class=""><span style="font-variant-ligatures:
no-common-ligatures; background-color: rgba(255, 255, 255,
0);" class="">cluster.data-self-heal-algorithm: full<span class="Apple-tab-span" style="white-space:pre"> </span>#
10G networking, uses more net/less cpu to heal. probably
don’t use this for 1G networking?</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures: no-common-ligatures;
background-color: rgba(255, 255, 255, 0);" class="">performance.stat-prefetch:
on</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color: rgba(255,
255, 255, 0);" class="">cluster.read-hash-mode: 3<span class="Apple-tab-span" style="white-space:pre"> </span>#
distribute reads to least loaded server (by read
queue depth)</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures; background-color: rgba(255,
255, 255, 0);" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures; background-color: rgba(255,
255, 255, 0);" class="">and these two only on my
HDD backed volume:</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures; background-color: rgba(255,
255, 255, 0);" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color:
rgba(255, 255, 255, 0);" class="">performance.cache-size:
1G</span></div>
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color:
rgba(255, 255, 255, 0);" class="">performance.write-behind-window-size:
64MB</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class=""><br class="">
</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">but I suspect
these two need another round or six of tuning
to tell if they are making a difference.</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
<div class=""><br class="">
</div>
<div class="">I use the throughput-performance tuned profile on my
servers, so you should be in good shape there.</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 19, 2019, at 12:22 PM, Guy Boisvert
<<a href="mailto:guy.boisvert@ingtegration.com"
class="" moz-do-not-send="true">guy.boisvert@ingtegration.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">On 2019-08-19 12:08 p.m., Darrell Budic
wrote:<br class="">
<blockquote type="cite" class="">You also need to make
sure your volume is setup properly for best
performance. Did you apply the gluster virt group to
your volumes, or at least features.shard = on on your
VM volume?<br class="">
</blockquote>
<br class="">
That's what we did here:<br class="">
<br class="">
<br class="">
gluster volume set W2K16_Rhenium cluster.quorum-type
auto<br class="">
gluster volume set W2K16_Rhenium network.ping-timeout 10<br
class="">
gluster volume set W2K16_Rhenium auth.allow \*<br
class="">
gluster volume set W2K16_Rhenium group virt<br class="">
gluster volume set W2K16_Rhenium storage.owner-uid 36<br
class="">
gluster volume set W2K16_Rhenium storage.owner-gid 36<br
class="">
gluster volume set W2K16_Rhenium features.shard on<br
class="">
gluster volume set W2K16_Rhenium
features.shard-block-size 256MB<br class="">
gluster volume set W2K16_Rhenium
cluster.data-self-heal-algorithm full<br class="">
gluster volume set W2K16_Rhenium
performance.low-prio-threads 32<br class="">
<br class="">
tuned-adm profile random-io (a profile i added in
CentOS 7)<br class="">
<br class="">
<br class="">
cat /usr/lib/tuned/random-io/tuned.conf<br class="">
===========================================<br class="">
[main]<br class="">
summary=Optimize for Gluster virtual machine storage<br
class="">
include=throughput-performance<br class="">
<br class="">
[sysctl]<br class="">
<br class="">
vm.dirty_ratio = 5<br class="">
vm.dirty_background_ratio = 2<br class="">
<br class="">
<br class="">
Any more optimization to add to this?<br class="">
<br class="">
<br class="">
Guy<br class="">
<br class="">
-- <br class="">
Guy Boisvert, ing.<br class="">
IngTegration inc.<br class="">
<a href="http://www.ingtegration.com" class=""
moz-do-not-send="true">http://www.ingtegration.com</a><br
class="">
<a class="moz-txt-link-freetext" href="https://www.linkedin.com/in/guy-boisvert-8990487">https://www.linkedin.com/in/guy-boisvert-8990487</a><br
class="">
<br class="">
AVIS DE CONFIDENTIALITE : ce message peut contenir des<br
class="">
renseignements confidentiels appartenant exclusivement a<br
class="">
IngTegration Inc. ou a ses filiales. Si vous n'etes pas<br
class="">
le destinataire indique ou prevu dans ce message (ou<br
class="">
responsable de livrer ce message a la personne indiquee
ou<br class="">
prevue) ou si vous pensez que ce message vous a ete
adresse<br class="">
par erreur, vous ne pouvez pas utiliser ou reproduire ce<br
class="">
message, ni le livrer a quelqu'un d'autre. Dans ce cas,
vous<br class="">
devez le detruire et vous etes prie d'avertir
l'expediteur<br class="">
en repondant au courriel.<br class="">
<br class="">
CONFIDENTIALITY NOTICE : Proprietary/Confidential
Information<br class="">
belonging to IngTegration Inc. and its affiliates may be<br
class="">
contained in this message. If you are not a recipient<br
class="">
indicated or intended in this message (or responsible
for<br class="">
delivery of this message to such person), or you think
for<br class="">
any reason that this message may have been addressed to
you<br class="">
in error, you may not use or copy or deliver this
message to<br class="">
anyone else. In such case, you should destroy this
message<br class="">
and are asked to notify the sender by reply email.<br
class="">
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
</body>
</html>