<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Okay,</p>
<p>so it means, at least I am not getting the expected behavior and
there is hope.</p>
<p>I put the quorum settings that I was told a couple of emails ago.</p>
<p>After applying virt group, they are</p>
<p>cluster.quorum-type
auto <br>
cluster.quorum-count
(null) <br>
cluster.server-quorum-type
server <br>
cluster.server-quorum-ratio
0 <br>
cluster.quorum-reads
no <br>
<br>
</p>
<p>Also,</p>
<p>I just put the ping timeout to 5 seconds now.</p>
<p><br>
Carl<br>
</p>
<div class="moz-cite-prefix">On 2019-08-23 5:45 p.m., Ingo Fischer
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:WM!70be0689af39e9a9c31a03ebbb8526a85e48659c6f910db0e14f7159e759eff604955b78624896a3e47e88cb0eb836d0!@filter1.lastspam.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
Hi Carl,
<div><br>
</div>
<div>In my understanding and experience (I have a replica 3 System
running too) this should not happen. Can you tell your client
and server quorum settings?<br>
<br>
<div dir="ltr">Ingo</div>
<div dir="ltr"><br>
Am 23.08.2019 um 15:53 schrieb Carl Sirotic <<a
href="mailto:csirotic@evoqarchitecture.com"
moz-do-not-send="true">csirotic@evoqarchitecture.com</a>>:<br>
<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<p>However,</p>
<p>I must have misunderstood the whole concept of gluster.</p>
<p>In a replica 3, for me, it's completely unacceptable,
regardless of the options, that all my VMs go down when I
reboot one node.</p>
<p>The whole purpose of having a full 3 copy of my data on
the fly is suposed to be this.</p>
<p>I am in the process of sharding every file.</p>
<p>But even if the healing time would be longer, I would
still expect a non-sharded replica 3 brick with vm boot
disk, to not go down if I reboot one of its copy.</p>
<p><br>
</p>
<p>I am not very impressed by gluster so far.<br>
</p>
<p>Carl<br>
</p>
<div class="moz-cite-prefix">On 2019-08-19 4:15 p.m.,
Darrell Budic wrote:<br>
</div>
<blockquote type="cite"
cite="mid:WM!70b2a24c324753176289e0b250d790e7f5ffa931f81e9072ffcc23c4f6fc1a7199617ab90bbd5e0e5170f02e1339ca54!@filter4.lastspam.com">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
/var/lib/glusterd/groups/virt is a good start for ideas,
notably some thread settings and choose-local=off to
improve read performance. If you don’t have at least 10
cores on your servers, you may want to lower the
recommended shd-max-threads=8 to no more than half your
CPU cores to keep healing from swamping out regular work.
<div class=""><br class="">
</div>
<div class="">It’s also starting to depend on what your
backing store and networking setup are, so you’re going
to want to test changes and find what works best for
your setup.</div>
<div class=""><br class="">
</div>
<div class="">In addition to the virt group settings, I
use these on most of my volumes, SSD or HDD backed, with
the default 64M shard size:</div>
<div class=""><br class="">
</div>
<div class="">
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures: no-common-ligatures;
background-color: rgba(255, 255, 255, 0);" class=""><a
href="http://performance.io" class=""
moz-do-not-send="true">performance.io</a>-thread-count:
32<span class="Apple-tab-span" style="white-space:pre"> </span>#
seemed good for my system, particularly a ZFS backed
volume with lots of spindles</span></div>
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures: no-common-ligatures;
background-color: rgba(255, 255, 255, 0);" class="">client.event-threads:
8<span class="Apple-tab-span" style="white-space:pre"> </span></span></div>
</div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color: rgba(255,
255, 255, 0);" class="">cluster.data-self-heal-algorithm:
full<span class="Apple-tab-span" style="white-space:pre"> </span>#
10G networking, uses more net/less cpu to heal.
probably don’t use this for 1G networking?</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color:
rgba(255, 255, 255, 0);" class="">performance.stat-prefetch:
on</span></div>
<div class=""><span style="font-variant-ligatures:
no-common-ligatures" class="">
<div style="margin: 0px; font-stretch: normal;
line-height: normal;" class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color:
rgba(255, 255, 255, 0);" class="">cluster.read-hash-mode:
3<span class="Apple-tab-span" style="white-space:pre"> </span>#
distribute reads to least loaded server
(by read queue depth)</span></div>
<div class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color:
rgba(255, 255, 255, 0);" class=""><br
class="">
</span></div>
<div class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color:
rgba(255, 255, 255, 0);" class="">and
these two only on my HDD backed volume:</span></div>
<div class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color:
rgba(255, 255, 255, 0);" class=""><br
class="">
</span></div>
<div class=""><span
style="font-variant-ligatures:
no-common-ligatures" class="">
<div style="margin: 0px; font-stretch:
normal; line-height: normal;" class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color:
rgba(255, 255, 255, 0);" class="">performance.cache-size:
1G</span></div>
<div style="margin: 0px; font-stretch:
normal; line-height: normal;" class=""><span
style="font-variant-ligatures:
no-common-ligatures; background-color:
rgba(255, 255, 255, 0);" class="">performance.write-behind-window-size:
64MB</span></div>
<div class=""><span
style="font-variant-ligatures:
no-common-ligatures" class=""><br
class="">
</span></div>
<div class=""><span
style="font-variant-ligatures:
no-common-ligatures" class="">but I
suspect these two need another round
or six of tuning to tell if they are
making a difference.</span></div>
</span></div>
</span></div>
</span></div>
</span></div>
<div class=""><br class="">
</div>
<div class="">I use the throughput-performance tuned
profile on my servers, so you should be in good shape
there.</div>
<div class="">
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Aug 19, 2019, at 12:22 PM, Guy
Boisvert <<a
href="mailto:guy.boisvert@ingtegration.com"
class="" moz-do-not-send="true">guy.boisvert@ingtegration.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">On 2019-08-19 12:08 p.m., Darrell
Budic wrote:<br class="">
<blockquote type="cite" class="">You also need
to make sure your volume is setup properly for
best performance. Did you apply the gluster
virt group to your volumes, or at least
features.shard = on on your VM volume?<br
class="">
</blockquote>
<br class="">
That's what we did here:<br class="">
<br class="">
<br class="">
gluster volume set W2K16_Rhenium
cluster.quorum-type auto<br class="">
gluster volume set W2K16_Rhenium
network.ping-timeout 10<br class="">
gluster volume set W2K16_Rhenium auth.allow \*<br
class="">
gluster volume set W2K16_Rhenium group virt<br
class="">
gluster volume set W2K16_Rhenium
storage.owner-uid 36<br class="">
gluster volume set W2K16_Rhenium
storage.owner-gid 36<br class="">
gluster volume set W2K16_Rhenium features.shard
on<br class="">
gluster volume set W2K16_Rhenium
features.shard-block-size 256MB<br class="">
gluster volume set W2K16_Rhenium
cluster.data-self-heal-algorithm full<br
class="">
gluster volume set W2K16_Rhenium
performance.low-prio-threads 32<br class="">
<br class="">
tuned-adm profile random-io (a profile i
added in CentOS 7)<br class="">
<br class="">
<br class="">
cat /usr/lib/tuned/random-io/tuned.conf<br
class="">
===========================================<br
class="">
[main]<br class="">
summary=Optimize for Gluster virtual machine
storage<br class="">
include=throughput-performance<br class="">
<br class="">
[sysctl]<br class="">
<br class="">
vm.dirty_ratio = 5<br class="">
vm.dirty_background_ratio = 2<br class="">
<br class="">
<br class="">
Any more optimization to add to this?<br
class="">
<br class="">
<br class="">
Guy<br class="">
<br class="">
-- <br class="">
Guy Boisvert, ing.<br class="">
IngTegration inc.<br class="">
<a href="http://www.ingtegration.com" class=""
moz-do-not-send="true">http://www.ingtegration.com</a><br
class="">
<a class="moz-txt-link-freetext"
href="https://www.linkedin.com/in/guy-boisvert-8990487"
moz-do-not-send="true">https://www.linkedin.com/in/guy-boisvert-8990487</a><br
class="">
<br class="">
AVIS DE CONFIDENTIALITE : ce message peut
contenir des<br class="">
renseignements confidentiels appartenant
exclusivement a<br class="">
IngTegration Inc. ou a ses filiales. Si vous
n'etes pas<br class="">
le destinataire indique ou prevu dans ce
message (ou<br class="">
responsable de livrer ce message a la personne
indiquee ou<br class="">
prevue) ou si vous pensez que ce message vous a
ete adresse<br class="">
par erreur, vous ne pouvez pas utiliser ou
reproduire ce<br class="">
message, ni le livrer a quelqu'un d'autre. Dans
ce cas, vous<br class="">
devez le detruire et vous etes prie d'avertir
l'expediteur<br class="">
en repondant au courriel.<br class="">
<br class="">
CONFIDENTIALITY NOTICE :
Proprietary/Confidential Information<br class="">
belonging to IngTegration Inc. and its
affiliates may be<br class="">
contained in this message. If you are not a
recipient<br class="">
indicated or intended in this message (or
responsible for<br class="">
delivery of this message to such person), or you
think for<br class="">
any reason that this message may have been
addressed to you<br class="">
in error, you may not use or copy or deliver
this message to<br class="">
anyone else. In such case, you should destroy
this message<br class="">
and are asked to notify the sender by reply
email.<br class="">
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</blockquote>
</div>
</blockquote>
<blockquote type="cite">
<div dir="ltr"><span>_______________________________________________</span><br>
<span>Gluster-users mailing list</span><br>
<span><a href="mailto:Gluster-users@gluster.org"
moz-do-not-send="true">Gluster-users@gluster.org</a></span><br>
<span><a
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></span></div>
</blockquote>
</div>
</blockquote>
</body>
</html>