[From nobody Wed Aug 28 06:43:10 2019 Received: from [192.222.158.19] (account csirotic@evoqarchitecture.com) by evoqarchitecture.com (CommuniGate Pro HTTP 6.2.12) with AIRSYNC id 290067; Fri, 23 Aug 2019 19:00:42 -0400 From: "Carl Sirotic" <csirotic@evoqarchitecture.com> Date: Fri, 23 Aug 2019 19:00:40 -0400 Subject: Re: [Gluster-users] Brick Reboot => VMs slowdown, client crashes Message-ID: <b8922469-ae22-465f-8f8e-63f081423c88@email.android.com> X-Android-Message-ID: <b8922469-ae22-465f-8f8e-63f081423c88@email.android.com> To: Joe Julian <joe@julianfamily.org> Importance: Normal X-Priority: 3 X-MSMail-Priority: Normal MIME-Version: 1.0 Content-Type: text/html; charset=utf-8 <div dir='auto'><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">I tried the script that is supposed to do so but didn't help.</div><div dir="auto">Do you have a clean order or doing?</div><div dir="auto"><br></div><div dir="auto">I'm all for it.</div><div dir="auto">However,</div><div dir="auto">wouldn't a failing node be the same as a bad shutdown?</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">I start to wonder if starting the vm through a native fuse mount would do this vs using the libgfapi in qemu.</div><div dir="auto"><br></div><div dir="auto">Carl</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Aug. 23, 2019 6:53 p.m., Joe Julian &lt;joe@julianfamily.org&gt; wrote:<br type="attribution" /> Or fix your shutdown order such that the bricks are shut down before the network so they never have to wait for ping-timeout. Ping-timeout only comes in to play when an open tcp connection stop responding. If the connection is closed <http://www.tcpipguide.com/free/t_TCPConnectionTermination-2.htm> then the client will never wait for it to respond. On 8/23/19 3:06 PM, Carl Sirotic wrote: > > Okay, > > so it means, at least I am not getting the expected behavior and there is hope. > > I put the quorum settings that I was told a couple of emails ago. > > After applying virt group, they are > > cluster.quorum-type auto > cluster.quorum-count (null) > cluster.server-quorum-type server > cluster.server-quorum-ratio 0 > cluster.quorum-reads no > > Also, > > I just put the ping timeout to 5 seconds now. > > > Carl > > On 2019-08-23 5:45 p.m., Ingo Fischer wrote: >> Hi Carl, >> >> In my understanding and experience (I have a replica 3 System running too) this should not happen. Can you tell your client and server quorum settings? >> >> Ingo >> >> Am 23.08.2019 um 15:53 schrieb Carl Sirotic <csirotic@evoqarchitecture.com <mailto:csirotic@evoqarchitecture.com>>: >> >>> However, >>> >>> I must have misunderstood the whole concept of gluster. >>> >>> In a replica 3, for me, it's completely unacceptable, regardless of the options, that all my VMs go down when I reboot one node. >>> >>> The whole purpose of having a full 3 copy of my data on the fly is suposed to be this. >>> >>> I am in the process of sharding every file. >>> >>> But even if the healing time would be longer, I would still expect a non-sharded replica 3 brick with vm boot disk, to not go down if I reboot one of its copy. >>> >>> >>> I am not very impressed by gluster so far. >>> >>> Carl >>> >>> On 2019-08-19 4:15 p.m., Darrell Budic wrote: >>>> /var/lib/glusterd/groups/virt is a good start for ideas, notably some thread settings and choose-local=off to improve read performance. If you don’t have at least 10 cores on your servers, you may want to lower the recommended shd-max-threads=8 to no more than half your CPU cores to keep healing from swamping out regular work. >>>> >>>> It’s also starting to depend on what your backing store and networking setup are, so you’re going to want to test changes and find what works best for your setup. >>>> >>>> In addition to the virt group settings, I use these on most of my volumes, SSD or HDD backed, with the default 64M shard size: >>>> >>>> performance.io <http://performance.io>-thread-count: 32# seemed good for my system, particularly a ZFS backed volume with lots of spindles >>>> client.event-threads: 8 >>>> cluster.data-self-heal-algorithm: full# 10G networking, uses more net/less cpu to heal. probably don’t use this for 1G networking? >>>> performance.stat-prefetch: on >>>> cluster.read-hash-mode: 3# distribute reads to least loaded server (by read queue depth) >>>> >>>> and these two only on my HDD backed volume: >>>> >>>> performance.cache-size: 1G >>>> performance.write-behind-window-size: 64MB >>>> >>>> but I suspect these two need another round or six of tuning to tell if they are making a difference. >>>> >>>> I use the throughput-performance tuned profile on my servers, so you should be in good shape there. >>>> >>>>> On Aug 19, 2019, at 12:22 PM, Guy Boisvert <guy.boisvert@ingtegration.com <mailto:guy.boisvert@ingtegration.com>> wrote: >>>>> >>>>> On 2019-08-19 12:08 p.m., Darrell Budic wrote: >>>>>> You also need to make sure your volume is setup properly for best performance. Did you apply the gluster virt group to your volumes, or at least features.shard = on on your VM volume? >>>>> >>>>> That's what we did here: >>>>> >>>>> >>>>> gluster volume set W2K16_Rhenium cluster.quorum-type auto >>>>> gluster volume set W2K16_Rhenium network.ping-timeout 10 >>>>> gluster volume set W2K16_Rhenium auth.allow \* >>>>> gluster volume set W2K16_Rhenium group virt >>>>> gluster volume set W2K16_Rhenium storage.owner-uid 36 >>>>> gluster volume set W2K16_Rhenium storage.owner-gid 36 >>>>> gluster volume set W2K16_Rhenium features.shard on >>>>> gluster volume set W2K16_Rhenium features.shard-block-size 256MB >>>>> gluster volume set W2K16_Rhenium cluster.data-self-heal-algorithm full >>>>> gluster volume set W2K16_Rhenium performance.low-prio-threads 32 >>>>> >>>>> tuned-adm profile random-io        (a profile i added in CentOS 7) >>>>> >>>>> >>>>> cat /usr/lib/tuned/random-io/tuned.conf >>>>> =========================================== >>>>> [main] >>>>> summary=Optimize for Gluster virtual machine storage >>>>> include=throughput-performance >>>>> >>>>> [sysctl] >>>>> >>>>> vm.dirty_ratio = 5 >>>>> vm.dirty_background_ratio = 2 >>>>> >>>>> >>>>> Any more optimization to add to this? >>>>> >>>>> >>>>> Guy >>>>> >>>>> -- >>>>> Guy Boisvert, ing. >>>>> IngTegration inc. >>>>> http://www.ingtegration.com >>>>> https://www.linkedin.com/in/guy-boisvert-8990487 >>>>> >>>>> AVIS DE CONFIDENTIALITE : ce message peut contenir des >>>>> renseignements confidentiels appartenant exclusivement a >>>>> IngTegration Inc. ou a ses filiales. Si vous n'etes pas >>>>> le destinataire indique ou prevu dans ce  message (ou >>>>> responsable de livrer ce message a la personne indiquee ou >>>>> prevue) ou si vous pensez que ce message vous a ete adresse >>>>> par erreur, vous ne pouvez pas utiliser ou reproduire ce >>>>> message, ni le livrer a quelqu'un d'autre. Dans ce cas, vous >>>>> devez le detruire et vous etes prie d'avertir l'expediteur >>>>> en repondant au courriel. >>>>> >>>>> CONFIDENTIALITY NOTICE : Proprietary/Confidential Information >>>>> belonging to IngTegration Inc. and its affiliates may be >>>>> contained in this message. If you are not a recipient >>>>> indicated or intended in this message (or responsible for >>>>> delivery of this message to such person), or you think for >>>>> any reason that this message may have been addressed to you >>>>> in error, you may not use or copy or deliver this message to >>>>> anyone else. In such case, you should destroy this message >>>>> and are asked to notify the sender by reply email. >>>>> >>>> >>> _______________________________________________ >>> Gluster-users mailing list >>> Gluster-users@gluster.org <mailto:Gluster-users@gluster.org> >>> https://lists.gluster.org/mailman/listinfo/gluster-users > > _______________________________________________ > Gluster-users mailing list > Gluster-users@gluster.org > https://lists.gluster.org/mailman/listinfo/gluster-users AVIS DE CONFIDENTIALITÉ : Ce courriel peut contenir de l'information privilégiée et confidentielle. Nous vous demandons de le détruire immédiatement si vous n'êtes pas le destinataire. CONFIDENTIALITY NOTICE: This email may contain information that is privileged and confidential. Please delete immediately if you are not the intended recipient. ]