<div dir="ltr">The options that worked best in my tests were as follows, to avoid split-brain<br><br>gluster vol set VMS cluster.heal-timeout 20<br>gluster volume heal VMS enable<br>gluster vol set VMS cluster.quorum-reads false<br>gluster vol set VMS cluster.quorum-count 1<br>gluster vol set VMS network.ping-timeout 2<br>gluster volume set VMS cluster.favorite-child-policy mtime<br>gluster volume heal VMS granular-entry-heal enable<br>gluster volume set VMS cluster.data-self-heal-algorithm full<br><br>Here <br>gluster volume set VMS cluster.favorite-child-policy mtime<br>I used "size" but I read in several places that mtime is better ...<br><br>I did several and exhaustive tests ... power off hosts, migrating vm, creating folders and files inside the vm ... activating HA etc ...<br>After the "crash" ie after the host that was restarted / shutdown comes back, the volume looks like this<br>Brick pve02: / DATA / brick<br>/images/100/vm-100-disk-0.qcow2 - Possibly undergoing heal<br>Status: Connected<br>Number of entries: 1<br><br>Indicating that healing is taking place ...<br>After a few minutes / hours depending on the hardware speed, "possibly undergoing" disappears ...<br><br>But at no time was there data loss ...<br>While possibly undergoing heals I migrate the vm from one side to another also without problems ...<br><br>Here in the tests I performed, the healing of a 10G VM HD, having 4G busy, took 30 minutes ...<br>Remembering that I'm using a virtualbox with 2 vms in it with 2 G of ram each, each vm being a proxmox.<br>In a real environment this time is much less and also depends on the size of the VM's HD!<div><br></div><div>Cheers</div><div><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>---</div><div><div><div>Gilberto Nunes Ferreira</div></div><div><span style="font-size:12.8px"></span></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Em qui., 6 de ago. de 2020 às 14:14, Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> escreveu:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The settings I got in my group is:<br>
[root@ovirt1 ~]# cat /var/lib/glusterd/groups/virt<br>
performance.quick-read=off<br>
performance.read-ahead=off<br>
performance.io-cache=off<br>
performance.low-prio-threads=32<br>
network.remote-dio=enable<br>
cluster.eager-lock=enable<br>
cluster.quorum-type=auto<br>
cluster.server-quorum-type=server<br>
cluster.data-self-heal-algorithm=full<br>
cluster.locking-scheme=granular<br>
cluster.shd-max-threads=8<br>
cluster.shd-wait-qlength=10000<br>
features.shard=on<br>
user.cifs=off<br>
cluster.choose-local=off<br>
client.event-threads=4<br>
server.event-threads=4<br>
performance.client-io-threads=on<br>
<br>
I'm not sure that sharded files are treated as big or not.If your brick disks are faster than your network bandwidth, you can enable 'cluster.choose-local' .<br>
<br>
Keep in mind that some users report issues with sparse qcow2 images during intensive writes (suspected shard xlator cannot create fast enough the shards -> default shard size (64MB) is way smaller than the RedHat's supported size which is 512MB) and I would recommend you to use preallocated qcow2 disks as much as possible or to bump the shard size.<br>
<br>
Sharding was developed especially for Virt usage.<br>
<br>
Consider using another cluster.favorite-child-policy , as all shards have the same size.<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
<br>
<br>
<br>
На 6 август 2020 г. 16:37:07 GMT+03:00, Gilberto Nunes <<a href="mailto:gilberto.nunes32@gmail.com" target="_blank">gilberto.nunes32@gmail.com</a>> написа:<br>
>Oh I see... I was confused because the terms... Now I read this and<br>
>everything becomes clear...<br>
><br>
><a href="https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/shard/" rel="noreferrer" target="_blank">https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/shard/</a><br>
><br>
><a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/configuring_red_hat_virtualization_with_red_hat_gluster_storage/chap-hosting_virtual_machine_images_on_red_hat_storage_volumes" rel="noreferrer" target="_blank">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/configuring_red_hat_virtualization_with_red_hat_gluster_storage/chap-hosting_virtual_machine_images_on_red_hat_storage_volumes</a><br>
><br>
><br>
>Should I use cluster.granular-entrey-heal-enable too, since I am<br>
>working<br>
>with big files?<br>
><br>
>Thanks<br>
><br>
>---<br>
>Gilberto Nunes Ferreira<br>
><br>
>(47) 3025-5907<br>
>(47) 99676-7530 - Whatsapp / Telegram<br>
><br>
>Skype: gilberto.nunes36<br>
><br>
><br>
><br>
><br>
><br>
>Em qui., 6 de ago. de 2020 às 09:32, Gilberto Nunes <<br>
><a href="mailto:gilberto.nunes32@gmail.com" target="_blank">gilberto.nunes32@gmail.com</a>> escreveu:<br>
><br>
>> What do you mean "sharding"? Do you mean sharing folders between two<br>
>> servers to host qcow2 or raw vm images?<br>
>> Here I am using Proxmox which uses qemu but not virsh.<br>
>><br>
>> Thanks<br>
>> ---<br>
>> Gilberto Nunes Ferreira<br>
>><br>
>> (47) 3025-5907<br>
>> (47) 99676-7530 - Whatsapp / Telegram<br>
>><br>
>> Skype: gilberto.nunes36<br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> Em qui., 6 de ago. de 2020 às 01:09, Strahil Nikolov <<br>
>> <a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> escreveu:<br>
>><br>
>>> As you mentioned qcow2 files, check the virt group<br>
>>> (/var/lib/glusterfs/group or something like that). It has optimal<br>
>setttins<br>
>>> for VMs and is used by oVirt.<br>
>>><br>
>>> WARNING: If you decide to enable the group, which will also enable<br>
>>> sharding, NEVER EVER DISABLE SHARDING -> ONCE ENABLED STAYS ENABLED<br>
>!!!<br>
>>> Sharding helps reduce loocking during replica heals.<br>
>>><br>
>>> WARNING2: As virt group uses sharding (fixes the size of file into<br>
>shard<br>
>>> size), you should consider cluster.favorite-child-policy with value<br>
>>> ctime/mtime.<br>
>>><br>
>>> Best Regards,<br>
>>> Strahil Nikolov<br>
>>><br>
>>> На 6 август 2020 г. 1:56:58 GMT+03:00, Gilberto Nunes <<br>
>>> <a href="mailto:gilberto.nunes32@gmail.com" target="_blank">gilberto.nunes32@gmail.com</a>> написа:<br>
>>> >Ok...Thanks a lot Strahil<br>
>>> ><br>
>>> >This gluster volume set VMS cluster.favorite-child-policy size do<br>
>the<br>
>>> >trick<br>
>>> >to me here!<br>
>>> ><br>
>>> >Cheers<br>
>>> >---<br>
>>> >Gilberto Nunes Ferreira<br>
>>> ><br>
>>> >(47) 3025-5907<br>
>>> >(47) 99676-7530 - Whatsapp / Telegram<br>
>>> ><br>
>>> >Skype: gilberto.nunes36<br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> >Em qua., 5 de ago. de 2020 às 18:15, Strahil Nikolov<br>
>>> ><<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>><br>
>>> >escreveu:<br>
>>> ><br>
>>> >> This could happen if you have pending heals. Did you reboot that<br>
>node<br>
>>> >> recently ?<br>
>>> >> Did you set automatic unsplit-brain ?<br>
>>> >><br>
>>> >> Check for pending heals and files in splitbrain.<br>
>>> >><br>
>>> >> If not, you can check<br>
>>> >><br>
>>><br>
>><a href="https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/" rel="noreferrer" target="_blank">https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/</a><br>
>>> >> (look at point 5).<br>
>>> >><br>
>>> >> Best Regards,<br>
>>> >> Strahil Nikolov<br>
>>> >><br>
>>> >> На 5 август 2020 г. 23:41:57 GMT+03:00, Gilberto Nunes <<br>
>>> >> <a href="mailto:gilberto.nunes32@gmail.com" target="_blank">gilberto.nunes32@gmail.com</a>> написа:<br>
>>> >> >I'm in trouble here.<br>
>>> >> >When I shutdown the pve01 server, the shared folder over<br>
>glusterfs<br>
>>> >is<br>
>>> >> >EMPTY!<br>
>>> >> >It's supposed to be a qcow2 file inside it.<br>
>>> >> >The content is show right, just after I power on pve01 backup...<br>
>>> >> ><br>
>>> >> >Some advice?<br>
>>> >> ><br>
>>> >> ><br>
>>> >> >Thanks<br>
>>> >> ><br>
>>> >> >---<br>
>>> >> >Gilberto Nunes Ferreira<br>
>>> >> ><br>
>>> >> >(47) 3025-5907<br>
>>> >> >(47) 99676-7530 - Whatsapp / Telegram<br>
>>> >> ><br>
>>> >> >Skype: gilberto.nunes36<br>
>>> >> ><br>
>>> >> ><br>
>>> >> ><br>
>>> >> ><br>
>>> >> ><br>
>>> >> >Em qua., 5 de ago. de 2020 às 11:07, Gilberto Nunes <<br>
>>> >> ><a href="mailto:gilberto.nunes32@gmail.com" target="_blank">gilberto.nunes32@gmail.com</a>> escreveu:<br>
>>> >> ><br>
>>> >> >> Well...<br>
>>> >> >> I do the follow:<br>
>>> >> >><br>
>>> >> >> gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1<br>
>>> >> >> pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2<br>
>pv<br>
>>> >> >> e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force<br>
>>> >> >><br>
>>> >> >> And now I have:<br>
>>> >> >> gluster vol info<br>
>>> >> >><br>
>>> >> >> Volume Name: VMS<br>
>>> >> >> Type: Distributed-Replicate<br>
>>> >> >> Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd<br>
>>> >> >> Status: Started<br>
>>> >> >> Snapshot Count: 0<br>
>>> >> >> Number of Bricks: 2 x (2 + 1) = 6<br>
>>> >> >> Transport-type: tcp<br>
>>> >> >> Bricks:<br>
>>> >> >> Brick1: pve01:/DATA/brick1<br>
>>> >> >> Brick2: pve02:/DATA/brick1.5<br>
>>> >> >> Brick3: pve01:/DATA/arbiter1.5 (arbiter)<br>
>>> >> >> Brick4: pve02:/DATA/brick2<br>
>>> >> >> Brick5: pve01:/DATA/brick2.5<br>
>>> >> >> Brick6: pve02:/DATA/arbiter2.5 (arbiter)<br>
>>> >> >> Options Reconfigured:<br>
>>> >> >> cluster.quorum-count: 1<br>
>>> >> >> cluster.quorum-reads: false<br>
>>> >> >> cluster.self-heal-daemon: enable<br>
>>> >> >> cluster.heal-timeout: 10<br>
>>> >> >> storage.fips-mode-rchecksum: on<br>
>>> >> >> transport.address-family: inet<br>
>>> >> >> nfs.disable: on<br>
>>> >> >> performance.client-io-threads: off<br>
>>> >> >><br>
>>> >> >> This values I have put it myself, in order to see if could<br>
>improve<br>
>>> >> >the<br>
>>> >> >> time to make the volume available, when pve01 goes down with<br>
>>> >ifupdown<br>
>>> >> >> cluster.quorum-count: 1<br>
>>> >> >> cluster.quorum-reads: false<br>
>>> >> >> cluster.self-heal-daemon: enable<br>
>>> >> >> cluster.heal-timeout: 10<br>
>>> >> >><br>
>>> >> >> Nevertheless, it took more than 1 minutes to the volume VMS<br>
>>> >available<br>
>>> >> >in<br>
>>> >> >> the other host (pve02).<br>
>>> >> >> Is there any trick to reduce this time ?<br>
>>> >> >><br>
>>> >> >> Thanks<br>
>>> >> >><br>
>>> >> >> ---<br>
>>> >> >> Gilberto Nunes Ferreira<br>
>>> >> >><br>
>>> >> >><br>
>>> >> >><br>
>>> >> >><br>
>>> >> >><br>
>>> >> >><br>
>>> >> >> Em qua., 5 de ago. de 2020 às 08:57, Gilberto Nunes <<br>
>>> >> >> <a href="mailto:gilberto.nunes32@gmail.com" target="_blank">gilberto.nunes32@gmail.com</a>> escreveu:<br>
>>> >> >><br>
>>> >> >>> hum I see... like this:<br>
>>> >> >>> [image: image.png]<br>
>>> >> >>> ---<br>
>>> >> >>> Gilberto Nunes Ferreira<br>
>>> >> >>><br>
>>> >> >>> (47) 3025-5907<br>
>>> >> >>> (47) 99676-7530 - Whatsapp / Telegram<br>
>>> >> >>><br>
>>> >> >>> Skype: gilberto.nunes36<br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>><br>
>>> >> >>> Em qua., 5 de ago. de 2020 às 02:14, Computerisms Corporation<br>
><<br>
>>> >> >>> <a href="mailto:bob@computerisms.ca" target="_blank">bob@computerisms.ca</a>> escreveu:<br>
>>> >> >>><br>
>>> >> >>>> check the example of the chained configuration on this page:<br>
>>> >> >>>><br>
>>> >> >>>><br>
>>> >> >>>><br>
>>> >> ><br>
>>> >><br>
>>> ><br>
>>><br>
><a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes" rel="noreferrer" target="_blank">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes</a><br>
>>> >> >>>><br>
>>> >> >>>> and apply it to two servers...<br>
>>> >> >>>><br>
>>> >> >>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:<br>
>>> >> >>>> > Hi Bob!<br>
>>> >> >>>> ><br>
>>> >> >>>> > Could you, please, send me more detail about this<br>
>>> >configuration?<br>
>>> >> >>>> > I will appreciate that!<br>
>>> >> >>>> ><br>
>>> >> >>>> > Thank you<br>
>>> >> >>>> > ---<br>
>>> >> >>>> > Gilberto Nunes Ferreira<br>
>>> >> >>>> ><br>
>>> >> >>>> > (47) 3025-5907<br>
>>> >> >>>> > **<br>
>>> >> >>>> > (47) 99676-7530 - Whatsapp / Telegram<br>
>>> >> >>>> ><br>
>>> >> >>>> > Skype: gilberto.nunes36<br>
>>> >> >>>> ><br>
>>> >> >>>> ><br>
>>> >> >>>> ><br>
>>> >> >>>> ><br>
>>> >> >>>> ><br>
>>> >> >>>> > Em ter., 4 de ago. de 2020 às 23:47, Computerisms<br>
>Corporation<br>
>>> >> >>>> > <<a href="mailto:bob@computerisms.ca" target="_blank">bob@computerisms.ca</a> <mailto:<a href="mailto:bob@computerisms.ca" target="_blank">bob@computerisms.ca</a>>><br>
>escreveu:<br>
>>> >> >>>> ><br>
>>> >> >>>> > Hi Gilberto,<br>
>>> >> >>>> ><br>
>>> >> >>>> > My understanding is there can only be one arbiter per<br>
>>> >> >replicated<br>
>>> >> >>>> > set. I<br>
>>> >> >>>> > don't have a lot of practice with gluster, so this<br>
>could<br>
>>> >be<br>
>>> >> >bad<br>
>>> >> >>>> advice,<br>
>>> >> >>>> > but the way I dealt with it on my two servers was to<br>
>use 6<br>
>>> >> >bricks<br>
>>> >> >>>> as<br>
>>> >> >>>> > distributed-replicated (this is also relatively easy<br>
>to<br>
>>> >> >migrate to<br>
>>> >> >>>> 3<br>
>>> >> >>>> > servers if that happens for you in the future):<br>
>>> >> >>>> ><br>
>>> >> >>>> > Server1 Server2<br>
>>> >> >>>> > brick1 brick1.5<br>
>>> >> >>>> > arbiter1.5 brick2<br>
>>> >> >>>> > brick2.5 arbiter2.5<br>
>>> >> >>>> ><br>
>>> >> >>>> > On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:<br>
>>> >> >>>> > > Hi there.<br>
>>> >> >>>> > > I have two physical servers deployed as replica 2<br>
>and,<br>
>>> >> >>>> obviously,<br>
>>> >> >>>> > I got<br>
>>> >> >>>> > > a split-brain.<br>
>>> >> >>>> > > So I am thinking in use two virtual machines,each<br>
>one<br>
>>> >in<br>
>>> >> >>>> physical<br>
>>> >> >>>> > > servers....<br>
>>> >> >>>> > > Then this two VMS act as a artiber of gluster<br>
>set....<br>
>>> >> >>>> > ><br>
>>> >> >>>> > > Is this doable?<br>
>>> >> >>>> > ><br>
>>> >> >>>> > > Thanks<br>
>>> >> >>>> > ><br>
>>> >> >>>> > > ________<br>
>>> >> >>>> > ><br>
>>> >> >>>> > ><br>
>>> >> >>>> > ><br>
>>> >> >>>> > > Community Meeting Calendar:<br>
>>> >> >>>> > ><br>
>>> >> >>>> > > Schedule -<br>
>>> >> >>>> > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
>>> >> >>>> > > Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
>>> >> >>>> > ><br>
>>> >> >>>> > > Gluster-users mailing list<br>
>>> >> >>>> > > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>>> >> ><mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
>>> >> >>>> > ><br>
>>> ><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
>>> >> >>>> > ><br>
>>> >> >>>> > ________<br>
>>> >> >>>> ><br>
>>> >> >>>> ><br>
>>> >> >>>> ><br>
>>> >> >>>> > Community Meeting Calendar:<br>
>>> >> >>>> ><br>
>>> >> >>>> > Schedule -<br>
>>> >> >>>> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
>>> >> >>>> > Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
>>> >> >>>> ><br>
>>> >> >>>> > Gluster-users mailing list<br>
>>> >> >>>> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>>> ><mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
>>> >> >>>> > <br>
><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
>>> >> >>>> ><br>
>>> >> >>>><br>
>>> >> >>><br>
>>> >><br>
>>><br>
>><br>
</blockquote></div>