<div dir="ltr">True but to work your way into that problem with replica 3 is a lot harder to achieve than with just replica 2 + arbiter.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On 7 September 2017 at 14:06, Pavel Szalbot <span dir="ltr"><<a href="mailto:pavel.szalbot@gmail.com" target="_blank">pavel.szalbot@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Hi Neil, docs mention two live nodes of replica 3 blaming each other and refusing to do IO. <div dir="auto"><br></div><div dir="auto"><a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/#1-replica-3-volume" target="_blank">https://gluster.readthedocs.<wbr>io/en/latest/Administrator%<wbr>20Guide/Split%20brain%20and%<wbr>20ways%20to%20deal%20with%<wbr>20it/#1-replica-3-volume</a><br></div><div dir="auto"><br></div><div dir="auto"><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sep 7, 2017 17:52, "Alastair Neil" <<a href="mailto:ajneil.tech@gmail.com" target="_blank">ajneil.tech@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">*shrug* I don't use arbiter for vm work loads just straight replica 3. There are some gotchas with using an arbiter for VM workloads. If quorum-type is auto and a brick that is not the arbiter drop out then if the up brick is dirty as far as the arbiter is concerned i.e. the only good copy is on the down brick you will get ENOTCONN and your VMs will halt on IO.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On 6 September 2017 at 16:06, <span dir="ltr"><<a href="mailto:lemonnierk@ulrar.net" target="_blank">lemonnierk@ulrar.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Mh, I never had to do that and I never had that problem. Is that an<br>
arbiter specific thing ? With replica 3 it just works.<br>
<div class="m_7538485454717747739m_868463869559434844HOEnZb"><div class="m_7538485454717747739m_868463869559434844h5"><br>
On Wed, Sep 06, 2017 at 03:59:14PM -0400, Alastair Neil wrote:<br>
> you need to set<br>
><br>
> cluster.server-quorum-ratio 51%<br>
><br>
> On 6 September 2017 at 10:12, Pavel Szalbot <<a href="mailto:pavel.szalbot@gmail.com" target="_blank">pavel.szalbot@gmail.com</a>> wrote:<br>
><br>
> > Hi all,<br>
> ><br>
> > I have promised to do some testing and I finally find some time and<br>
> > infrastructure.<br>
> ><br>
> > So I have 3 servers with Gluster 3.10.5 on CentOS 7. I created<br>
> > replicated volume with arbiter (2+1) and VM on KVM (via Openstack)<br>
> > with disk accessible through gfapi. Volume group is set to virt<br>
> > (gluster volume set gv_openstack_1 virt). VM runs current (all<br>
> > packages updated) Ubuntu Xenial.<br>
> ><br>
> > I set up following fio job:<br>
> ><br>
> > [job1]<br>
> > ioengine=libaio<br>
> > size=1g<br>
> > loops=16<br>
> > bs=512k<br>
> > direct=1<br>
> > filename=/tmp/fio.data2<br>
> ><br>
> > When I run fio fio.job and reboot one of the data nodes, IO statistics<br>
> > reported by fio drop to 0KB/0KB and 0 IOPS. After a while, root<br>
> > filesystem gets remounted as read-only.<br>
> ><br>
> > If you care about infrastructure, setup details etc., do not hesitate to<br>
> > ask.<br>
> ><br>
> > Gluster info on volume:<br>
> ><br>
> > Volume Name: gv_openstack_1<br>
> > Type: Replicate<br>
> > Volume ID: 2425ae63-3765-4b5e-915b-e132e0<wbr>d3fff1<br>
> > Status: Started<br>
> > Snapshot Count: 0<br>
> > Number of Bricks: 1 x (2 + 1) = 3<br>
> > Transport-type: tcp<br>
> > Bricks:<br>
> > Brick1: gfs-2.san:/export/gfs/gv_1<br>
> > Brick2: gfs-3.san:/export/gfs/gv_1<br>
> > Brick3: docker3.san:/export/gfs/gv_1 (arbiter)<br>
> > Options Reconfigured:<br>
> > nfs.disable: on<br>
> > transport.address-family: inet<br>
> > performance.quick-read: off<br>
> > performance.read-ahead: off<br>
> > performance.io-cache: off<br>
> > performance.stat-prefetch: off<br>
> > performance.low-prio-threads: 32<br>
> > network.remote-dio: enable<br>
> > cluster.eager-lock: enable<br>
> > cluster.quorum-type: auto<br>
> > cluster.server-quorum-type: server<br>
> > cluster.data-self-heal-algorit<wbr>hm: full<br>
> > cluster.locking-scheme: granular<br>
> > cluster.shd-max-threads: 8<br>
> > cluster.shd-wait-qlength: 10000<br>
> > features.shard: on<br>
> > user.cifs: off<br>
> ><br>
> > Partial KVM XML dump:<br>
> ><br>
> > <disk type='network' device='disk'><br>
> > <driver name='qemu' type='raw' cache='none'/><br>
> > <source protocol='gluster'<br>
> > name='gv_openstack_1/volume-77<wbr>ebfd13-6a92-4f38-b036-e9e55d75<wbr>2e1e'><br>
> > <host name='10.0.1.201' port='24007'/><br>
> > </source><br>
> > <backingStore/><br>
> > <target dev='vda' bus='virtio'/><br>
> > <serial>77ebfd13-6a92-4f38-b0<wbr>36-e9e55d752e1e</serial><br>
> > <alias name='virtio-disk0'/><br>
> > <address type='pci' domain='0x0000' bus='0x00' slot='0x04'<br>
> > function='0x0'/><br>
> > </disk><br>
> ><br>
> > Networking is LACP on data nodes, stack of Juniper EX4550's (10Gbps<br>
> > SFP+), separate VLAN for Gluster traffic, SSD only on Gluster all<br>
> > nodes (including arbiter).<br>
> ><br>
> > I would really love to know what am I doing wrong, because this is my<br>
> > experience with Gluster for a long time a and a reason I would not<br>
> > recommend it as VM storage backend in production environment where you<br>
> > cannot start/stop VMs on your own (e.g. providing private clouds for<br>
> > customers).<br>
> > -ps<br>
> ><br>
> ><br>
> > On Sun, Sep 3, 2017 at 10:21 PM, Gionatan Danti <<a href="mailto:g.danti@assyoma.it" target="_blank">g.danti@assyoma.it</a>><br>
> > wrote:<br>
> > > Il 30-08-2017 17:07 Ivan Rossi ha scritto:<br>
> > >><br>
> > >> There has ben a bug associated to sharding that led to VM corruption<br>
> > >> that has been around for a long time (difficult to reproduce I<br>
> > >> understood). I have not seen reports on that for some time after the<br>
> > >> last fix, so hopefully now VM hosting is stable.<br>
> > ><br>
> > ><br>
> > > Mmmm... this is precisely the kind of bug that scares me... data<br>
> > corruption<br>
> > > :|<br>
> > > Any more information on what causes it and how to resolve? Even if in<br>
> > newer<br>
> > > Gluster releases it is a solved bug, knowledge on how to treat it would<br>
> > be<br>
> > > valuable.<br>
> > ><br>
> > ><br>
> > > Thanks.<br>
> > ><br>
> > > --<br>
> > > Danti Gionatan<br>
> > > Supporto Tecnico<br>
> > > Assyoma S.r.l. - <a href="http://www.assyoma.it" rel="noreferrer" target="_blank">www.assyoma.it</a><br>
> > > email: <a href="mailto:g.danti@assyoma.it" target="_blank">g.danti@assyoma.it</a> - <a href="mailto:info@assyoma.it" target="_blank">info@assyoma.it</a><br>
> > > GPG public key ID: FF5F32A8<br>
> > > ______________________________<wbr>_________________<br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > > <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
> > ______________________________<wbr>_________________<br>
> > Gluster-users mailing list<br>
> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
> ><br>
<br>
> ______________________________<wbr>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
<br>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div></div>
</div></div></blockquote></div><br></div>