<div dir="ltr">Hi Rifat,<div><br></div><div>I suspect you are hitting <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1773856">https://bugzilla.redhat.com/show_bug.cgi?id=1773856</a>. This bug has been fixed in master and will be backported to release branches soon. </div><div><br></div><div>Although the bug says, "with volume set operation when a node is down we see the issue", according to the RCA of the bug it can be hit when there is any kind of configuration change in volumes.</div><div><br></div><div>Hope this helps.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 28, 2020 at 11:52 PM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On February 28, 2020 4:49:45 PM GMT+02:00, Rifat Ucal <<a href="mailto:rucal@arcor.de" target="_blank">rucal@arcor.de</a>> wrote:<br>
>Hi Gluster Team,<br>
><br>
><br>
>I am trying to implment gluster fs in podman containers, which is<br>
>running except the problems described below.<br>
><br>
><br>
>My observations:<br>
><br>
>- The bricks on the server is gong offline when one of the podman<br>
>container is restarted or the appropriate server is rebooted.<br>
><br>
>- Althoug the status of the bricks are offline, the replication seems<br>
>to be working, as data will be replicated.<br>
><br>
>- I see that the replicated data will also be replicated on arbiter<br>
>node, where I was expecting to see only meta data.<br>
><br>
> <br>
><br>
>My configuration.<br>
><br>
>I created glusterfs for replication in 3 nodes on centos7 but in podman<br>
>containers<br>
><br>
>The containers in the first and second nodes should be normal<br>
>replication and 3rd node arbiter node.<br>
><br>
>After creation replication and enabling heal processes I can see also<br>
>that 3rd node is marked as arbiter node.<br>
><br>
>According to description of arbiter, the arbiter node should store only<br>
>metadata but in my configuration the replicated data will be stored in<br>
>all bricks including arbiter node.<br>
><br>
> <br>
><br>
>Questions:<br>
><br>
>When rebooting one of the server or restarting one of the glusterfs<br>
>container the restarted container is not going online until gluster<br>
>volume is stopped and started again. Is it a solution inbetween to<br>
>resolve this problem?<br>
><br>
>-why Arbiter node stores all the data, allthough it should only have<br>
>some metadata to restore the replicated data on other nodes. I would<br>
>not have problem that replication is done in all three nodes. I just<br>
>need to know <br>
><br>
>- Can you give me feedback, whether some one experience or similar<br>
>porblems with glusterfs implemented in podman containers?<br>
><br>
><br>
>Here are my configurations:<br>
><br>
>on all containers I have :CentOS Linux release 7.7.1908 glusterfs<br>
>version 7.3 and systemctl is enabled for glusterd service<br>
><br>
><br>
>My gluster volume creation:<br>
><br>
>gluster volume create cgvol1 replica 2 arbiter 1 transport tcp<br>
>avm1:/cbricks/brick1/data avm2:/cbricks/brick1/data<br>
>dvm1:/cbricks/brick1/data force<br>
><br>
><br>
>gluster peer status excuted on avm2:<br>
>Number of Peers: 2<br>
><br>
>Hostname: avm1<br>
>Uuid: 5d1dc6a7-8f34-45a3-a7c9-c69c442b66dc<br>
>State: Peer in Cluster (Connected)<br>
><br>
>Hostname: dvm1<br>
>Uuid: 310ffd58-28ab-43f1-88d3-1e381bd46ab3<br>
>State: Peer in Cluster (Connected)<br>
><br>
><br>
>gluster volume info<br>
><br>
>Volume Name: cgvol1<br>
>Type: Replicate<br>
>Volume ID: da975178-b68f-410c-884c-a7f635e4381a<br>
>Status: Started<br>
>Snapshot Count: 0<br>
>Number of Bricks: 1 x (2 + 1) = 3<br>
>Transport-type: tcp<br>
>Bricks:<br>
>Brick1: arvm1:/cbricks/brick1/data<br>
>Brick2: avm2:/cbricks/brick1/data<br>
>Brick3: devm1:/cbricks/brick1/data (arbiter)<br>
>Options Reconfigured:<br>
>cluster.self-heal-daemon: on<br>
>cluster.entry-self-heal: on<br>
>cluster.metadata-self-heal: on<br>
>cluster.data-self-heal: on<br>
>transport.address-family: inet<br>
>storage.fips-mode-rchecksum: on<br>
>nfs.disable: on<br>
>performance.client-io-threads: off<br>
><br>
><br>
>gluster volume status<br>
>Status of volume: cgvol1<br>
>Gluster process TCP Port RDMA Port Online Pid<br>
><br>
>------------------------------------------------------------------------------<br>
>Brick avm1:/cbricks/brick1/data 49152 0 Y 516<br>
>Brick avm2:/cbricks/brick1/data 49152 0 Y 353<br>
>Brick dvm1:/cbricks/brick1/data 49152 0 Y 572<br>
>Self-heal Daemon on localhost N/A N/A Y 537<br>
>Self-heal Daemon on dvm1 N/A N/A Y 593<br>
>Self-heal Daemon on avm2 N/A N/A Y 374<br>
><br>
>Task Status of Volume cgvol1<br>
>------------------------------------------------------------------------------<br>
>There are no active volume tasks<br>
><br>
>gluster volume heal cgvol1 info<br>
>Brick avm1:/cbricks/brick1/data<br>
>Status: Connected<br>
>Number of entries: 0<br>
><br>
>Brick avm2:/cbricks/brick1/data<br>
>Status: Connected<br>
>Number of entries: 0<br>
><br>
>Brick dvm1:/cbricks/brick1/data<br>
>Status: Connected<br>
>Number of entries: 0<br>
><br>
><br>
><br>
>Best Regards,<br>
><br>
>Rifat Ucal<br>
><br>
><br>
><br>
>> Jorick Astrego <<a href="mailto:jorick@netbulae.eu" target="_blank">jorick@netbulae.eu</a>> hat am 14. Februar 2020 um 10:10<br>
>geschrieben:<br>
>> <br>
>> <br>
>> Hi,<br>
>> <br>
>> It looks like you have a two node setup?<br>
>> <br>
>> Then it's expected as with two nodes you don't have quorum and<br>
>this can lead to split brains.<br>
>> <br>
>> To have HA, add another node or an arbiter node.<br>
>> <br>
>> <br>
><a href="https://docs.gluster.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/" rel="noreferrer" target="_blank">https://docs.gluster.org/en/latest/Administrator%20Guide/arbiter-volumes-and-quorum/</a><br>
>> <br>
>> You can also modify the quorum but then you shouldn't be too<br>
>attachted to the data you have on it.<br>
>> <br>
>> Regards, Jorick<br>
>> <br>
>> On 2/14/20 9:27 AM, Cloud Udupi wrote:<br>
>> <br>
>> > > Hi,<br>
>> > <br>
>> > I am new to glusterfs. I have used this guide on<br>
>CentOS-7.6. <br>
>> > <br>
><a href="https://microdevsys.com/wp/glusterfs-configuration-and-setup-w-nfs-ganesha-for-an-ha-nfs-cluster/" rel="noreferrer" target="_blank">https://microdevsys.com/wp/glusterfs-configuration-and-setup-w-nfs-ganesha-for-an-ha-nfs-cluster/</a><br>
>> > <br>
>> > glusterfs —version<br>
>> > glusterfs 7.2<br>
>> > <br>
>> > Firewall is disabled. Self heal is enabled.<br>
>> > Everything works fine until I reboot one of the servers.<br>
>When the server reboots the brick doesn't come online.<br>
>> > <br>
>> > gluster volume status<br>
>> > <br>
>> > Status of volume: gv01<br>
>> > Gluster process TCP Port RDMA<br>
>Port Online Pid<br>
>> > <br>
>------------------------------------------------------------------------------<br>
>> > Brick server1:/bricks/0/gv0 N/A N/A <br>
> N N/A <br>
>> > Brick server2:/bricks/0/gv0 49152 0 <br>
> Y 99870<br>
>> > Self-heal Daemon on localhost N/A N/A <br>
> Y 109802<br>
>> > Self-heal Daemon on server1 N/A N/A <br>
> Y 2142 <br>
>> > <br>
>> > Task Status of Volume gv01<br>
>> > <br>
>------------------------------------------------------------------------------<br>
>> > There are no active volume tasks<br>
>> > <br>
>> > gluster volume heal gv01<br>
>> > <br>
>> > Launching heal operation to perform index self heal on<br>
>volume gv01 has been unsuccessful:<br>
>> > <br>
>> > Glusterd Syncop Mgmt brick op 'Heal' failed. Please check<br>
>glustershd log file for details.<br>
>> > <br>
>> > gluster volume heal gv01 info<br>
>> > <br>
>> > gluster volume heal gv01 info<br>
>> > Brick server1:/bricks/0/gv0<br>
>> > Status: Transport endpoint is not connected<br>
>> > <br>
>> > Number of entries: -<br>
>> > <br>
>> > <br>
>> > When I do "gluster volume start gv01 force" brick starts.<br>
>> > <br>
>> > I want the brick to come online automatically after the<br>
>reboot. I have attached log file.<br>
>> > Please help.<br>
>> > <br>
>> > Regards,<br>
>> > Mark.<br>
>> > <br>
>> > <br>
>> > ________<br>
>> > <br>
>> > Community Meeting Calendar:<br>
>> > <br>
>> > APAC Schedule -<br>
>> > Every 2nd and 4th Tuesday at 11:30 AM IST<br>
>> > Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
>> > <br>
>> > NA/EMEA Schedule -<br>
>> > Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
>> > Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
>> > <br>
>> > Gluster-users mailing list<br>
>> > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a> mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>> > <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
>> > <br>
>> > > <br>
>> <br>
>> <br>
>> Met vriendelijke groet, With kind regards,<br>
>> <br>
>> Jorick Astrego<br>
>> <br>
>> Netbulae Virtualization Experts<br>
>> <br>
>> ---------------------------------------------<br>
>> Tel: 053 20 30 270 <a href="mailto:info@netbulae.eu" target="_blank">info@netbulae.eu</a> Staalsteden 4-3A KvK<br>
>08198180<br>
>> Fax: 053 20 30 271 <a href="http://www.netbulae.eu" rel="noreferrer" target="_blank">www.netbulae.eu</a> 7547 TA Enschede BTW<br>
>NL821234584B01<br>
>> <br>
>> <br>
>> ---------------------------------------------<br>
>> <br>
>> <br>
><br>
><br>
> <br>
><br>
>> ________<br>
>> <br>
>> Community Meeting Calendar:<br>
>> <br>
>> APAC Schedule -<br>
>> Every 2nd and 4th Tuesday at 11:30 AM IST<br>
>> Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
>> <br>
>> NA/EMEA Schedule -<br>
>> Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
>> Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
>> <br>
>> Gluster-users mailing list<br>
>> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
>> <br>
><br>
><br>
> <br>
<br>
Hi Rifat,<br>
<br>
Can you reproduce the same behaviour on VMs or physical machines ?<br>
If yes, then it could be an issue in the gluster version you are using, otherwise it will be related to the containerization of gluster.<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Thanks,<br></div>Sanju<br></div></div>