<div dir="ltr"><div dir="ltr"><div>Noticed the same when upgrading from 5.3 to 5.4, as mentioned.</div><div><br></div><div>I'm confused though. Is actual replication affected, because the 5.4 server and the 3x 5.3 servers still show heal info as all 4 connected, and the files seem to be replicating correctly as well.</div><div><br></div><div>So what's actually affected - just the status command, or leaving 5.4 on one of the nodes is doing some damage to the underlying fs? Is it fixable by tweaking transport.socket.ssl-enabled? Does upgrading all servers to 5.4 resolve it, or should we revert back to 5.3?</div><div><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8px" target="_blank">APK Mirror</a><span style="font-size:12.8px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="https://plus.google.com/+ArtemRussakovskii" target="_blank">+ArtemRussakovskii</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 5, 2019 at 2:02 AM Hu Bert <<a href="mailto:revirii@googlemail.com">revirii@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">fyi: did a downgrade 5.4 -> 5.3 and it worked. all replicas are up and<br>
running. Awaiting updated v5.4.<br>
<br>
thx :-)<br>
<br>
Am Di., 5. März 2019 um 09:26 Uhr schrieb Hari Gowtham <<a href="mailto:hgowtham@redhat.com" target="_blank">hgowtham@redhat.com</a>>:<br>
><br>
> There are plans to revert the patch causing this error and rebuilt 5.4.<br>
> This should happen faster. the rebuilt 5.4 should be void of this upgrade issue.<br>
><br>
> In the meantime, you can use 5.3 for this cluster.<br>
> Downgrading to 5.3 will work if it was just one node that was upgrade to 5.4<br>
> and the other nodes are still in 5.3.<br>
><br>
> On Tue, Mar 5, 2019 at 1:07 PM Hu Bert <<a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a>> wrote:<br>
> ><br>
> > Hi Hari,<br>
> ><br>
> > thx for the hint. Do you know when this will be fixed? Is a downgrade<br>
> > 5.4 -> 5.3 a possibility to fix this?<br>
> ><br>
> > Hubert<br>
> ><br>
> > Am Di., 5. März 2019 um 08:32 Uhr schrieb Hari Gowtham <<a href="mailto:hgowtham@redhat.com" target="_blank">hgowtham@redhat.com</a>>:<br>
> > ><br>
> > > Hi,<br>
> > ><br>
> > > This is a known issue we are working on.<br>
> > > As the checksum differs between the updated and non updated node, the<br>
> > > peers are getting rejected.<br>
> > > The bricks aren't coming because of the same issue.<br>
> > ><br>
> > > More about the issue: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1685120" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1685120</a><br>
> > ><br>
> > > On Tue, Mar 5, 2019 at 12:56 PM Hu Bert <<a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a>> wrote:<br>
> > > ><br>
> > > > Interestingly: gluster volume status misses gluster1, while heal<br>
> > > > statistics show gluster1:<br>
> > > ><br>
> > > > gluster volume status workdata<br>
> > > > Status of volume: workdata<br>
> > > > Gluster process TCP Port RDMA Port Online Pid<br>
> > > > ------------------------------------------------------------------------------<br>
> > > > Brick gluster2:/gluster/md4/workdata 49153 0 Y 1723<br>
> > > > Brick gluster3:/gluster/md4/workdata 49153 0 Y 2068<br>
> > > > Self-heal Daemon on localhost N/A N/A Y 1732<br>
> > > > Self-heal Daemon on gluster3 N/A N/A Y 2077<br>
> > > ><br>
> > > > vs.<br>
> > > ><br>
> > > > gluster volume heal workdata statistics heal-count<br>
> > > > Gathering count of entries to be healed on volume workdata has been successful<br>
> > > ><br>
> > > > Brick gluster1:/gluster/md4/workdata<br>
> > > > Number of entries: 0<br>
> > > ><br>
> > > > Brick gluster2:/gluster/md4/workdata<br>
> > > > Number of entries: 10745<br>
> > > ><br>
> > > > Brick gluster3:/gluster/md4/workdata<br>
> > > > Number of entries: 10744<br>
> > > ><br>
> > > > Am Di., 5. März 2019 um 08:18 Uhr schrieb Hu Bert <<a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a>>:<br>
> > > > ><br>
> > > > > Hi Miling,<br>
> > > > ><br>
> > > > > well, there are such entries, but those haven't been a problem during<br>
> > > > > install and the last kernel update+reboot. The entries look like:<br>
> > > > ><br>
> > > > > PUBLIC_IP <a href="http://gluster2.alpserver.de" rel="noreferrer" target="_blank">gluster2.alpserver.de</a> gluster2<br>
> > > > ><br>
> > > > > 192.168.0.50 gluster1<br>
> > > > > 192.168.0.51 gluster2<br>
> > > > > 192.168.0.52 gluster3<br>
> > > > ><br>
> > > > > 'ping gluster2' resolves to LAN IP; I removed the last entry in the<br>
> > > > > 1st line, did a reboot ... no, didn't help. From<br>
> > > > > /var/log/glusterfs/glusterd.log<br>
> > > > > on gluster 2:<br>
> > > > ><br>
> > > > > [2019-03-05 07:04:36.188128] E [MSGID: 106010]<br>
> > > > > [glusterd-utils.c:3483:glusterd_compare_friend_volume] 0-management:<br>
> > > > > Version of Cksums persistent differ. local cksum = 3950307018, remote<br>
> > > > > cksum = 455409345 on peer gluster1<br>
> > > > > [2019-03-05 07:04:36.188314] I [MSGID: 106493]<br>
> > > > > [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd:<br>
> > > > > Responded to gluster1 (0), ret: 0, op_ret: -1<br>
> > > > ><br>
> > > > > Interestingly there are no entries in the brick logs of the rejected<br>
> > > > > server. Well, not surprising as no brick process is running. The<br>
> > > > > server gluster1 is still in rejected state.<br>
> > > > ><br>
> > > > > 'gluster volume start workdata force' starts the brick process on<br>
> > > > > gluster1, and some heals are happening on gluster2+3, but via 'gluster<br>
> > > > > volume status workdata' the volumes still aren't complete.<br>
> > > > ><br>
> > > > > gluster1:<br>
> > > > > ------------------------------------------------------------------------------<br>
> > > > > Brick gluster1:/gluster/md4/workdata 49152 0 Y 2523<br>
> > > > > Self-heal Daemon on localhost N/A N/A Y 2549<br>
> > > > ><br>
> > > > > gluster2:<br>
> > > > > Gluster process TCP Port RDMA Port Online Pid<br>
> > > > > ------------------------------------------------------------------------------<br>
> > > > > Brick gluster2:/gluster/md4/workdata 49153 0 Y 1723<br>
> > > > > Brick gluster3:/gluster/md4/workdata 49153 0 Y 2068<br>
> > > > > Self-heal Daemon on localhost N/A N/A Y 1732<br>
> > > > > Self-heal Daemon on gluster3 N/A N/A Y 2077<br>
> > > > ><br>
> > > > ><br>
> > > > > Hubert<br>
> > > > ><br>
> > > > > Am Di., 5. März 2019 um 07:58 Uhr schrieb Milind Changire <<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>>:<br>
> > > > > ><br>
> > > > > > There are probably DNS entries or /etc/hosts entries with the public IP Addresses that the host names (gluster1, gluster2, gluster3) are getting resolved to.<br>
> > > > > > /etc/resolv.conf would tell which is the default domain searched for the node names and the DNS servers which respond to the queries.<br>
> > > > > ><br>
> > > > > ><br>
> > > > > > On Tue, Mar 5, 2019 at 12:14 PM Hu Bert <<a href="mailto:revirii@googlemail.com" target="_blank">revirii@googlemail.com</a>> wrote:<br>
> > > > > >><br>
> > > > > >> Good morning,<br>
> > > > > >><br>
> > > > > >> i have a replicate 3 setup with 2 volumes, running on version 5.3 on<br>
> > > > > >> debian stretch. This morning i upgraded one server to version 5.4 and<br>
> > > > > >> rebooted the machine; after the restart i noticed that:<br>
> > > > > >><br>
> > > > > >> - no brick process is running<br>
> > > > > >> - gluster volume status only shows the server itself:<br>
> > > > > >> gluster volume status workdata<br>
> > > > > >> Status of volume: workdata<br>
> > > > > >> Gluster process TCP Port RDMA Port Online Pid<br>
> > > > > >> ------------------------------------------------------------------------------<br>
> > > > > >> Brick gluster1:/gluster/md4/workdata N/A N/A N N/A<br>
> > > > > >> NFS Server on localhost N/A N/A N N/A<br>
> > > > > >><br>
> > > > > >> - gluster peer status on the server<br>
> > > > > >> gluster peer status<br>
> > > > > >> Number of Peers: 2<br>
> > > > > >><br>
> > > > > >> Hostname: gluster3<br>
> > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a<br>
> > > > > >> State: Peer Rejected (Connected)<br>
> > > > > >><br>
> > > > > >> Hostname: gluster2<br>
> > > > > >> Uuid: 162fea82-406a-4f51-81a3-e90235d8da27<br>
> > > > > >> State: Peer Rejected (Connected)<br>
> > > > > >><br>
> > > > > >> - gluster peer status on the other 2 servers:<br>
> > > > > >> gluster peer status<br>
> > > > > >> Number of Peers: 2<br>
> > > > > >><br>
> > > > > >> Hostname: gluster1<br>
> > > > > >> Uuid: 9a360776-7b58-49ae-831e-a0ce4e4afbef<br>
> > > > > >> State: Peer Rejected (Connected)<br>
> > > > > >><br>
> > > > > >> Hostname: gluster3<br>
> > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a<br>
> > > > > >> State: Peer in Cluster (Connected)<br>
> > > > > >><br>
> > > > > >> I noticed that, in the brick logs, i see that the public IP is used<br>
> > > > > >> instead of the LAN IP. brick logs from one of the volumes:<br>
> > > > > >><br>
> > > > > >> rejected node: <a href="https://pastebin.com/qkpj10Sd" rel="noreferrer" target="_blank">https://pastebin.com/qkpj10Sd</a><br>
> > > > > >> connected nodes: <a href="https://pastebin.com/8SxVVYFV" rel="noreferrer" target="_blank">https://pastebin.com/8SxVVYFV</a><br>
> > > > > >><br>
> > > > > >> Why is the public IP suddenly used instead of the LAN IP? Killing all<br>
> > > > > >> gluster processes and rebooting (again) didn't help.<br>
> > > > > >><br>
> > > > > >><br>
> > > > > >> Thx,<br>
> > > > > >> Hubert<br>
> > > > > >> _______________________________________________<br>
> > > > > >> Gluster-users mailing list<br>
> > > > > >> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > > > > >> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > --<br>
> > > > > > Milind<br>
> > > > > ><br>
> > > > _______________________________________________<br>
> > > > Gluster-users mailing list<br>
> > > > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
> > > > <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Regards,<br>
> > > Hari Gowtham.<br>
><br>
><br>
><br>
> --<br>
> Regards,<br>
> Hari Gowtham.<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>