<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hari-<div class=""><br class=""></div><div class="">I was upgrading my test cluster from 5.5 to 6 and I hit this bug (<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1694010" style="caret-color: rgb(0, 0, 0);" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1694010</a><span style="caret-color: rgb(0, 0, 0);" class="">)</span> or something similar. In my case, the workaround did not work, and I was left with a gluster that had gone into no-quorum mode and stopped all the bricks. Wasn’t much in the logs either, but I noticed my /etc/glusterfs/glusterd.vol files were not the same as the newer versions, so I updated them, restarted glusterd, and suddenly the updated node showed as peer-in-cluster again. Once I updated other notes the same way, things started working again. Maybe a place to look?</div><div class=""><br class=""></div><div class="">My old config (all nodes):</div><div class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">volume management</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> type mgmt/glusterd</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option working-directory /var/lib/glusterd</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport-type socket</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport.socket.keepalive-time 10</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport.socket.keepalive-interval 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport.socket.read-fail-log off</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option ping-timeout 10</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option event-threads 1</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option rpc-auth-allow-insecure on</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""># option transport.address-family inet6</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""># option base-port 49152</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">end-volume</span></div></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">changed to:</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">volume management</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> type mgmt/glusterd</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option working-directory /var/lib/glusterd</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport-type socket,rdma</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport.socket.keepalive-time 10</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport.socket.keepalive-interval 2</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport.socket.read-fail-log off</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport.socket.listen-port 24007</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option transport.rdma.listen-port 24008</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option ping-timeout 0</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option event-threads 1</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option rpc-auth-allow-insecure on</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""># option lock-timer 180</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""># option transport.address-family inet6</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""># option base-port 49152</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class=""> option max-port 60999</span></div><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">end-volume</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class="">the only thing I found in the glusterd logs that looks relevant was (repeated for both of the other nodes in this cluster), so no clue why it happened:</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><div style="margin: 0px; font-stretch: normal; line-height: normal;" class=""><span style="font-variant-ligatures: no-common-ligatures; background-color: rgba(255, 255, 255, 0);" class="">[2019-04-03 20:19:16.802638] I [MSGID: 106004] [glusterd-handler.c:6427:__glusterd_peer_rpc_notify] 0-management: Peer <ossuary-san> (<0ecbf953-681b-448f-9746-d1c1fe7a0978>), in state <Peer in Cluster>, has disconnected from glusterd.</span></div><div class=""><span style="font-variant-ligatures: no-common-ligatures" class=""><br class=""></span></div></span></div></span></div><div class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Apr 2, 2019, at 4:53 AM, Atin Mukherjee <<a href="mailto:atin.mukherjee83@gmail.com" class="">atin.mukherjee83@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><br class="Apple-interchange-newline"><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 1 Apr 2019 at 10:28, Hari Gowtham <<a href="mailto:hgowtham@redhat.com" class="">hgowtham@redhat.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">Comments inline.<br class=""><br class="">On Mon, Apr 1, 2019 at 5:55 AM Sankarshan Mukhopadhyay<br class=""><<a href="mailto:sankarshan.mukhopadhyay@gmail.com" target="_blank" class="">sankarshan.mukhopadhyay@gmail.com</a>> wrote:<br class="">><br class="">> Quite a considerable amount of detail here. Thank you!<br class="">><br class="">> On Fri, Mar 29, 2019 at 11:42 AM Hari Gowtham <<a href="mailto:hgowtham@redhat.com" target="_blank" class="">hgowtham@redhat.com</a>> wrote:<br class="">> ><br class="">> > Hello Gluster users,<br class="">> ><br class="">> > As you all aware that glusterfs-6 is out, we would like to inform you<br class="">> > that, we have spent a significant amount of time in testing<br class="">> > glusterfs-6 in upgrade scenarios. We have done upgrade testing to<br class="">> > glusterfs-6 from various releases like 3.12, 4.1 and 5.3.<br class="">> ><br class="">> > As glusterfs-6 has got in a lot of changes, we wanted to test those portions.<br class="">> > There were xlators (and respective options to enable/disable them)<br class="">> > added and deprecated in glusterfs-6 from various versions [1].<br class="">> ><br class="">> > We had to check the following upgrade scenarios for all such options<br class="">> > Identified in [1]:<br class="">> > 1) option never enabled and upgraded<br class="">> > 2) option enabled and then upgraded<br class="">> > 3) option enabled and then disabled and then upgraded<br class="">> ><br class="">> > We weren't manually able to check all the combinations for all the options.<br class="">> > So the options involving enabling and disabling xlators were prioritized.<br class="">> > The below are the result of the ones tested.<br class="">> ><br class="">> > Never enabled and upgraded:<br class="">> > checked from 3.12, 4.1, 5.3 to 6 the upgrade works.<br class="">> ><br class="">> > Enabled and upgraded:<br class="">> > Tested for tier which is deprecated, It is not a recommended upgrade.<br class="">> > As expected the volume won't be consumable and will have a few more<br class="">> > issues as well.<br class="">> > Tested with 3.12, 4.1 and 5.3 to 6 upgrade.<br class="">> ><br class="">> > Enabled, disabled before upgrade.<br class="">> > Tested for tier with 3.12 and the upgrade went fine.<br class="">> ><br class="">> > There is one common issue to note in every upgrade. The node being<br class="">> > upgraded is going into disconnected state. You have to flush the iptables<br class="">> > and the restart glusterd on all nodes to fix this.<br class="">> ><br class="">><br class="">> Is this something that is written in the upgrade notes? I do not seem<br class="">> to recall, if not, I'll send a PR<br class=""><br class="">No this wasn't mentioned in the release notes. PRs are welcome.<br class=""><br class="">><br class="">> > The testing for enabling new options is still pending. The new options<br class="">> > won't cause as much issues as the deprecated ones so this was put at<br class="">> > the end of the priority list. It would be nice to get contributions<br class="">> > for this.<br class="">> ><br class="">><br class="">> Did the range of tests lead to any new issues?<br class=""><br class="">Yes. In the first round of testing we found an issue and had to postpone the<br class="">release of 6 until the fix was made available.<br class=""><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1684029" rel="noreferrer" target="_blank" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1684029</a><br class=""><br class="">And then we tested it again after this patch was made available.<br class="">and came across this:<br class=""><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1694010" rel="noreferrer" target="_blank" class="">https://bugzilla.redhat.com/show_bug.cgi?id=1694010</a></blockquote><div dir="auto" class=""><br class=""></div><div dir="auto" class="">This isn’t a bug as we found that upgrade worked seamelessly in two different setup. So we have no issues in the upgrade path to glusterfs-6 release.</div><div dir="auto" class=""><br class=""></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1694010" rel="noreferrer" target="_blank" class=""></a><br class=""><br class="">Have mentioned this in the second mail as to how to over this situation<br class="">for now until the fix is available.<br class=""><br class="">><br class="">> > For the disable testing, tier was used as it covers most of the xlator<br class="">> > that was removed. And all of these tests were done on a replica 3 volume.<br class="">> ><br class="">><br class="">> I'm not sure if the Glusto team is reading this, but it would be<br class="">> pertinent to understand if the approach you have taken can be<br class="">> converted into a form of automated testing pre-release.<br class=""><br class="">I don't have an answer for this, have CCed Vijay.<br class="">He might have an idea.<br class=""><br class="">><br class="">> > Note: This is only for upgrade testing of the newly added and removed<br class="">> > xlators. Does not involve the normal tests for the xlator.<br class="">> ><br class="">> > If you have any questions, please feel free to reach us.<br class="">> ><br class="">> > [1]<span class="Apple-converted-space"> </span><a href="https://docs.google.com/spreadsheets/d/1nh7T5AXaV6kc5KgILOy2pEqjzC3t_R47f1XUXSVFetI/edit?usp=sharing" rel="noreferrer" target="_blank" class="">https://docs.google.com/spreadsheets/d/1nh7T5AXaV6kc5KgILOy2pEqjzC3t_R47f1XUXSVFetI/edit?usp=sharing</a><br class="">> ><br class="">> > Regards,<br class="">> > Hari and Sanju.<br class="">> _______________________________________________<br class="">> Gluster-users mailing list<br class="">><span class="Apple-converted-space"> </span><a href="mailto:Gluster-users@gluster.org" target="_blank" class="">Gluster-users@gluster.org</a><br class="">><span class="Apple-converted-space"> </span><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br class=""><br class=""><br class=""><br class="">--<span class="Apple-converted-space"> </span><br class="">Regards,<br class="">Hari Gowtham.<br class="">_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" target="_blank" class="">Gluster-users@gluster.org</a><br class=""><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br class=""></blockquote></div></div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">--<span class="Apple-converted-space"> </span></span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">--Atin</div><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">_______________________________________________</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;" class="">Gluster-users mailing list</span><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="mailto:Gluster-users@gluster.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Gluster-users@gluster.org</a><br style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;" class=""><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div></blockquote></div><br class=""></div></body></html>