[Gluster-users] Upgrade 5.3 -> 5.4 on debian: public IP is used instead of LAN IP

Wed Mar 13 16:34:14 UTC 2019

Wednesday now with no update :-/

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | +ArtemRussakovskii
<https://plus.google.com/+ArtemRussakovskii> | @ArtemR
<http://twitter.com/ArtemR>

On Tue, Mar 12, 2019 at 10:28 AM Artem Russakovskii <archon810 at gmail.com>
wrote:

> Hi Amar,
>
> Any updates on this? I'm still not seeing it in OpenSUSE build repos.
> Maybe later today?
>
> Thanks.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | +ArtemRussakovskii
> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
> <http://twitter.com/ArtemR>
>
>
> On Wed, Mar 6, 2019 at 10:30 PM Amar Tumballi Suryanarayan <
> atumball at redhat.com> wrote:
>
>> We are talking days. Not weeks. Considering already it is Thursday here.
>> 1 more day for tagging, and packaging. May be ok to expect it on Monday.
>>
>> -Amar
>>
>> On Thu, Mar 7, 2019 at 11:54 AM Artem Russakovskii <archon810 at gmail.com>
>> wrote:
>>
>>> Is the next release going to be an imminent hotfix, i.e. something like
>>> today/tomorrow, or are we talking weeks?
>>>
>>> Sincerely,
>>> Artem
>>>
>>> --
>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>> beerpla.net | +ArtemRussakovskii
>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>> <http://twitter.com/ArtemR>
>>>
>>>
>>> On Tue, Mar 5, 2019 at 11:09 AM Artem Russakovskii <archon810 at gmail.com>
>>> wrote:
>>>
>>>> Ended up downgrading to 5.3 just in case. Peer status and volume status
>>>> are OK now.
>>>>
>>>> zypper install --oldpackage glusterfs-5.3-lp150.100.1
>>>> Loading repository data...
>>>> Reading installed packages...
>>>> Resolving package dependencies...
>>>>
>>>> Problem: glusterfs-5.3-lp150.100.1.x86_64 requires libgfapi0 = 5.3, but
>>>> this requirement cannot be provided
>>>>   not installable providers: libgfapi0-5.3-lp150.100.1.x86_64[glusterfs]
>>>>  Solution 1: Following actions will be done:
>>>>   downgrade of libgfapi0-5.4-lp150.100.1.x86_64 to
>>>> libgfapi0-5.3-lp150.100.1.x86_64
>>>>   downgrade of libgfchangelog0-5.4-lp150.100.1.x86_64 to
>>>> libgfchangelog0-5.3-lp150.100.1.x86_64
>>>>   downgrade of libgfrpc0-5.4-lp150.100.1.x86_64 to
>>>> libgfrpc0-5.3-lp150.100.1.x86_64
>>>>   downgrade of libgfxdr0-5.4-lp150.100.1.x86_64 to
>>>> libgfxdr0-5.3-lp150.100.1.x86_64
>>>>   downgrade of libglusterfs0-5.4-lp150.100.1.x86_64 to
>>>> libglusterfs0-5.3-lp150.100.1.x86_64
>>>>  Solution 2: do not install glusterfs-5.3-lp150.100.1.x86_64
>>>>  Solution 3: break glusterfs-5.3-lp150.100.1.x86_64 by ignoring some of
>>>> its dependencies
>>>>
>>>> Choose from above solutions by number or cancel [1/2/3/c] (c): 1
>>>> Resolving dependencies...
>>>> Resolving package dependencies...
>>>>
>>>> The following 6 packages are going to be downgraded:
>>>>   glusterfs libgfapi0 libgfchangelog0 libgfrpc0 libgfxdr0 libglusterfs0
>>>>
>>>> 6 packages to downgrade.
>>>>
>>>> Sincerely,
>>>> Artem
>>>>
>>>> --
>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>>> beerpla.net | +ArtemRussakovskii
>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>>> <http://twitter.com/ArtemR>
>>>>
>>>>
>>>> On Tue, Mar 5, 2019 at 10:57 AM Artem Russakovskii <archon810 at gmail.com>
>>>> wrote:
>>>>
>>>>> Noticed the same when upgrading from 5.3 to 5.4, as mentioned.
>>>>>
>>>>> I'm confused though. Is actual replication affected, because the 5.4
>>>>> server and the 3x 5.3 servers still show heal info as all 4 connected, and
>>>>> the files seem to be replicating correctly as well.
>>>>>
>>>>> So what's actually affected - just the status command, or leaving 5.4
>>>>> on one of the nodes is doing some damage to the underlying fs? Is it
>>>>> fixable by tweaking transport.socket.ssl-enabled? Does upgrading all
>>>>> servers to 5.4 resolve it, or should we revert back to 5.3?
>>>>>
>>>>> Sincerely,
>>>>> Artem
>>>>>
>>>>> --
>>>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>>>> beerpla.net | +ArtemRussakovskii
>>>>> <https://plus.google.com/+ArtemRussakovskii> | @ArtemR
>>>>> <http://twitter.com/ArtemR>
>>>>>
>>>>>
>>>>> On Tue, Mar 5, 2019 at 2:02 AM Hu Bert <revirii at googlemail.com> wrote:
>>>>>
>>>>>> fyi: did a downgrade 5.4 -> 5.3 and it worked. all replicas are up and
>>>>>> running. Awaiting updated v5.4.
>>>>>>
>>>>>> thx :-)
>>>>>>
>>>>>> Am Di., 5. März 2019 um 09:26 Uhr schrieb Hari Gowtham <
>>>>>> hgowtham at redhat.com>:
>>>>>> >
>>>>>> > There are plans to revert the patch causing this error and rebuilt
>>>>>> 5.4.
>>>>>> > This should happen faster. the rebuilt 5.4 should be void of this
>>>>>> upgrade issue.
>>>>>> >
>>>>>> > In the meantime, you can use 5.3 for this cluster.
>>>>>> > Downgrading to 5.3 will work if it was just one node that was
>>>>>> upgrade to 5.4
>>>>>> > and the other nodes are still in 5.3.
>>>>>> >
>>>>>> > On Tue, Mar 5, 2019 at 1:07 PM Hu Bert <revirii at googlemail.com>
>>>>>> wrote:
>>>>>> > >
>>>>>> > > Hi Hari,
>>>>>> > >
>>>>>> > > thx for the hint. Do you know when this will be fixed? Is a
>>>>>> downgrade
>>>>>> > > 5.4 -> 5.3 a possibility to fix this?
>>>>>> > >
>>>>>> > > Hubert
>>>>>> > >
>>>>>> > > Am Di., 5. März 2019 um 08:32 Uhr schrieb Hari Gowtham <
>>>>>> hgowtham at redhat.com>:
>>>>>> > > >
>>>>>> > > > Hi,
>>>>>> > > >
>>>>>> > > > This is a known issue we are working on.
>>>>>> > > > As the checksum differs between the updated and non updated
>>>>>> node, the
>>>>>> > > > peers are getting rejected.
>>>>>> > > > The bricks aren't coming because of the same issue.
>>>>>> > > >
>>>>>> > > > More about the issue:
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1685120
>>>>>> > > >
>>>>>> > > > On Tue, Mar 5, 2019 at 12:56 PM Hu Bert <revirii at googlemail.com>
>>>>>> wrote:
>>>>>> > > > >
>>>>>> > > > > Interestingly: gluster volume status misses gluster1, while
>>>>>> heal
>>>>>> > > > > statistics show gluster1:
>>>>>> > > > >
>>>>>> > > > > gluster volume status workdata
>>>>>> > > > > Status of volume: workdata
>>>>>> > > > > Gluster process                             TCP Port  RDMA
>>>>>> Port  Online  Pid
>>>>>> > > > >
>>>>>> ------------------------------------------------------------------------------
>>>>>> > > > > Brick gluster2:/gluster/md4/workdata        49153     0
>>>>>>     Y       1723
>>>>>> > > > > Brick gluster3:/gluster/md4/workdata        49153     0
>>>>>>     Y       2068
>>>>>> > > > > Self-heal Daemon on localhost               N/A       N/A
>>>>>>     Y       1732
>>>>>> > > > > Self-heal Daemon on gluster3                N/A       N/A
>>>>>>     Y       2077
>>>>>> > > > >
>>>>>> > > > > vs.
>>>>>> > > > >
>>>>>> > > > > gluster volume heal workdata statistics heal-count
>>>>>> > > > > Gathering count of entries to be healed on volume workdata
>>>>>> has been successful
>>>>>> > > > >
>>>>>> > > > > Brick gluster1:/gluster/md4/workdata
>>>>>> > > > > Number of entries: 0
>>>>>> > > > >
>>>>>> > > > > Brick gluster2:/gluster/md4/workdata
>>>>>> > > > > Number of entries: 10745
>>>>>> > > > >
>>>>>> > > > > Brick gluster3:/gluster/md4/workdata
>>>>>> > > > > Number of entries: 10744
>>>>>> > > > >
>>>>>> > > > > Am Di., 5. März 2019 um 08:18 Uhr schrieb Hu Bert <
>>>>>> revirii at googlemail.com>:
>>>>>> > > > > >
>>>>>> > > > > > Hi Miling,
>>>>>> > > > > >
>>>>>> > > > > > well, there are such entries, but those haven't been a
>>>>>> problem during
>>>>>> > > > > > install and the last kernel update+reboot. The entries look
>>>>>> like:
>>>>>> > > > > >
>>>>>> > > > > > PUBLIC_IP  gluster2.alpserver.de gluster2
>>>>>> > > > > >
>>>>>> > > > > > 192.168.0.50 gluster1
>>>>>> > > > > > 192.168.0.51 gluster2
>>>>>> > > > > > 192.168.0.52 gluster3
>>>>>> > > > > >
>>>>>> > > > > > 'ping gluster2' resolves to LAN IP; I removed the last
>>>>>> entry in the
>>>>>> > > > > > 1st line, did a reboot ... no, didn't help. From
>>>>>> > > > > > /var/log/glusterfs/glusterd.log
>>>>>> > > > > >  on gluster 2:
>>>>>> > > > > >
>>>>>> > > > > > [2019-03-05 07:04:36.188128] E [MSGID: 106010]
>>>>>> > > > > > [glusterd-utils.c:3483:glusterd_compare_friend_volume]
>>>>>> 0-management:
>>>>>> > > > > > Version of Cksums persistent differ. local cksum =
>>>>>> 3950307018, remote
>>>>>> > > > > > cksum = 455409345 on peer gluster1
>>>>>> > > > > > [2019-03-05 07:04:36.188314] I [MSGID: 106493]
>>>>>> > > > > > [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp]
>>>>>> 0-glusterd:
>>>>>> > > > > > Responded to gluster1 (0), ret: 0, op_ret: -1
>>>>>> > > > > >
>>>>>> > > > > > Interestingly there are no entries in the brick logs of the
>>>>>> rejected
>>>>>> > > > > > server. Well, not surprising as no brick process is
>>>>>> running. The
>>>>>> > > > > > server gluster1 is still in rejected state.
>>>>>> > > > > >
>>>>>> > > > > > 'gluster volume start workdata force' starts the brick
>>>>>> process on
>>>>>> > > > > > gluster1, and some heals are happening on gluster2+3, but
>>>>>> via 'gluster
>>>>>> > > > > > volume status workdata' the volumes still aren't complete.
>>>>>> > > > > >
>>>>>> > > > > > gluster1:
>>>>>> > > > > >
>>>>>> ------------------------------------------------------------------------------
>>>>>> > > > > > Brick gluster1:/gluster/md4/workdata        49152     0
>>>>>>       Y       2523
>>>>>> > > > > > Self-heal Daemon on localhost               N/A       N/A
>>>>>>       Y       2549
>>>>>> > > > > >
>>>>>> > > > > > gluster2:
>>>>>> > > > > > Gluster process                             TCP Port  RDMA
>>>>>> Port  Online  Pid
>>>>>> > > > > >
>>>>>> ------------------------------------------------------------------------------
>>>>>> > > > > > Brick gluster2:/gluster/md4/workdata        49153     0
>>>>>>       Y       1723
>>>>>> > > > > > Brick gluster3:/gluster/md4/workdata        49153     0
>>>>>>       Y       2068
>>>>>> > > > > > Self-heal Daemon on localhost               N/A       N/A
>>>>>>       Y       1732
>>>>>> > > > > > Self-heal Daemon on gluster3                N/A       N/A
>>>>>>       Y       2077
>>>>>> > > > > >
>>>>>> > > > > >
>>>>>> > > > > > Hubert
>>>>>> > > > > >
>>>>>> > > > > > Am Di., 5. März 2019 um 07:58 Uhr schrieb Milind Changire <
>>>>>> mchangir at redhat.com>:
>>>>>> > > > > > >
>>>>>> > > > > > > There are probably DNS entries or /etc/hosts entries with
>>>>>> the public IP Addresses that the host names (gluster1, gluster2, gluster3)
>>>>>> are getting resolved to.
>>>>>> > > > > > > /etc/resolv.conf would tell which is the default domain
>>>>>> searched for the node names and the DNS servers which respond to the
>>>>>> queries.
>>>>>> > > > > > >
>>>>>> > > > > > >
>>>>>> > > > > > > On Tue, Mar 5, 2019 at 12:14 PM Hu Bert <
>>>>>> revirii at googlemail.com> wrote:
>>>>>> > > > > > >>
>>>>>> > > > > > >> Good morning,
>>>>>> > > > > > >>
>>>>>> > > > > > >> i have a replicate 3 setup with 2 volumes, running on
>>>>>> version 5.3 on
>>>>>> > > > > > >> debian stretch. This morning i upgraded one server to
>>>>>> version 5.4 and
>>>>>> > > > > > >> rebooted the machine; after the restart i noticed that:
>>>>>> > > > > > >>
>>>>>> > > > > > >> - no brick process is running
>>>>>> > > > > > >> - gluster volume status only shows the server itself:
>>>>>> > > > > > >> gluster volume status workdata
>>>>>> > > > > > >> Status of volume: workdata
>>>>>> > > > > > >> Gluster process                             TCP Port
>>>>>> RDMA Port  Online  Pid
>>>>>> > > > > > >>
>>>>>> ------------------------------------------------------------------------------
>>>>>> > > > > > >> Brick gluster1:/gluster/md4/workdata        N/A
>>>>>>  N/A        N       N/A
>>>>>> > > > > > >> NFS Server on localhost                     N/A
>>>>>>  N/A        N       N/A
>>>>>> > > > > > >>
>>>>>> > > > > > >> - gluster peer status on the server
>>>>>> > > > > > >> gluster peer status
>>>>>> > > > > > >> Number of Peers: 2
>>>>>> > > > > > >>
>>>>>> > > > > > >> Hostname: gluster3
>>>>>> > > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a
>>>>>> > > > > > >> State: Peer Rejected (Connected)
>>>>>> > > > > > >>
>>>>>> > > > > > >> Hostname: gluster2
>>>>>> > > > > > >> Uuid: 162fea82-406a-4f51-81a3-e90235d8da27
>>>>>> > > > > > >> State: Peer Rejected (Connected)
>>>>>> > > > > > >>
>>>>>> > > > > > >> - gluster peer status on the other 2 servers:
>>>>>> > > > > > >> gluster peer status
>>>>>> > > > > > >> Number of Peers: 2
>>>>>> > > > > > >>
>>>>>> > > > > > >> Hostname: gluster1
>>>>>> > > > > > >> Uuid: 9a360776-7b58-49ae-831e-a0ce4e4afbef
>>>>>> > > > > > >> State: Peer Rejected (Connected)
>>>>>> > > > > > >>
>>>>>> > > > > > >> Hostname: gluster3
>>>>>> > > > > > >> Uuid: c7b4a448-ca6a-4051-877f-788f9ee9bc4a
>>>>>> > > > > > >> State: Peer in Cluster (Connected)
>>>>>> > > > > > >>
>>>>>> > > > > > >> I noticed that, in the brick logs, i see that the public
>>>>>> IP is used
>>>>>> > > > > > >> instead of the LAN IP. brick logs from one of the
>>>>>> volumes:
>>>>>> > > > > > >>
>>>>>> > > > > > >> rejected node: https://pastebin.com/qkpj10Sd
>>>>>> > > > > > >> connected nodes: https://pastebin.com/8SxVVYFV
>>>>>> > > > > > >>
>>>>>> > > > > > >> Why is the public IP suddenly used instead of the LAN
>>>>>> IP? Killing all
>>>>>> > > > > > >> gluster processes and rebooting (again) didn't help.
>>>>>> > > > > > >>
>>>>>> > > > > > >>
>>>>>> > > > > > >> Thx,
>>>>>> > > > > > >> Hubert
>>>>>> > > > > > >> _______________________________________________
>>>>>> > > > > > >> Gluster-users mailing list
>>>>>> > > > > > >> Gluster-users at gluster.org
>>>>>> > > > > > >> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>> > > > > > >
>>>>>> > > > > > >
>>>>>> > > > > > >
>>>>>> > > > > > > --
>>>>>> > > > > > > Milind
>>>>>> > > > > > >
>>>>>> > > > > _______________________________________________
>>>>>> > > > > Gluster-users mailing list
>>>>>> > > > > Gluster-users at gluster.org
>>>>>> > > > > https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>> > > >
>>>>>> > > >
>>>>>> > > >
>>>>>> > > > --
>>>>>> > > > Regards,
>>>>>> > > > Hari Gowtham.
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Regards,
>>>>>> > Hari Gowtham.
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> --
>> Amar Tumballi (amarts)
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190313/9b27a4c3/attachment.html>