<div dir="ltr"><div dir="ltr">Hi Marcus,<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 20, 2023 at 2:53 PM Marcus Pedersén <<a href="mailto:marcus.pedersen@slu.se">marcus.pedersen@slu.se</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi again Xavi,<br>
<br>
I did some more testing on my virt machines<br>
with same setup:<br>
Number of Bricks: 1 x (2 + 1) = 3<br>
If I do it the same way, I upgrade the arbiter first,<br>
I get the same behavior that the bricks do not start<br>
and the other nodes does not "see" the upgraded node.<br>
If I upgrade one of the other nodes (non arbiter) and restart<br>
glusterd on both the arbiter and the other the arbiter starts<br>
the bricks and connects with the other upgraded node as expected.<br>
If I upgrade the last node (non arbiter) it will fail to start<br>
the bricks, same behaviour as the arbiter at first.<br>
If I then copy the /var/lib/gluster/vols/<myvol> from the<br>
upgraded (non arbiter) node to the other node that does not start the bricks<br>
and replace /var/lib/gluster/vols/<myvol> with the copied directory<br>
and restarts glusterd it works nicely after that.<br>
Everything then works the way it should.<br>
<br>
So the question is if the arbiter is treated in some other way<br>
compared to the other nodes?<br></blockquote><div><br></div><div>It seems so, but at this point I'm not sure what could be the difference.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Some type of config is happening at the start of the glusterd that<br>
makes the node fail?<br></blockquote><div><br></div><div>Gluster requires that all glusterd share the same configuration. In this case it seems that the "info" file in the volume definition has different contents on the servers. One of the servers has the value "nfs.disable=on" but the others do not. This can be the difference that causes the checksum error.</div><div><br></div><div>You can try to copy the "info" file from one node to the one that doesn't start and try restarting glusterd.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Do I dare to continue to upgrade my real cluster with the above described way?<br>
<br>
Thanks!<br>
<br>
Regards<br>
Marcus<br>
<br>
<br>
<br>
On Mon, Feb 20, 2023 at 01:42:47PM +0100, Marcus Pedersén wrote:<br>
> I made a recusive diff on the upgraded arbiter.<br>
><br>
> /var/lib/glusterd/vols/gds-common is the upgraded aribiter<br>
> /home/marcus/gds-common is one of the other nodes still on gluster 10<br>
><br>
> diff -r /var/lib/glusterd/vols/gds-common/bricks/urd-gds-030:-urd-gds-gds-common /home/marcus/gds-common/bricks/urd-gds-030:-urd-gds-gds-common<br>
> 5c5<br>
> < listen-port=60419<br>
> ---<br>
> > listen-port=0<br>
> 11c11<br>
> < brick-fsid=14764358630653534655<br>
> ---<br>
> > brick-fsid=0<br>
> diff -r /var/lib/glusterd/vols/gds-common/bricks/urd-gds-031:-urd-gds-gds-common /home/marcus/gds-common/bricks/urd-gds-031:-urd-gds-gds-common<br>
> 5c5<br>
> < listen-port=0<br>
> ---<br>
> > listen-port=60891<br>
> 11c11<br>
> < brick-fsid=0<br>
> ---<br>
> > brick-fsid=1088380223149770683<br>
> diff -r /var/lib/glusterd/vols/gds-common/cksum /home/marcus/gds-common/cksum<br>
> 1c1<br>
> < info=3948700922<br>
> ---<br>
> > info=458813151<br>
> diff -r /var/lib/glusterd/vols/gds-common/gds-common.urd-gds-030.urd-gds-gds-common.vol /home/marcus/gds-common/gds-common.urd-gds-030.urd-gds-gds-common.vol<br>
> 3c3<br>
> < option shared-brick-count 1<br>
> ---<br>
> > option shared-brick-count 0<br>
> diff -r /var/lib/glusterd/vols/gds-common/gds-common.urd-gds-031.urd-gds-gds-common.vol /home/marcus/gds-common/gds-common.urd-gds-031.urd-gds-gds-common.vol<br>
> 3c3<br>
> < option shared-brick-count 0<br>
> ---<br>
> > option shared-brick-count 1<br>
> diff -r /var/lib/glusterd/vols/gds-common/info /home/marcus/gds-common/info<br>
> 23a24<br>
> > nfs.disable=on<br>
><br>
><br>
> I setup 3 virt machines and configured them with gluster 10 (arbiter 1).<br>
> After that I upgraded to 11 and the first 2 nodes was fine but on the third<br>
> node I got the same behaviour: the brick never started.<br>
><br>
> Thanks for the help!<br>
><br>
> Regards<br>
> Marcus<br>
><br>
><br>
> On Mon, Feb 20, 2023 at 12:30:37PM +0100, Xavi Hernandez wrote:<br>
> > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.<br>
> ><br>
> ><br>
> > Hi Marcus,<br>
> ><br>
> > On Mon, Feb 20, 2023 at 8:50 AM Marcus Pedersén <<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a><mailto:<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>>> wrote:<br>
> > Hi Xavi,<br>
> > I stopped glusterd and killall glusterd glusterfs glusterfsd<br>
> > and started glusterd again.<br>
> ><br>
> > The only log that is not empty is glusterd.log, I attach the log<br>
> > from the restart time. The brick log, glustershd.log and glfsheal-gds-common.log is empty.<br>
> ><br>
> > This are the errors in the log:<br>
> > [2023-02-20 07:23:46.235263 +0000] E [MSGID: 106061] [glusterd.c:597:glusterd_crt_georep_folders] 0-glusterd: Dict get failed [{Key=log-group}, {errno=2}, {error=No such file or directory}]<br>
> > [2023-02-20 07:23:47.359917 +0000] E [MSGID: 106010] [glusterd-utils.c:3542:glusterd_compare_friend_volume] 0-management: Version of Cksums gds-common differ. local cksum = 3017846959, remote cksum = 2065453698 on peer urd-gds-031<br>
> > [2023-02-20 07:23:47.438052 +0000] E [MSGID: 106010] [glusterd-utils.c:3542:glusterd_compare_friend_volume] 0-management: Version of Cksums gds-common differ. local cksum = 3017846959, remote cksum = 2065453698 on peer urd-gds-032<br>
> ><br>
> > Geo replication is not setup so I guess there is nothing strange that there is an error regarding georep.<br>
> > The checksum error seems natural to be there as the other nodes are still on version 10.<br>
> ><br>
> > No. The configurations should be identical.<br>
> ><br>
> > Can you try to compare volume definitions in /var/lib/glusterd/vols/gds-common between the upgraded server and one of the old ones ?<br>
> ><br>
> > Regards,<br>
> ><br>
> > Xavi<br>
> ><br>
> ><br>
> > My previous exprience with upgrades is that the local bricks starts and<br>
> > gluster is up and running. No connection with the other nodes until they are upgraded as well.<br>
> ><br>
> ><br>
> > gluster peer status, gives the output:<br>
> > Number of Peers: 2<br>
> ><br>
> > Hostname: urd-gds-032<br>
> > Uuid: e6f96ad2-0fea-4d80-bd42-8236dd0f8439<br>
> > State: Peer Rejected (Connected)<br>
> ><br>
> > Hostname: urd-gds-031<br>
> > Uuid: 2d7c0ad7-dfcf-4eaf-9210-f879c7b406bf<br>
> > State: Peer Rejected (Connected)<br>
> ><br>
> > I suppose and guess that this is due to that the arbiter is version 11<br>
> > and the other 2 nodes are version 10.<br>
> ><br>
> > Please let me know if I can provide any other information<br>
> > to try to solve this issue.<br>
> ><br>
> > Many thanks!<br>
> > Marcus<br>
> ><br>
> ><br>
> > On Mon, Feb 20, 2023 at 07:29:20AM +0100, Xavi Hernandez wrote:<br>
> > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.<br>
> > ><br>
> > ><br>
> > > Hi Marcus,<br>
> > ><br>
> > > these errors shouldn't prevent the bricks from starting. Isn't there any other error or warning ?<br>
> > ><br>
> > > Regards,<br>
> > ><br>
> > > Xavi<br>
> > ><br>
> > > On Fri, Feb 17, 2023 at 3:06 PM Marcus Pedersén <<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a><mailto:<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>><mailto:<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a><mailto:<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>>>> wrote:<br>
> > > Hi all,<br>
> > > I started an upgrade to gluster 11.0 from 10.3 on one of my clusters.<br>
> > > OS: Debian bullseye<br>
> > ><br>
> > > Volume Name: gds-common<br>
> > > Type: Replicate<br>
> > > Volume ID: 42c9fa00-2d57-4a58-b5ae-c98c349cfcb6<br>
> > > Status: Started<br>
> > > Snapshot Count: 0<br>
> > > Number of Bricks: 1 x (2 + 1) = 3<br>
> > > Transport-type: tcp<br>
> > > Bricks:<br>
> > > Brick1: urd-gds-031:/urd-gds/gds-common<br>
> > > Brick2: urd-gds-032:/urd-gds/gds-common<br>
> > > Brick3: urd-gds-030:/urd-gds/gds-common (arbiter)<br>
> > > Options Reconfigured:<br>
> > > cluster.granular-entry-heal: on<br>
> > > storage.fips-mode-rchecksum: on<br>
> > > transport.address-family: inet<br>
> > > nfs.disable: on<br>
> > > performance.client-io-threads: off<br>
> > ><br>
> > > I started with the arbiter node, stopped all of gluster<br>
> > > upgraded to 11.0 and all went fine.<br>
> > > After upgrade I was able to see the other nodes and<br>
> > > all nodes were connected.<br>
> > > After a reboot on the arbiter nothing works the way it should.<br>
> > > Both brick1 and brick2 has connection but no connection<br>
> > > with the arbiter.<br>
> > > On the arbiter glusterd has started and is listening on port 24007,<br>
> > > the problem seems to be glusterfsd, it never starts!<br>
> > ><br>
> > > If I run: gluster volume status<br>
> > ><br>
> > > Status of volume: gds-common<br>
> > > Gluster process TCP Port RDMA Port Online Pid<br>
> > > ------------------------------------------------------------------------------<br>
> > > Brick urd-gds-030:/urd-gds/gds-common N/A N/A N N/A<br>
> > > Self-heal Daemon on localhost N/A N/A N N/A<br>
> > ><br>
> > > Task Status of Volume gds-common<br>
> > > ------------------------------------------------------------------------------<br>
> > > There are no active volume tasks<br>
> > ><br>
> > ><br>
> > > In glusterd.log I find the following errors (arbiter node):<br>
> > > [2023-02-17 12:30:40.519585 +0000] E [gf-io-uring.c:404:gf_io_uring_setup] 0-io: [MSGID:101240] Function call failed <{function=io_uring_setup()}, {error=12 (Cannot allocate memory)}><br>
> > > [2023-02-17 12:30:40.678031 +0000] E [MSGID: 106061] [glusterd.c:597:glusterd_crt_georep_folders] 0-glusterd: Dict get failed [{Key=log-group}, {errno=2}, {error=No such file or directory}]<br>
> > ><br>
> > > In brick/urd-gds-gds-common.log I find the following error:<br>
> > > [2023-02-17 12:30:43.550753 +0000] E [gf-io-uring.c:404:gf_io_uring_setup] 0-io: [MSGID:101240] Function call failed <{function=io_uring_setup()}, {error=12 (Cannot allocate memory)}><br>
> > ><br>
> > > I enclose both logfiles.<br>
> > ><br>
> > > How do I resolve this issue??<br>
> > ><br>
> > > Many thanks in advance!!<br>
> > ><br>
> > > Marcus<br>
> > > ---<br>
> > > När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" rel="noreferrer" target="_blank">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br>
> > > E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" rel="noreferrer" target="_blank">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br>
> > > ________<br>
> > ><br>
> > ><br>
> > ><br>
> > > Community Meeting Calendar:<br>
> > ><br>
> > > Schedule -<br>
> > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
> > > Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
> > > Gluster-users mailing list<br>
> > > <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><mailto:<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>>><br>
> > > <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
> > ---<br>
> > När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" rel="noreferrer" target="_blank">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br>
> > E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" rel="noreferrer" target="_blank">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br>
---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" rel="noreferrer" target="_blank">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" rel="noreferrer" target="_blank">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br>
<br>
</blockquote></div></div>