<div dir="ltr"><div>Hi everyone,</div><div><br></div><div>Well, I went back to Gluster's own repo (<a href="https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.2-9/openSUSE_Leap_15.2/x86_64/">https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.2-9/openSUSE_Leap_15.2/x86_64/</a>) rather than using OpenSUSE's filesystem one (<a href="https://download.opensuse.org/repositories/filesystems/openSUSE_Leap_15.2/x86_64/">https://download.opensuse.org/repositories/filesystems/openSUSE_Leap_15.2/x86_64/</a>), and tried upgrading from 7.9 to 9.1 again. This time, everything worked fine, though the status commands were failing until all the nodes got upgraded due to this:<br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><font face="monospace">gluster volume status</font></div><div><font face="monospace">Locking failed on hive. Please check log file for details.</font></div><div><font face="monospace">Locking failed on citadel. Please check log file for details.</font></div></blockquote><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div><br></div><div>Something about RPC:</div></div></div></div></div></div></div></div></div></div></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div><font face="monospace">[2021-07-27 00:47:49.288575] E [rpcsvc.c:194:rpcsvc_get_program_vector_sizer] 0-rpc-service: RPC procedure 7 not available for Program GlusterD svc mgmt v3</font></div></div></div></div></div></div></div></div></div></div></div></div></div></div><div><div><div class="gmail_signature" data-smartmail="gmail_signature"><div><div><div><div><div><div><div><div><div><div><div><font face="monospace">[2021-07-27 00:47:49.288608] E [rpcsvc.c:350:rpcsvc_program_actor] 0-rpc-service: RPC Program procedure not available for procedure 7 in GlusterD svc mgmt v3 for <a href="http://192.168.210.155:49141">192.168.210.155:49141</a></font></div></div></div></div></div></div></div></div></div></div></div></div></div></div></blockquote><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><br></div><div>During this time, the servers kept syncing, and the fuse fs was available, so there was no downtime. And as I mentioned, after upgrading, everything seems fine.</div><div><br></div><div>With this in mind, it'd have been much more user-friendly if the version
from the filesystem repo without IPv6 options compiled in didn't explode in the way it did. I couldn't even probe for peers after installing. IMO there's still a legitimate issue to be fixed here and a test to add to the codebase.</div><div dir="ltr"><br>Sincerely,<br>Artem<br><br>--<br>Founder, <a href="http://www.androidpolice.com" target="_blank">Android Police</a>, <a href="http://www.apkmirror.com/" style="font-size:12.8px" target="_blank">APK Mirror</a><span style="font-size:12.8px">, Illogical Robot LLC</span></div><div dir="ltr"><a href="http://beerpla.net/" target="_blank">beerpla.net</a> | <a href="http://twitter.com/ArtemR" target="_blank">@ArtemR</a><br></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jul 23, 2021 at 7:09 PM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Can you try setting "transport.address-family: inet" at /etc/glusterfs/glusterd.vol on all nodes ?<br>
<br>
About the rpms, if they are not yet built - the only other option is to build them from source.<br>
<br>
I assume , that the second try is on a fresh set of systems without any remnants of old Gluster install.<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
<br>
<br>
<br>
<br>
<br>
<br>
В петък, 23 юли 2021 г., 07:55:01 ч. Гринуич+3, Artem Russakovskii <<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>> написа: <br>
<br>
<br>
<br>
<br>
<br>
Hi Strahil,<br>
<br>
I am using repo builds from <a href="https://download.opensuse.org/repositories/filesystems/openSUSE_Leap_15.2/x86_64/" rel="noreferrer" target="_blank">https://download.opensuse.org/repositories/filesystems/openSUSE_Leap_15.2/x86_64/</a> (currently glusterfs-9.1-lp152.88.2.x86_64.rpm) and don't build them.<br>
<br>
Perhaps the builds at <a href="https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.2-9/openSUSE_Leap_15.2/x86_64/" rel="noreferrer" target="_blank">https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.2-9/openSUSE_Leap_15.2/x86_64/</a> are better (currently glusterfs-9.1-lp152.112.1.x86_64.rpm), does anyone know?<br>
<br>
None of the repos currently have 9.3.<br>
<br>
And regardless, I don't care for gluster using IPv6 if IPv4 works fine. Is there a way to make it stop trying to use IPv6 and only use IPv4?<br>
<br>
Sincerely,<br>
Artem<br>
<br>
--<br>
Founder, Android Police, APK Mirror, Illogical Robot LLC<br>
<a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> | @ArtemR<br>
<br>
<br>
On Thu, Jul 22, 2021 at 9:09 PM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br>
> Did you try with latest 9.X ? Based on the release notes that should be 9.3 .<br>
> <br>
> Best Regards,<br>
> Strahil Nikolov<br>
> <br>
> <br>
>> <br>
>> <br>
>> On Fri, Jul 23, 2021 at 3:06, Artem Russakovskii<br>
>> <<a href="mailto:archon810@gmail.com" target="_blank">archon810@gmail.com</a>> wrote:<br>
>> <br>
>> <br>
>> <br>
>> Hi all,<br>
>> <br>
>> I just filed this ticket <a href="https://github.com/gluster/glusterfs/issues/2648" rel="noreferrer" target="_blank">https://github.com/gluster/glusterfs/issues/2648</a>, and wanted to bring it to your attention. Any feedback would be appreciated.<br>
>> <br>
>> Description of problem:<br>
>> We have a 4-node replicate cluster running gluster 7.9. I'm currently setting up a new cluster on a new set of machines and went straight for gluster 9.1.<br>
>> However, I was unable to probe any servers due to this error:<br>
>> [2021-07-17 00:31:05.228609 +0000] I [MSGID: 106487] [glusterd-handler.c:1160:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req nexus2 24007<br>
>> [2021-07-17 00:31:05.229727 +0000] E [MSGID: 101075] [common-utils.c:3657:gf_is_local_addr] 0-management: error in getaddrinfo [{ret=Name or service not known}]<br>
>> [2021-07-17 00:31:05.230785 +0000] E [MSGID: 106408] [glusterd-peer-utils.c:217:glusterd_peerinfo_find_by_hostname] 0-management: error in getaddrinfo: Name or service not known<br>
>> [Unknown error -2]<br>
>> [2021-07-17 00:31:05.353971 +0000] I [MSGID: 106128] [glusterd-handler.c:3719:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: nexus2 (24007)<br>
>> [2021-07-17 00:31:05.375871 +0000] W [MSGID: 106061] [glusterd-handler.c:3488:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout<br>
>> [2021-07-17 00:31:05.375903 +0000] I [rpc-clnt.c:1010:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br>
>> [2021-07-17 00:31:05.377021 +0000] E [MSGID: 101075] [common-utils.c:520:gf_resolve_ip6] 0-resolver: error in getaddrinfo [{family=10}, {ret=Name or service not known}]<br>
>> [2021-07-17 00:31:05.377043 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host nexus2<br>
>> [2021-07-17 00:31:05.377147 +0000] I [MSGID: 106498] [glusterd-handler.c:3648:glusterd_friend_add] 0-management: connect returned 0<br>
>> [2021-07-17 00:31:05.377201 +0000] I [MSGID: 106004] [glusterd-handler.c:6427:__glusterd_peer_rpc_notify] 0-management: Peer <nexus2> (<00000000-0000-0000-0000-000000000000>), in state <Establishing Connection>, has disconnected from glusterd.<br>
>> [2021-07-17 00:31:05.377453 +0000] E [MSGID: 101032] [store.c:464:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/<a href="http://glusterd.info" rel="noreferrer" target="_blank">glusterd.info</a>. [No such file or directory]<br>
>> <br>
>> I then wiped the /var/lib/glusterd dir to start clean and downgraded to 7.9, then attempted to peer probe again. This time, it worked fine, proving 7.9 is working, same as it is on prod.<br>
>> At this point, I made a volume, started it, and played around with testing to my satisfaction. Then I decided to see what would happen if I tried to upgrade this working volume from 7.9 to 9.1.<br>
>> The end result is:<br>
>> * gluster volume status is only showing the local gluster node and not any of the remote nodes<br>
>> * data does seem to replicate, so the connection between the servers is actually established<br>
>> * logs are now filled with constantly repeating messages like so:<br>
>> [2021-07-22 23:29:31.039004 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host nexus2<br>
>> [2021-07-22 23:29:31.039212 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host citadel<br>
>> [2021-07-22 23:29:31.039304 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host hive<br>
>> The message "E [MSGID: 101075] [common-utils.c:520:gf_resolve_ip6] 0-resolver: error in getaddrinfo [{family=10}, {ret=Name or service not known}]" repeated 119 times between [2021-07-22 23:27:34.025983 +0000] and [2021-07-22 23:29:31.039302 +0000]<br>
>> [2021-07-22 23:29:34.039369 +0000] E [MSGID: 101075] [common-utils.c:520:gf_resolve_ip6] 0-resolver: error in getaddrinfo [{family=10}, {ret=Name or service not known}]<br>
>> [2021-07-22 23:29:34.039441 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host nexus2<br>
>> [2021-07-22 23:29:34.039558 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host citadel<br>
>> [2021-07-22 23:29:34.039659 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host hive<br>
>> [2021-07-22 23:29:37.039741 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host nexus2<br>
>> [2021-07-22 23:29:37.039921 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host citadel<br>
>> [2021-07-22 23:29:37.040015 +0000] E [name.c:265:af_inet_client_get_remote_sockaddr] 0-management: DNS resolution failed on host hive<br>
>> <br>
>> When I issue a command in cli:<br>
>> ==> cli.log <==<br>
>> [2021-07-22 23:38:11.802596 +0000] I [cli.c:840:main] 0-cli: Started running gluster with version 9.1<br>
>> **[2021-07-22 23:38:11.804007 +0000] W [socket.c:3434:socket_connect] 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Operation not supported"**<br>
>> [2021-07-22 23:38:11.906865 +0000] I [MSGID: 101190] [event-epoll.c:670:event_dispatch_epoll_worker] 0-epoll: Started thread with index [{index=0}]<br>
>> <br>
>> **Mandatory info:** **- The output of the `gluster volume info` command**:<br>
>> gluster volume info<br>
>> <br>
>> Volume Name: ap<br>
>> Type: Replicate<br>
>> Volume ID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX<br>
>> Status: Started<br>
>> Snapshot Count: 0<br>
>> Number of Bricks: 1 x 4 = 4<br>
>> Transport-type: tcp<br>
>> Bricks:<br>
>> Brick1: nexus2:/mnt/nexus2_block1/ap<br>
>> Brick2: forge:/mnt/forge_block1/ap<br>
>> Brick3: hive:/mnt/hive_block1/ap<br>
>> Brick4: citadel:/mnt/citadel_block1/ap<br>
>> Options Reconfigured:<br>
>> performance.client-io-threads: on<br>
>> nfs.disable: on<br>
>> storage.fips-mode-rchecksum: on<br>
>> transport.address-family: inet<br>
>> cluster.self-heal-daemon: enable<br>
>> client.event-threads: 4<br>
>> cluster.data-self-heal-algorithm: full<br>
>> cluster.lookup-optimize: on<br>
>> cluster.quorum-count: 1<br>
>> cluster.quorum-type: fixed<br>
>> cluster.readdir-optimize: on<br>
>> cluster.heal-timeout: 1800<br>
>> disperse.eager-lock: on<br>
>> features.cache-invalidation: on<br>
>> features.cache-invalidation-timeout: 600<br>
>> network.inode-lru-limit: 500000<br>
>> network.ping-timeout: 7<br>
>> network.remote-dio: enable<br>
>> performance.cache-invalidation: on<br>
>> performance.cache-size: 1GB<br>
>> performance.io-thread-count: 4<br>
>> performance.md-cache-timeout: 600<br>
>> performance.rda-cache-limit: 256MB<br>
>> performance.read-ahead: off<br>
>> performance.readdir-ahead: on<br>
>> performance.stat-prefetch: on<br>
>> performance.write-behind-window-size: 32MB<br>
>> server.event-threads: 4<br>
>> cluster.background-self-heal-count: 1<br>
>> performance.cache-refresh-timeout: 10<br>
>> features.ctime: off<br>
>> cluster.granular-entry-heal: enable<br>
>> <br>
>> - The output of the gluster volume status command:<br>
>> gluster volume status<br>
>> Status of volume: ap<br>
>> Gluster process TCP Port RDMA Port Online Pid<br>
>> ------------------------------------------------------------------------------<br>
>> Brick forge:/mnt/forge_block1/ap 49152 0 Y 2622 <br>
>> Self-heal Daemon on localhost N/A N/A N N/A <br>
>> <br>
>> Task Status of Volume ap<br>
>> ------------------------------------------------------------------------------<br>
>> There are no active volume tasks<br>
>> <br>
>> - The output of the gluster volume heal command:<br>
>> gluster volume heal ap enable<br>
>> Enable heal on volume ap has been successful <br>
>> <br>
>> gluster volume heal ap <br>
>> Launching heal operation to perform index self heal on volume ap has been unsuccessful:<br>
>> Self-heal daemon is not running. Check self-heal daemon log file.<br>
>> <br>
>> - The operating system / glusterfs version:<br>
>> OpenSUSE 15.2, glusterfs 9.1.<br>
>> <br>
>> <br>
>> Sincerely,<br>
>> Artem<br>
>> <br>
>> --<br>
>> Founder, Android Police, APK Mirror, Illogical Robot LLC<br>
>> <a href="http://beerpla.net" rel="noreferrer" target="_blank">beerpla.net</a> | @ArtemR<br>
>> <br>
>> ________<br>
>> <br>
>> <br>
>> <br>
>> Community Meeting Calendar:<br>
>> <br>
>> Schedule -<br>
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
>> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
>> Gluster-users mailing list<br>
>> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
>> <br>
>> <br>
<br>
</blockquote></div>