Start with the removal of tier-enabled from all volume files (of course back them up) and restart glusterd. Check if the errors will be less.<div><br></div><div>Best Regards,</div><div>Strahil Nikolov <br> <br> <blockquote style="margin: 0 0 20px 0;"> <div style="font-family:Roboto, sans-serif; color:#6D00F6;"> <div>On Mon, Mar 6, 2023 at 17:16, Marcus Pedersén</div><div><marcus.pedersen@slu.se> wrote:</div> </div> <div style="padding: 10px 0 0 20px; margin: 10px 0 0 0; border-left: 1px solid #6D00F6;"> Hi again,<br clear="none">As I got the error:<br clear="none">[2023-03-06 15:09:14.594977 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none"><br clear="none">multiple times I tried to remove all device_path in config in<br clear="none">/var/lib/glusterd/snaps/...<br clear="none"><br clear="none">But that did not make any difference glusterd still crashes with the same log output,<br clear="none">except that these error log lines do not exist.<br clear="none"><br clear="none">I do not know how to continue to figure this problem out!<br clear="none"><br clear="none">Best regards<br clear="none">Marcus<br clear="none"><br clear="none"><br clear="none"><br clear="none">On Mon, Mar 06, 2023 at 09:13:05AM +0100, Marcus Pedersén wrote:<br clear="none">> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.<br clear="none">><br clear="none">><br clear="none">> Hi Strahil,<br clear="none">><br clear="none">> Volume info says:<br clear="none">><br clear="none">> Volume Name: gds-home<br clear="none">> Type: Replicate<br clear="none">> Volume ID: 3d9d7182-47a8-43ac-8cd1-6a090bb4b8b9<br clear="none">> Status: Started<br clear="none">> Snapshot Count: 10<br clear="none">> Number of Bricks: 1 x (2 + 1) = 3<br clear="none">> Transport-type: tcp<br clear="none">> Bricks:<br clear="none">> Brick1: urd-gds-022:/urd-gds/gds-home<br clear="none">> Brick2: urd-gds-021:/urd-gds/gds-home<br clear="none">> Brick3: urd-gds-020:/urd-gds/gds-home (arbiter)<br clear="none">> Options Reconfigured:<br clear="none">> features.barrier: disable<br clear="none">> storage.fips-mode-rchecksum: on<br clear="none">> transport.address-family: inet<br clear="none">> nfs.disable: on<br clear="none">> performance.client-io-threads: off<br clear="none">><br clear="none">> If I look in /var/lib/glusterd/vols/gds-home/info on all three nodes it says:<br clear="none">> tier-enabled=0<br clear="none">><br clear="none">> The snapshot config also says:<br clear="none">> tier-enabled=0<br clear="none">><br clear="none">> As far as I can tell there are no depricated features enabled.<br clear="none">> This is the /var/lib/glusterd/vols/gds-home/info file from the arbiter:<br clear="none">><br clear="none">> type=2<br clear="none">> count=3<br clear="none">> status=1<br clear="none">> sub_count=3<br clear="none">> replica_count=3<br clear="none">> arbiter_count=1<br clear="none">> disperse_count=0<br clear="none">> redundancy_count=0<br clear="none">> version=1588<br clear="none">> transport-type=0<br clear="none">> volume-id=3d9d7182-47a8-43ac-8cd1-6a090bb4b8b9<br clear="none">> username=<username><br clear="none">> password=<password><br clear="none">> op-version=40000<br clear="none">> client-op-version=2<br clear="none">> quota-version=0<br clear="none">> tier-enabled=0<br clear="none">> parent_volname=N/A<br clear="none">> restored_from_snap=00000000-0000-0000-0000-000000000000<br clear="none">> snap-max-hard-limit=256<br clear="none">> features.barrier=disable<br clear="none">> storage.fips-mode-rchecksum=on<br clear="none">> transport.address-family=inet<br clear="none">> nfs.disable=on<br clear="none">> performance.client-io-threads=off<br clear="none">> brick-0=urd-gds-022:-urd-gds-gds-home<br clear="none">> brick-1=urd-gds-021:-urd-gds-gds-home<br clear="none">> brick-2=urd-gds-020:-urd-gds-gds-home<br clear="none">><br clear="none">><br clear="none">> No, you will not see the arbiter in the status report as glusterd<br clear="none">> does not run at all.<br clear="none">><br clear="none">> Thanks for your support Strahil!<br clear="none">><br clear="none">> Best regards<br clear="none">> Marcus<br clear="none">><br clear="none">><br clear="none">><br clear="none">> On Mon, Mar 06, 2023 at 06:06:00AM +0000, Strahil Nikolov wrote:<br clear="none">> > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.<br clear="none">> ><br clear="none">> ><br clear="none">> > Somewhere tiering is enabled.<br clear="none">> > Check the deprecated options in https://docs.gluster.org/en/main/Upgrade-Guide/upgrade-to-11/#the-following-options-are-removed-from-the-code-base-and-require-to-be-unset.<br clear="none">> ><br clear="none">> > The simplest way would be to downgrade the arbiter, ensure it works (or readd it back to the TSP), and remove any deprecated options before upgrading .<br clear="none">> ><br clear="none">> > Best Regards,<br clear="none">> > Strahil Nikolov<br clear="none">> ><br clear="none">> ><br clear="none">> > On Mon, Mar 6, 2023 at 8:02, Strahil Nikolov<br clear="none">> > <<a shape="rect" ymailto="mailto:hunter86_bg@yahoo.com" href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> wrote:<br clear="none">> > I don't see the arbiter in the status report.<br clear="none">> > Maybe the volfiles on host1 and host2 were changed ?<br clear="none">> ><br clear="none">> > What is the volume info ?<br clear="none">> ><br clear="none">> > Best Regards,<br clear="none">> > Strahil Nikolov<br clear="none">> ><br clear="none">> > On Fri, Mar 3, 2023 at 17:30, Marcus Pedersén<br clear="none">> > <<a shape="rect" ymailto="mailto:marcus.pedersen@slu.se" href="mailto:marcus.pedersen@slu.se">marcus.pedersen@slu.se</a>> wrote:<br clear="none">> > Hi again,<br clear="none">> > I turned up the logging level so here is a more detailed start of glusterd.<br clear="none">> > File is enclosed.<br clear="none">> ><br clear="none">> > Thanks alot for help!<br clear="none">> ><br clear="none">> > Regards<br clear="none">> > Marcus<br clear="none">> ><br clear="none">> ><br clear="none">> > On Fri, Mar 03, 2023 at 03:00:46PM +0100, Marcus Pedersén wrote:<br clear="none">> > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.<br clear="none">> > ><br clear="none">> > ><br clear="none">> > > Hi all,<br clear="none">> > ><br clear="none">> > > I just started to upgrade from gluster 10.3 to gluster 11.<br clear="none">> > > I started with my arbiter node and upgraded.<br clear="none">> > > After upgrade gluster started after change in info file.<br clear="none">> > > Rebooted machine and after that glusterd crashes.<br clear="none">> > ><br clear="none">> > > I have double checked config in /var/lib/glusterd<br clear="none">> > > and updated /var/lib/glusterd/vols/gv0/info<br clear="none">> > > so the checksum is correct.<br clear="none">> > ><br clear="none">> > > OS: debian bullseye (11)<br clear="none">> > ><br clear="none">> > > gluster volume status (from onte of the other nodes that is not upgraded)<br clear="none">> > ><br clear="none">> > > Status of volume: gv0<br clear="none">> > > Gluster process                            TCP Port  RDMA Port  Online  Pid<br clear="none">> > > ------------------------------------------------------------------------------<br clear="none">> > > Brick host2:/urd-gds/gv0                    52902    0          Y      113172<br clear="none">> > > Brick host1:/urd-gds/gv0                    61235    0          Y      5487<br clear="none">> > > Self-heal Daemon on localhost              N/A      N/A        Y      5550<br clear="none">> > > Self-heal Daemon on host2                  N/A      N/A        Y      113236<br clear="none">> > ><br clear="none">> > > Task Status of Volume gds-home<br clear="none">> > > ------------------------------------------------------------------------------<br clear="none">> > > There are no active volume tasks<br clear="none">> > ><br clear="none">> > > I do not know where to start looking,<br clear="none">> > > enclosed is a boot part from glusterd.log<br clear="none">> > ><br clear="none">> > > Thanks alot for your help!!<br clear="none">> > ><br clear="none">> > > Regards<br clear="none">> > > Marcus<br clear="none">> > ><br clear="none">> > ><br clear="none">> > ><br clear="none">> > > --<br clear="none">> > > **************************************************<br clear="none">> > > * Marcus Pedersén                                *<br clear="none">> > > * System administrator                          *<br clear="none">> > > **************************************************<br clear="none">> > > * Interbull Centre                              *<br clear="none">> > > * ================                              *<br clear="none">> > > * Department of Animal Breeding & Genetics — SLU *<br clear="none">> > > * Box 7023, SE-750 07                            *<br clear="none">> > > * Uppsala, Sweden                                *<br clear="none">> > > **************************************************<br clear="none">> > > * Visiting address:                              *<br clear="none">> > > * Room 55614, Ulls väg 26, Ultuna                *<br clear="none">> > > * Uppsala                                        *<br clear="none">> > > * Sweden                                        *<br clear="none">> > > *                                                *<br clear="none">> > > * Tel: +46-(0)18-67 1962                        *<br clear="none">> > > *                                                *<br clear="none">> > > **************************************************<br clear="none">> > > *    ISO 9001 Bureau Veritas No SE004561-1      *<br clear="none">> > > **************************************************<br clear="none">> > > ---<br clear="none">> > > När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <<a shape="rect" href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br clear="none">> > > E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <<a shape="rect" href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br clear="none">> ><br clear="none">> > > [2023-03-03 13:46:01.248805 +0000] I [MSGID: 100030] [glusterfsd.c:2872:main] 0-/usr/sbin/glusterd: Started running version [{arg=/usr/sbin/glusterd}, {version=11.0}, {cmdlinestr=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO}]<br clear="none">> > > [2023-03-03 13:46:01.249174 +0000] I [glusterfsd.c:2562:daemonize] 0-glusterfs: Pid of current running process is 4256<br clear="none">> > > [2023-03-03 13:46:01.250935 +0000] I [MSGID: 0] [glusterfsd.c:1597:volfile_init] 0-glusterfsd-mgmt: volume not found, continuing with init<br clear="none">> > > [2023-03-03 13:46:01.282327 +0000] I [MSGID: 106479] [glusterd.c:1660:init] 0-management: Using /var/lib/glusterd as working directory<br clear="none">> > > [2023-03-03 13:46:01.282371 +0000] I [MSGID: 106479] [glusterd.c:1664:init] 0-management: Using /var/run/gluster as pid file working directory<br clear="none">> > > [2023-03-03 13:46:01.288793 +0000] I [socket.c:973:__socket_server_bind] 0-socket.management: process started listening on port (24007)<br clear="none">> > > [2023-03-03 13:46:01.291749 +0000] I [socket.c:916:__socket_server_bind] 0-socket.management: closing (AF_UNIX) reuse check socket 13<br clear="none">> > > [2023-03-03 13:46:01.292549 +0000] I [MSGID: 106059] [glusterd.c:1923:init] 0-management: max-port override: 60999<br clear="none">> > > [2023-03-03 13:46:01.338404 +0000] E [MSGID: 106061] [glusterd.c:597:glusterd_crt_georep_folders] 0-glusterd: Dict get failed [{Key=log-group}, {errno=2}, {error=No such file or directory}]<br clear="none">> > > [2023-03-03 13:46:02.342898 +0000] I [MSGID: 106513] [glusterd-store.c:2198:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 100000<br clear="none">> > > [2023-03-03 13:46:02.347302 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.347333 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.347341 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.347346 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.347351 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.351708 +0000] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: d5d7204c-aa42-4b7d-a559-f48f7917e98e<br clear="none">> > > [2023-03-03 13:46:02.370758 +0000] I [MSGID: 106498] [glusterd-handler.c:3794:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0<br clear="none">> > > [2023-03-03 13:46:02.375132 +0000] I [MSGID: 106498] [glusterd-handler.c:3794:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0<br clear="none">> > > [2023-03-03 13:46:02.375188 +0000] W [MSGID: 106061] [glusterd-handler.c:3589:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout<br clear="none">> > > [2023-03-03 13:46:02.375250 +0000] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br clear="none">> > > [2023-03-03 13:46:02.376627 +0000] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600<br clear="none">> > > [2023-03-03 13:46:02.376618 +0000] W [MSGID: 106061] [glusterd-handler.c:3589:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout<br clear="none">> > > [2023-03-03 13:46:02.382039 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.382145 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.382177 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.382195 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.382211 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.386720 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.386720 +0000] and [2023-03-03 13:46:02.391550 +0000]<br clear="none">> > > [2023-03-03 13:46:02.391576 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/9974c18368084c4b89bb7447c080d92b/brick3/gds-home [No such file or directory]<br clear="none">> > > [2023-03-03 13:46:02.408551 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.408634 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.408670 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.408689 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.408705 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.413325 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.413325 +0000] and [2023-03-03 13:46:02.418058 +0000]<br clear="none">> > > [2023-03-03 13:46:02.418083 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/d41f6dbb52d449d7a22554a7795d1938/brick3/gds-home [No such file or directory]<br clear="none">> > > [2023-03-03 13:46:02.435395 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.435480 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.435517 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.435535 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.435551 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.440269 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.440269 +0000] and [2023-03-03 13:46:02.444922 +0000]<br clear="none">> > > [2023-03-03 13:46:02.444953 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/2cfe90e27d2e4e25ab445b784f834aa8/brick3/gds-home [No such file or directory]<br clear="none">> > > [2023-03-03 13:46:02.462175 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.462262 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.462295 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.462315 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.462331 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.466999 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.466999 +0000] and [2023-03-03 13:46:02.471797 +0000]<br clear="none">> > > [2023-03-03 13:46:02.471832 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/78659cc641da4d19b18ccddc71b0620b/brick3/gds-home [No such file or directory]<br clear="none">> > > [2023-03-03 13:46:02.489287 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.489376 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.489410 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.489429 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.489446 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.494318 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > [2023-03-03 13:46:02.514385 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.514468 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.514518 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.514537 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.514553 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.517199 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.517199 +0000] and [2023-03-03 13:46:02.521029 +0000]<br clear="none">> > > [2023-03-03 13:46:02.521059 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/083893ba17734106a34b1092f1dc9273/brick3/gds-home [No such file or directory]<br clear="none">> > > [2023-03-03 13:46:02.542478 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.542571 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.542606 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.542625 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.542641 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.545410 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.545410 +0000] and [2023-03-03 13:46:02.548239 +0000]<br clear="none">> > > [2023-03-03 13:46:02.548263 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/071c7bfc377d4110aefad0c30fd37dc7/brick3/gds-home [No such file or directory]<br clear="none">> > > [2023-03-03 13:46:02.569362 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.569447 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.569480 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.569498 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.569514 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.572205 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.572205 +0000] and [2023-03-03 13:46:02.574871 +0000]<br clear="none">> > > [2023-03-03 13:46:02.574895 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/a12bff428d9240bfadcb1af85fe0bbfc/brick3/gds-home [No such file or directory]<br clear="none">> > > [2023-03-03 13:46:02.596284 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.596390 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.596430 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.596466 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.596484 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.599265 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.599265 +0000] and [2023-03-03 13:46:02.602148 +0000]<br clear="none">> > > [2023-03-03 13:46:02.624580 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled<br clear="none">> > > [2023-03-03 13:46:02.624682 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable<br clear="none">> > > [2023-03-03 13:46:02.624717 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0<br clear="none">> > > [2023-03-03 13:46:02.624742 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1<br clear="none">> > > [2023-03-03 13:46:02.624767 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2<br clear="none">> > > [2023-03-03 13:46:02.627593 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path<br clear="none">> > > The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.627593 +0000] and [2023-03-03 13:46:02.630483 +0000]<br clear="none">> > > [2023-03-03 13:46:02.630512 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/e49131c77f4848d49f0ff313c628e97c/brick3/gds-home [No such file or directory]<br clear="none">> > > pending frames:<br clear="none">> > > patchset: git://git.gluster.org/glusterfs.git<br clear="none">> > > signal received: 11<br clear="none">> > > time of crash:<br clear="none">> > > 2023-03-03 13:46:02 +0000<br clear="none">> > > configuration details:<br clear="none">> > > argp 1<br clear="none">> > > backtrace 1<br clear="none">> > > dlfcn 1<br clear="none">> > > libpthread 1<br clear="none">> > > llistxattr 1<br clear="none">> > > setfsid 1<br clear="none">> > > epoll.h 1<br clear="none">> > > xattr.h 1<br clear="none">> > > st_atim.tv_nsec 1<br clear="none">> > > package-string: glusterfs 11.0<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x28b24)[0x7f99dcd5fb24]<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x6a8)[0x7f99dcd69108]<br clear="none">> > > /lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x7f99dcb30d60]<br clear="none">> > > /usr/lib/x86_64-linux-gnu/glusterfs/11.0/xlator/mgmt/glusterd.so(+0x85ad0)[0x7f99d8552ad0]<br clear="none">> > > /usr/lib/x86_64-linux-gnu/glusterfs/11.0/xlator/mgmt/glusterd.so(+0x88485)[0x7f99d8555485]<br clear="none">> > > /usr/lib/x86_64-linux-gnu/glusterfs/11.0/xlator/mgmt/glusterd.so(+0x192ce)[0x7f99d84e62ce]<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_init+0xc2)[0x7f99dcd5a452]<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x51029)[0x7f99dcd88029]<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_activate+0x9d)[0x7f99dcd882dd]<br clear="none">> > > /usr/sbin/glusterd(glusterfs_process_volfp+0x18f)[0x5652a9f6a4af]<br clear="none">> > > /usr/sbin/glusterd(+0x10936)[0x5652a9f6a936]<br clear="none">> > > /usr/sbin/glusterd(+0x12fb8)[0x5652a9f6cfb8]<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x82c60)[0x7f99dcdb9c60]<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x83247)[0x7f99dcdba247]<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x8a2ae)[0x7f99dcdc12ae]<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x849ae)[0x7f99dcdbb9ae]<br clear="none">> > > /lib/x86_64-linux-gnu/libglusterfs.so.0(+0x89736)[0x7f99dcdc0736]<br clear="none">> > > /lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f99dccd4ea7]<br clear="none">> > > /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f99dcbf4a2f]<br clear="none">> > > ---------<br clear="none">> ><br clear="none">> > > ________<br clear="none">> > ><br clear="none">> > ><br clear="none">> > ><br clear="none">> > > Community Meeting Calendar:<br clear="none">> > ><br clear="none">> > > Schedule -<br clear="none">> > > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">> > > Bridge: <a shape="rect" href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br clear="none">> > > Gluster-users mailing list<br clear="none">> > > <a shape="rect" ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><mailto:Gluster-users@gluster.org><br clear="none">> > > <a shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br clear="none">> ><br clear="none">> ><br clear="none">> ><br clear="none">> > --<br clear="none">> > **************************************************<br clear="none">> > * Marcus Pedersén                                *<br clear="none">> > * System administrator                          *<br clear="none">> > **************************************************<br clear="none">> > * Interbull Centre                              *<br clear="none">> > * ================                              *<br clear="none">> > * Department of Animal Breeding & Genetics — SLU *<br clear="none">> > * Box 7023, SE-750 07                            *<br clear="none">> > * Uppsala, Sweden                                *<br clear="none">> > **************************************************<br clear="none">> > * Visiting address:                              *<br clear="none">> > * Room 55614, Ulls väg 26, Ultuna                *<br clear="none">> > * Uppsala                                        *<br clear="none">> > * Sweden                                        *<br clear="none">> > *                                                *<br clear="none">> > * Tel: +46-(0)18-67 1962                        *<br clear="none">> > *                                                *<br clear="none">> > **************************************************<br clear="none">> > *    ISO 9001 Bureau Veritas No SE004561-1      *<br clear="none">> > **************************************************<br clear="none">> > ---<br clear="none">> > När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <<a shape="rect" href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br clear="none">> > E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <<a shape="rect" href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br clear="none">><br clear="none">> --<br clear="none">> **************************************************<br clear="none">> * Marcus Pedersén                                *<br clear="none">> * System administrator                           *<br clear="none">> **************************************************<br clear="none">> * Interbull Centre                               *<br clear="none">> * ================                               *<br clear="none">> * Department of Animal Breeding & Genetics — SLU *<br clear="none">> * Box 7023, SE-750 07                            *<br clear="none">> * Uppsala, Sweden                                *<br clear="none">> **************************************************<br clear="none">> * Visiting address:                              *<br clear="none">> * Room 55614, Ulls väg 26, Ultuna                *<br clear="none">> * Uppsala                                        *<br clear="none">> * Sweden                                         *<br clear="none">> *                                                *<br clear="none">> * Tel: +46-(0)18-67 1962                         *<br clear="none">> *                                                *<br clear="none">> **************************************************<br clear="none">> *     ISO 9001 Bureau Veritas No SE004561-1      *<br clear="none">> **************************************************<br clear="none">> ---<br clear="none">> När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <<a shape="rect" href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br clear="none">> E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <<a shape="rect" href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br clear="none">> ________<br clear="none">><br clear="none">><br clear="none">><br clear="none">> Community Meeting Calendar:<br clear="none">><br clear="none">> Schedule -<br clear="none">> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">> Bridge: <a shape="rect" href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br clear="none">> Gluster-users mailing list<br clear="none">> <a shape="rect" ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br clear="none">> <a shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><div class="yqt4787803632" id="yqtfd03773"><br clear="none"><br clear="none">--<br clear="none">**************************************************<br clear="none">* Marcus Pedersén                                *<br clear="none">* System administrator                           *<br clear="none">**************************************************<br clear="none">* Interbull Centre                               *<br clear="none">* ================                               *<br clear="none">* Department of Animal Breeding & Genetics — SLU *<br clear="none">* Box 7023, SE-750 07                            *<br clear="none">* Uppsala, Sweden                                *<br clear="none">**************************************************<br clear="none">* Visiting address:                              *<br clear="none">* Room 55614, Ulls väg 26, Ultuna                *<br clear="none">* Uppsala                                        *<br clear="none">* Sweden                                         *<br clear="none">*                                                *<br clear="none">* Tel: +46-(0)18-67 1962                         *<br clear="none">*                                                *<br clear="none">**************************************************<br clear="none">*     ISO 9001 Bureau Veritas No SE004561-1      *<br clear="none">**************************************************<br clear="none">---<br clear="none">När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <<a shape="rect" href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br clear="none">E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <<a shape="rect" href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br clear="none">________<br clear="none"><br clear="none"><br clear="none"><br clear="none">Community Meeting Calendar:<br clear="none"><br clear="none">Schedule -<br clear="none">Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">Bridge: <a shape="rect" href="https://meet.google.com/cpu-eiue-hvk" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br clear="none">Gluster-users mailing list<br clear="none"><a shape="rect" ymailto="mailto:Gluster-users@gluster.org" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br clear="none"><a shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br clear="none"></div> </div> </blockquote></div>