[Gluster-users] Gluster 11 upgrade, glusterd crash
Marcus Pedersén
marcus.pedersen at slu.se
Fri Mar 3 14:00:46 UTC 2023
Hi all,
I just started to upgrade from gluster 10.3 to gluster 11.
I started with my arbiter node and upgraded.
After upgrade gluster started after change in info file.
Rebooted machine and after that glusterd crashes.
I have double checked config in /var/lib/glusterd
and updated /var/lib/glusterd/vols/gv0/info
so the checksum is correct.
OS: debian bullseye (11)
gluster volume status (from onte of the other nodes that is not upgraded)
Status of volume: gv0
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick host2:/urd-gds/gv0 52902 0 Y 113172
Brick host1:/urd-gds/gv0 61235 0 Y 5487
Self-heal Daemon on localhost N/A N/A Y 5550
Self-heal Daemon on host2 N/A N/A Y 113236
Task Status of Volume gds-home
------------------------------------------------------------------------------
There are no active volume tasks
I do not know where to start looking,
enclosed is a boot part from glusterd.log
Thanks alot for your help!!
Regards
Marcus
--
**************************************************
* Marcus Pedersén *
* System administrator *
**************************************************
* Interbull Centre *
* ================ *
* Department of Animal Breeding & Genetics — SLU *
* Box 7023, SE-750 07 *
* Uppsala, Sweden *
**************************************************
* Visiting address: *
* Room 55614, Ulls väg 26, Ultuna *
* Uppsala *
* Sweden *
* *
* Tel: +46-(0)18-67 1962 *
* *
**************************************************
* ISO 9001 Bureau Veritas No SE004561-1 *
**************************************************
---
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka här <https://www.slu.se/om-slu/kontakta-slu/personuppgifter/>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <https://www.slu.se/en/about-slu/contact-slu/personal-data/>
-------------- next part --------------
[2023-03-03 13:46:01.248805 +0000] I [MSGID: 100030] [glusterfsd.c:2872:main] 0-/usr/sbin/glusterd: Started running version [{arg=/usr/sbin/glusterd}, {version=11.0}, {cmdlinestr=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO}]
[2023-03-03 13:46:01.249174 +0000] I [glusterfsd.c:2562:daemonize] 0-glusterfs: Pid of current running process is 4256
[2023-03-03 13:46:01.250935 +0000] I [MSGID: 0] [glusterfsd.c:1597:volfile_init] 0-glusterfsd-mgmt: volume not found, continuing with init
[2023-03-03 13:46:01.282327 +0000] I [MSGID: 106479] [glusterd.c:1660:init] 0-management: Using /var/lib/glusterd as working directory
[2023-03-03 13:46:01.282371 +0000] I [MSGID: 106479] [glusterd.c:1664:init] 0-management: Using /var/run/gluster as pid file working directory
[2023-03-03 13:46:01.288793 +0000] I [socket.c:973:__socket_server_bind] 0-socket.management: process started listening on port (24007)
[2023-03-03 13:46:01.291749 +0000] I [socket.c:916:__socket_server_bind] 0-socket.management: closing (AF_UNIX) reuse check socket 13
[2023-03-03 13:46:01.292549 +0000] I [MSGID: 106059] [glusterd.c:1923:init] 0-management: max-port override: 60999
[2023-03-03 13:46:01.338404 +0000] E [MSGID: 106061] [glusterd.c:597:glusterd_crt_georep_folders] 0-glusterd: Dict get failed [{Key=log-group}, {errno=2}, {error=No such file or directory}]
[2023-03-03 13:46:02.342898 +0000] I [MSGID: 106513] [glusterd-store.c:2198:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 100000
[2023-03-03 13:46:02.347302 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.347333 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.347341 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.347346 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.347351 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.351708 +0000] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: d5d7204c-aa42-4b7d-a559-f48f7917e98e
[2023-03-03 13:46:02.370758 +0000] I [MSGID: 106498] [glusterd-handler.c:3794:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2023-03-03 13:46:02.375132 +0000] I [MSGID: 106498] [glusterd-handler.c:3794:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2023-03-03 13:46:02.375188 +0000] W [MSGID: 106061] [glusterd-handler.c:3589:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2023-03-03 13:46:02.375250 +0000] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2023-03-03 13:46:02.376627 +0000] I [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2023-03-03 13:46:02.376618 +0000] W [MSGID: 106061] [glusterd-handler.c:3589:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2023-03-03 13:46:02.382039 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.382145 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.382177 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.382195 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.382211 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.386720 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.386720 +0000] and [2023-03-03 13:46:02.391550 +0000]
[2023-03-03 13:46:02.391576 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/9974c18368084c4b89bb7447c080d92b/brick3/gds-home [No such file or directory]
[2023-03-03 13:46:02.408551 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.408634 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.408670 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.408689 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.408705 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.413325 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.413325 +0000] and [2023-03-03 13:46:02.418058 +0000]
[2023-03-03 13:46:02.418083 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/d41f6dbb52d449d7a22554a7795d1938/brick3/gds-home [No such file or directory]
[2023-03-03 13:46:02.435395 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.435480 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.435517 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.435535 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.435551 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.440269 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.440269 +0000] and [2023-03-03 13:46:02.444922 +0000]
[2023-03-03 13:46:02.444953 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/2cfe90e27d2e4e25ab445b784f834aa8/brick3/gds-home [No such file or directory]
[2023-03-03 13:46:02.462175 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.462262 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.462295 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.462315 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.462331 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.466999 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.466999 +0000] and [2023-03-03 13:46:02.471797 +0000]
[2023-03-03 13:46:02.471832 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/78659cc641da4d19b18ccddc71b0620b/brick3/gds-home [No such file or directory]
[2023-03-03 13:46:02.489287 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.489376 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.489410 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.489429 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.489446 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.494318 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
[2023-03-03 13:46:02.514385 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.514468 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.514518 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.514537 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.514553 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.517199 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.517199 +0000] and [2023-03-03 13:46:02.521029 +0000]
[2023-03-03 13:46:02.521059 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/083893ba17734106a34b1092f1dc9273/brick3/gds-home [No such file or directory]
[2023-03-03 13:46:02.542478 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.542571 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.542606 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.542625 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.542641 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.545410 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.545410 +0000] and [2023-03-03 13:46:02.548239 +0000]
[2023-03-03 13:46:02.548263 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/071c7bfc377d4110aefad0c30fd37dc7/brick3/gds-home [No such file or directory]
[2023-03-03 13:46:02.569362 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.569447 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.569480 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.569498 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.569514 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.572205 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.572205 +0000] and [2023-03-03 13:46:02.574871 +0000]
[2023-03-03 13:46:02.574895 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/a12bff428d9240bfadcb1af85fe0bbfc/brick3/gds-home [No such file or directory]
[2023-03-03 13:46:02.596284 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.596390 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.596430 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.596466 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.596484 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.599265 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.599265 +0000] and [2023-03-03 13:46:02.602148 +0000]
[2023-03-03 13:46:02.624580 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: tier-enabled
[2023-03-03 13:46:02.624682 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: nfs.disable
[2023-03-03 13:46:02.624717 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-0
[2023-03-03 13:46:02.624742 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-1
[2023-03-03 13:46:02.624767 +0000] W [MSGID: 106204] [glusterd-store.c:3273:glusterd_store_update_volinfo] 0-management: Unknown key: brick-2
[2023-03-03 13:46:02.627593 +0000] E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path
The message "E [MSGID: 106204] [glusterd-store.c:2622:glusterd_store_retrieve_bricks] 0-management: Unknown key: device_path" repeated 2 times between [2023-03-03 13:46:02.627593 +0000] and [2023-03-03 13:46:02.630483 +0000]
[2023-03-03 13:46:02.630512 +0000] W [MSGID: 106425] [glusterd-store.c:2690:glusterd_store_retrieve_bricks] 0-management: failed to get statfs() call on brick /run/gluster/snaps/e49131c77f4848d49f0ff313c628e97c/brick3/gds-home [No such file or directory]
pending frames:
patchset: git://git.gluster.org/glusterfs.git
signal received: 11
time of crash:
2023-03-03 13:46:02 +0000
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 11.0
/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x28b24)[0x7f99dcd5fb24]
/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x6a8)[0x7f99dcd69108]
/lib/x86_64-linux-gnu/libc.so.6(+0x38d60)[0x7f99dcb30d60]
/usr/lib/x86_64-linux-gnu/glusterfs/11.0/xlator/mgmt/glusterd.so(+0x85ad0)[0x7f99d8552ad0]
/usr/lib/x86_64-linux-gnu/glusterfs/11.0/xlator/mgmt/glusterd.so(+0x88485)[0x7f99d8555485]
/usr/lib/x86_64-linux-gnu/glusterfs/11.0/xlator/mgmt/glusterd.so(+0x192ce)[0x7f99d84e62ce]
/lib/x86_64-linux-gnu/libglusterfs.so.0(xlator_init+0xc2)[0x7f99dcd5a452]
/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x51029)[0x7f99dcd88029]
/lib/x86_64-linux-gnu/libglusterfs.so.0(glusterfs_graph_activate+0x9d)[0x7f99dcd882dd]
/usr/sbin/glusterd(glusterfs_process_volfp+0x18f)[0x5652a9f6a4af]
/usr/sbin/glusterd(+0x10936)[0x5652a9f6a936]
/usr/sbin/glusterd(+0x12fb8)[0x5652a9f6cfb8]
/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x82c60)[0x7f99dcdb9c60]
/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x83247)[0x7f99dcdba247]
/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x8a2ae)[0x7f99dcdc12ae]
/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x849ae)[0x7f99dcdbb9ae]
/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x89736)[0x7f99dcdc0736]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7ea7)[0x7f99dccd4ea7]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f99dcbf4a2f]
---------
More information about the Gluster-users
mailing list