[Gluster-users] Help: Client application cannot open existing gluster file and write over it
jrwillen at aol.com
jrwillen at aol.com
Wed Sep 30 21:25:49 UTC 2020
Problem: Although there are indications of trouble in the log files, gluster appears normal on boot. Clients can copy files to the distributed file system, but disconnect when an application tries to open an existing file and write over it.
I hope this is a newbie installation error, but I'm tired of pounding my head against the wall. I started with a much more complex application that had file-access problems and came up with this simple example. Thanks in advance for any guidance about how to address this problem.
Here is a short list of my observations, followed by more details for each one:
1) At boot, lots of complaints in the file system log2) At boot, glusterd log warns of errors through RPC layer and peers being disconnected3) Client A can copy a directory to the gluster file system4) Client A can write a gluster file from an application5) Client B verifies the write6) Client A tries to open the file and overwrite, but crashes7) Client A is now disconnected from gluster; sees core dump in root8) After rebooting client A, file system log provides some details of the crash.9) glusterd log doesn't provide any additional details10) Gluster was installed via apt, not compiled from source11) Running Raspbian GNU/Linux 10 (buster). 32 RPI-4 clients. Testing with 2 RPI-3b+ servers. 12) Clients see the 4 Tb distributed storage under /gfdata13) At boot, clusterd isn't always ready when the clients get to the point of mounting disks, so I forcibly mount them with sudo mount -a14) The gluster volume looks sensible at boot time15) The pool looks okay too
=================================Details=================================
1) At boot, lots of complaints in the file system log:pi at pi00:~ $ sudo cat /var/log/glusterfs/gfdata.log[2020-09-30 11:57:58.596738] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing[2020-09-30 11:57:59.049660] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing[2020-09-30 11:57:59.051730] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuingThe message "I [MSGID: 100011] [glusterfsd.c:1570:reincarnate] 0-glusterfsd: Fetching the volume file from server..." repeated 2 times between [2020-09-30 11:57:57.312930] and [2020-09-30 11:57:57.685434][2020-09-30 12:08:07.427162] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:08:07.427319] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:08:07.429373] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:08:17.161867] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:08:17.161877] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:08:17.163889] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:10:54.285700] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:10:54.285853] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:10:54.287936] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:10:54.701882] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in (null) (gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088). Holes=1 overlaps=0[2020-09-30 12:10:54.702033] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in (null) (gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6). Holes=1 overlaps=0[2020-09-30 12:10:54.702176] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in (null) (gfid = e81dc1be-b7f7-4baf-8049-1122c5844019). Holes=1 overlaps=0[2020-09-30 12:10:55.081494] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /examples (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0[2020-09-30 12:10:55.106417] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0[2020-09-30 12:10:55.167731] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test3 (gfid = 00000000-0000-0000-0000-000000000000). Holes=1 overlaps=0[2020-09-30 12:11:06.349404] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:11:06.361029] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test3: Disk layout missing, gfid = e81dc1be-b7f7-4baf-8049-1122c5844019[2020-09-30 12:11:06.361248] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test3, gfid = e81dc1be-b7f7-4baf-8049-1122c5844019[2020-09-30 12:11:06.364042] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test3 (gfid = e81dc1be-b7f7-4baf-8049-1122c5844019). Holes=1 overlaps=0[2020-09-30 12:11:06.963973] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:06.964619] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.014836] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.014970] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.188570] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.189452] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.192062] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.192346] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.222268] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.222880] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.225861] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.227039] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.227411] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.227699] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.228019] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:11:07.231825] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-sizeThe message "I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001" repeated 2 times between [2020-09-30 12:10:54.285853] and [2020-09-30 12:11:07.410603][2020-09-30 12:11:07.412422] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0The message "I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001" repeated 3 times between [2020-09-30 12:10:54.285700] and [2020-09-30 12:11:10.533318][2020-09-30 12:11:10.533328] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:11:10.551494] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:11:10.574455] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /examples: Disk layout missing, gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088[2020-09-30 12:11:10.574620] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /examples, gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088[2020-09-30 12:11:10.577251] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /examples (gfid = 4202cc06-0ec3-45f7-914a-b7d9fc872088). Holes=1 overlaps=0[2020-09-30 12:11:10.592811] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test: Disk layout missing, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6[2020-09-30 12:11:10.592927] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6[2020-09-30 12:11:10.595673] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test (gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6). Holes=1 overlaps=0[2020-09-30 12:14:35.163945] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:14:35.164099] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:14:35.166219] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:14:35.178003] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test: Disk layout missing, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6[2020-09-30 12:14:35.178111] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test, gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6[2020-09-30 12:14:35.180824] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test (gfid = a8a9efaf-2493-49ff-926c-1b0abb6107c6). Holes=1 overlaps=0[2020-09-30 12:14:35.694281] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.694455] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.695229] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.698612] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.698766] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:35.699413] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.184500] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.191364] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.192896] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.193355] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.193607] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:14:36.195633] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:17:39.677016] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:17:39.677160] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:17:39.679343] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
2) On boot, glusterd log warns of errors through RPC layer and peers being disconnectedpi at pi00:~ $ sudo cat /var/log/glusterfs/glusterd.log[2020-09-30 11:57:57.678768] E [MSGID: 106167] [glusterd-handshake.c:1996:__glusterd_mgmt_hndsk_version_cbk] 0-management: Error through RPC layer, retry again later[2020-09-30 11:57:57.678999] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(2)), xid = 0x5, unique = 0, sent = 2020-09-29 21:30:38.833820, timeout = 600 for 192.168.1.79:24007[2020-09-30 11:57:57.679015] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8004fcb7-2a9f-481a-8e2d-79409ea39375[2020-09-30 11:57:58.276231] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:57:59.341029] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Gluster MGMT Handshake), op(MGMT-VERS-ACK(2)), xid = 0x4, unique = 0, sent = 2020-09-29 21:30:47.473226, timeout = 600 for 192.168.1.35:24007[2020-09-30 11:57:59.341224] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 83800431-f92d-4c2b-a402-160bdd4a4935[2020-09-30 11:57:59.341328] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:57:59.379583] E [MSGID: 106167] [glusterd-handshake.c:1885:__glusterd_mgmt_hndsk_version_ack_cbk] 0-management: Error through RPC layer, retry again later[2020-09-30 11:57:59.379769] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(2)), xid = 0x5, unique = 0, sent = 2020-09-29 21:30:45.579517, timeout = 600 for 192.168.1.19:24007[2020-09-30 11:57:59.379886] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 03d20361-606d-4040-8b70-76473821706a[2020-09-30 11:57:59.379945] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (5)[2020-09-30 11:57:59.380004] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.19:24007[2020-09-30 11:57:59.380154] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.19> (<95e08fe4-534e-4388-94f9-c4db97b51795>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:57:59.380374] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:57:59.380419] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:00.137761] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a[2020-09-30 11:58:00.379959] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x7, unique = 0, sent = 2020-09-29 21:30:36.588912, timeout = 600 for 192.168.1.61:24007[2020-09-30 11:58:04.189199] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.189357] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.189444] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:36.557644, timeout = 600 for 192.168.1.61:24007[2020-09-30 11:58:04.189506] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.189574] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.189686] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:42.471874, timeout = 600 for 192.168.1.13:24007[2020-09-30 11:58:04.189744] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.189863] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.189956] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:48.137287, timeout = 600 for 192.168.1.52:24007[2020-09-30 11:58:04.190015] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.190083] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.190213] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x7, unique = 0, sent = 2020-09-29 21:30:33.681570, timeout = 600 for 192.168.1.48:24007[2020-09-30 11:58:04.190271] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.190339] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.190881] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:47.454357, timeout = 600 for 192.168.1.15:24007[2020-09-30 11:58:04.190946] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.191018] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.191150] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(2)), xid = 0x5, unique = 0, sent = 2020-09-29 21:30:49.384128, timeout = 600 for 192.168.1.69:24007[2020-09-30 11:58:04.191239] E [rpc-clnt.c:180:call_bail] 0-management: bailing out frame type(Peer mgmt), op(--(4)), xid = 0x6, unique = 0, sent = 2020-09-29 21:30:49.399827, timeout = 600 for 192.168.1.20:24007[2020-09-30 11:58:04.191295] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.191362] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.191614] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.192070] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.20:24007[2020-09-30 11:58:04.192233] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.20> (<ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.192628] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.192718] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.193072] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.27> (<1e90f804-99fa-43b8-8df3-07d5da49c0da>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.193768] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.194105] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.194398] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (7)[2020-09-30 11:58:04.194964] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.48:24007[2020-09-30 11:58:04.195226] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.48> (<40e7d4fe-702d-4b5d-93f0-cf2ff4880f27>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.195604] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.195686] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.196108] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:04.223503] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.223944] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.79 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.267435] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.268218] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (5)[2020-09-30 11:58:04.268377] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.79:24007[2020-09-30 11:58:04.268513] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.79> (<1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.268914] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.268994] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.269128] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.269189] E [rpc-clnt.c:346:saved_frames_unwind] (null) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2020-09-30 11:58:04.246959 (xid=0x6)[2020-09-30 11:58:04.269251] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.269321] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.269608] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.269754] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.52:24007[2020-09-30 11:58:04.269920] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.52> (<6beeff80-a573-4571-b83e-2c14e3add80a>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.270294] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.270376] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.271032] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:04.292492] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.292896] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.53 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.329487] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9[2020-09-30 11:58:04.346693] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.347000] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.13 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.420122] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:04.455643] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.455779] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.15:24007[2020-09-30 11:58:04.455902] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.15> (<8b612b17-7f2b-47b5-a27a-c1e87b845241>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.456220] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.456293] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.456643] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:04.472699] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.473023] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.70 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.528589] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.70> (<8f357d63-f96f-4a09-b3a7-b128e4780419>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.528934] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.528993] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.529256] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.35> (<99f71a38-0935-46a3-853e-ab09a1a4903a>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.529477] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.529642] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.530071] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.531818] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.61:24007[2020-09-30 11:58:04.531933] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.61> (<75306501-6238-4f4e-b562-74c6b10a614a>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.532228] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.532293] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.532875] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a[2020-09-30 11:58:04.552980] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.553292] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.65 (0), ret: 0, op_ret: 0[2020-09-30 11:58:04.599958] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (5)[2020-09-30 11:58:04.600235] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.69:24007[2020-09-30 11:58:04.601589] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.69> (<d98ebd55-514c-4c00-ab5b-7d06ba7b402d>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.601987] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.602067] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.602464] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:04.669632] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c, host: 192.168.1.53, port: 0[2020-09-30 11:58:04.728815] C [rpc-clnt.c:437:rpc_clnt_fill_request_info] 0-management: cannot lookup the saved frame corresponding to xid (6)[2020-09-30 11:58:04.728949] W [socket.c:2156:__socket_read_reply] 0-management: notify for event MAP_XID failed for 192.168.1.13:24007[2020-09-30 11:58:04.729039] I [MSGID: 106004] [glusterd-handler.c:6430:__glusterd_peer_rpc_notify] 0-management: Peer <192.168.1.13> (<4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9>), in state <Peer in Cluster>, has disconnected from glusterd.[2020-09-30 11:58:04.729308] W [MSGID: 106565] [glusterd-locks.c:795:glusterd_mgmt_v3_unlock] 0-management: Lock for vol data not held[2020-09-30 11:58:04.729357] W [MSGID: 106117] [glusterd-handler.c:6451:__glusterd_peer_rpc_notify] 0-management: Lock not released for data[2020-09-30 11:58:04.729445] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:04.729481] E [rpc-clnt.c:346:saved_frames_unwind] (null) 0-management: forced unwinding frame type(Peer mgmt) op(--(4)) called at 2020-09-30 11:58:04.402849 (xid=0x7)[2020-09-30 11:58:04.729519] E [MSGID: 106157] [glusterd-rpc-ops.c:665:__glusterd_friend_update_cbk] 0-management: RPC Error[2020-09-30 11:58:04.729562] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received RJT from uuid: 00000000-0000-0000-0000-000000000000[2020-09-30 11:58:04.729844] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 6ee2eb12-592f-486c-83dd-b9508066fe51, host: 192.168.1.47, port: 0[2020-09-30 11:58:04.754469] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 6ee2eb12-592f-486c-83dd-b9508066fe51[2020-09-30 11:58:04.959493] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:05.586162] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:05.869649] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:06.643933] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:06.975254] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:14.481239] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: b93ce138-cce4-4a95-994a-319a0387efb3[2020-09-30 11:58:14.544391] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:14.544614] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.68 (0), ret: 0, op_ret: 0[2020-09-30 11:58:14.614575] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8004fcb7-2a9f-481a-8e2d-79409ea39375[2020-09-30 11:58:14.614718] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 6ee2eb12-592f-486c-83dd-b9508066fe51[2020-09-30 11:58:14.614809] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:14.663601] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9[2020-09-30 11:58:14.903834] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:15.739950] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a[2020-09-30 11:58:16.030240] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:16.564846] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d[2020-09-30 11:58:16.565899] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ab97b2b5-2e15-4358-86d0-62324b93007a[2020-09-30 11:58:16.566003] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:16.566134] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:16.777107] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:17.278005] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:17.278249] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.69 (0), ret: 0, op_ret: 0[2020-09-30 11:58:18.515292] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: b93ce138-cce4-4a95-994a-319a0387efb3[2020-09-30 11:58:18.666299] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:18.949557] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:18.956347] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:18.956620] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.19 (0), ret: 0, op_ret: 0[2020-09-30 11:58:19.011957] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: b93ce138-cce4-4a95-994a-319a0387efb3[2020-09-30 11:58:19.014177] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:19.014935] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:19.132556] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:19.464173] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:19.516527] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:19.543397] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:19.549901] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:19.550132] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.61 (0), ret: 0, op_ret: 0[2020-09-30 11:58:19.576012] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9, host: 192.168.1.13, port: 0[2020-09-30 11:58:19.623547] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 75306501-6238-4f4e-b562-74c6b10a614a, host: 192.168.1.61, port: 0[2020-09-30 11:58:19.659552] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241, host: 192.168.1.15, port: 0[2020-09-30 11:58:19.669103] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9[2020-09-30 11:58:19.759223] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:21.934219] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:22.112225] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:22.516139] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd, host: 192.168.1.20, port: 0[2020-09-30 11:58:22.532749] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:22.729363] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:43.939426] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd[2020-09-30 11:58:43.986065] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:44.452170] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 1e90f804-99fa-43b8-8df3-07d5da49c0da, host: 192.168.1.27, port: 0[2020-09-30 11:58:44.463225] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 6beeff80-a573-4571-b83e-2c14e3add80a, host: 192.168.1.52, port: 0[2020-09-30 11:58:44.474079] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1e90f804-99fa-43b8-8df3-07d5da49c0da[2020-09-30 11:58:44.539423] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:45.327726] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 6beeff80-a573-4571-b83e-2c14e3add80a[2020-09-30 11:58:45.569036] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:45.810147] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:45.810337] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a, host: 192.168.1.35, port: 0[2020-09-30 11:58:45.820901] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:45.821070] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:45.821073] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:45.830232] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9[2020-09-30 11:58:45.830418] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:45.830615] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d[2020-09-30 11:58:45.907496] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:46.092804] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:46.093769] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:46.101145] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:46.101300] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.15 (0), ret: 0, op_ret: 0[2020-09-30 11:58:46.120581] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:46.188794] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:46.404964] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: ae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd[2020-09-30 11:58:46.405549] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 6beeff80-a573-4571-b83e-2c14e3add80a[2020-09-30 11:58:46.405686] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1e90f804-99fa-43b8-8df3-07d5da49c0da[2020-09-30 11:58:46.406132] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 75306501-6238-4f4e-b562-74c6b10a614a[2020-09-30 11:58:46.406493] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:46.415743] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:46.492945] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:46.681793] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8b612b17-7f2b-47b5-a27a-c1e87b845241[2020-09-30 11:58:46.682456] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:46.690024] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:46.690306] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.70 (0), ret: 0, op_ret: 0[2020-09-30 11:58:46.747693] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a[2020-09-30 11:58:46.747909] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a[2020-09-30 11:58:46.756824] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:46.757107] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.35 (0), ret: 0, op_ret: 0[2020-09-30 11:58:46.778487] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:46.849136] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:47.063765] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795, host: 192.168.1.19, port: 0[2020-09-30 11:58:47.073740] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 40e7d4fe-702d-4b5d-93f0-cf2ff4880f27, host: 192.168.1.48, port: 0[2020-09-30 11:58:47.084813] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419, host: 192.168.1.70, port: 0[2020-09-30 11:58:47.095474] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:47.183420] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:47.396429] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d, host: 192.168.1.69, port: 0[2020-09-30 11:58:47.407953] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a[2020-09-30 11:58:47.439796] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:47.674369] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 40e7d4fe-702d-4b5d-93f0-cf2ff4880f27[2020-09-30 11:58:47.780995] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:48.012688] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:48.076704] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:48.267824] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d[2020-09-30 11:58:48.343564] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:48.548045] I [MSGID: 106493] [glusterd-rpc-ops.c:468:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e, host: 192.168.1.79, port: 0[2020-09-30 11:58:48.559020] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 99f71a38-0935-46a3-853e-ab09a1a4903a[2020-09-30 11:58:48.559193] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 8f357d63-f96f-4a09-b3a7-b128e4780419[2020-09-30 11:58:48.559320] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 40e7d4fe-702d-4b5d-93f0-cf2ff4880f27[2020-09-30 11:58:48.559441] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 95e08fe4-534e-4388-94f9-c4db97b51795[2020-09-30 11:58:48.559670] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:48.631854] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:48.879914] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d98ebd55-514c-4c00-ab5b-7d06ba7b402d[2020-09-30 11:58:48.880121] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:48.912996] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:48.939447] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:48.939731] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.53 (0), ret: 0, op_ret: 0[2020-09-30 11:58:49.049668] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:49.263217] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:49.728404] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 3d64bd36-4078-49af-b251-07d9cbfddf9c[2020-09-30 11:58:50.950398] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:50.951296] I [MSGID: 106163] [glusterd-handshake.c:1389:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 50400[2020-09-30 11:58:50.969707] I [MSGID: 106490] [glusterd-handler.c:2586:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:50.985716] W [MSGID: 101088] [common-utils.c:4398:gf_backtrace_save] 0-management: Failed to save the backtrace.[2020-09-30 11:58:50.986057] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 192.168.1.79 (0), ret: 0, op_ret: 0[2020-09-30 11:58:51.017655] I [MSGID: 106492] [glusterd-handler.c:2771:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 11:58:51.233916] I [MSGID: 106502] [glusterd-handler.c:2812:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend[2020-09-30 11:58:51.724591] I [MSGID: 106493] [glusterd-rpc-ops.c:681:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e[2020-09-30 12:09:07.963638] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2020-09-30 12:09:07.964701] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2020-09-30 12:09:37.014817] I [MSGID: 106487] [glusterd-handler.c:1497:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
3) The bit about peers disconnecting from glusterd seems to be a red herring:pi at pi00:~ $ ssh 192.168.1.15 'df 'Filesystem 1K-blocks Used Available Use% Mounted on/dev/root 30491968 2517040 26703808 9% /devtmpfs 829976 0 829976 0% /devtmpfs 962072 0 962072 0% /dev/shmtmpfs 962072 9564 952508 1% /runtmpfs 5120 4 5116 1% /run/locktmpfs 962072 0 962072 0% /sys/fs/cgroup/dev/mmcblk0p1 258096 54605 203491 22% /boot192.168.1.41:/data 3907024444 39370916 3867653528 2% /gfdatatmpfs 192412 0 192412 0% /run/user/1000
3) Client A can copy a directory to the gluster file systempi at pi00:~ $ ls test3a.out filetest.f95 filetest.f95~pi at pi00:~ $ cat test3/filetest.f95program mainimplicit noneinteger:: i, n, nmaxreal:: x(1024)!! Try banging on a gluster file.!nmax = 1do n=1, nmax do i=1, 1024 x(i) = float(n)*float(i) enddo open(10, file='testfile', form='unformatted') write(10) x close(10) write(6,*) 'n = ', n enddostopendpi at pi00:~ $ cp -r test3 /gfdata
4) Client A can write a gluster file from an applicationpi at pi00:~ $ cd /gfdata/test3pi at pi00:/gfdata/test3 $ dateWed 30 Sep 07:31:07 CDT 2020pi at pi00:/gfdata/test3 $ ./a.out n = 1
5) Client B verifies the writepi at pi10:~ $ ls -altr /gfdata/test3total 22-rwxrwxrwx 1 root root 8684 Sep 30 07:29 a.out-rwxrwxrwx 1 root root 291 Sep 30 07:29 filetest.f95-rwxrwxrwx 1 root root 265 Sep 30 07:29 filetest.f95~-rwxrwxrwx 1 root root 4104 Sep 30 07:31 testfiledrwxrwxrwx 1 root root 4096 Sep 30 07:31 ..drwxrwxrwx 1 root root 4096 Sep 30 07:31 .pi at pi10:~ $ od -f /gfdata/test3/testfile0000000 5.74e-42 1 2 30000020 4 5 6 7...0007760 1020 1021 1022 10230010000 1024 5.74e-420010010pi at pi10:~ $
6) Client A tries to open the file and write again, but crashespi at pi00:/gfdata/test3 $ dateWed 30 Sep 07:31:46 CDT 2020pi at pi00:/gfdata/test3 $ ./a.outAt line 14 of file filetest.f95 (unit = 10, file = 'testfile')Fortran runtime error: Software caused connection abort
Error termination. Backtrace:
7) Client A is now disconnected from gluster. Sees core dump in rootpi at pi00:/gfdata/test3 $ ls /ls: cannot access '/gfdata': Transport endpoint is not connectedbin core etc home lost+found mnt proc run srv sys usrboot dev gfdata lib media opt root sbin ssh tmp varpi at pi00:/gfdata/test3 $ dfdf: /gfdata: Transport endpoint is not connectedFilesystem 1K-blocks Used Available Use% Mounted on/dev/root 30491968 3982908 25237940 14% /devtmpfs 829976 0 829976 0% /devtmpfs 962072 0 962072 0% /dev/shmtmpfs 962072 9580 952492 1% /runtmpfs 5120 4 5116 1% /run/locktmpfs 962072 0 962072 0% /sys/fs/cgroup/dev/mmcblk0p1 258096 54612 203484 22% /boottmpfs 192412 0 192412 0% /run/user/1000
8) After reboot, file system log provides some details of the crashpi at pi00:~ $ sudo cat /var/log/glusterfs/gfdata.log...[2020-09-30 12:29:44.427052] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:29:44.427212] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:29:44.429319] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:29:44.472259] E [MSGID: 114031] [client-rpc-fops_v2.c:216:client4_0_mkdir_cbk] 0-data-client-1: remote operation failed. Path: /test3 [No data available][2020-09-30 12:29:44.480524] W [MSGID: 114031] [client-rpc-fops_v2.c:1985:client4_0_setattr_cbk] 0-data-client-1: remote operation failed [Stale file handle]The message "I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0" repeated 2 times between [2020-09-30 12:29:44.429319] and [2020-09-30 12:31:02.580946][2020-09-30 12:31:02.592738] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /test3: Disk layout missing, gfid = 9873ee11-e2e6-4ee8-9322-3401863dfa8f[2020-09-30 12:31:02.592966] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /test3, gfid = 9873ee11-e2e6-4ee8-9322-3401863dfa8f[2020-09-30 12:31:02.595817] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in /test3 (gfid = 9873ee11-e2e6-4ee8-9322-3401863dfa8f). Holes=1 overlaps=0[2020-09-30 12:31:14.008297] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:14.009026] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-sizeThe message "I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001" repeated 2 times between [2020-09-30 12:29:44.427052] and [2020-09-30 12:31:02.578949]The message "I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001" repeated 2 times between [2020-09-30 12:29:44.427212] and [2020-09-30 12:31:02.578960][2020-09-30 12:31:48.616479] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:48.616780] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:48.627752] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:48.628042] E [stripe-helpers.c:346:stripe_ctx_handle] 0-data-stripe-0: Failed to get stripe-size[2020-09-30 12:31:48.635271] E [stripe.c:2949:stripe_ftruncate] 0-data-stripe-0: no stripe countpending frames:frame : type(1) op(FTRUNCATE)frame : type(1) op(FTRUNCATE)frame : type(1) op(OPEN)frame : type(0) op(0)patchset: git://git.gluster.org/glusterfs.gitsignal received: 6time of crash:2020-09-30 12:31:48configuration details:argp 1backtrace 1dlfcn 1libpthread 1llistxattr 1setfsid 1spinlock 1epoll.h 1xattr.h 1st_atim.tv_nsec 1package-string: glusterfs 5.5---------[2020-09-30 12:42:21.300134] I [MSGID: 100030] [glusterfsd.c:2715:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 5.5 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=192.168.1.41 --volfile-id=/data /gfdata)[2020-09-30 12:42:21.398194] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1[2020-09-30 12:42:21.694558] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2[2020-09-30 12:42:21.704186] I [MSGID: 114020] [client.c:2358:notify] 0-data-client-0: parent translators are ready, attempting connect on transport[2020-09-30 12:42:21.708976] I [MSGID: 114020] [client.c:2358:notify] 0-data-client-1: parent translators are ready, attempting connect on transportFinal graph:+------------------------------------------------------------------------------+ 1: volume data-client-0 2: type protocol/client 3: option ping-timeout 42 4: option remote-host 192.168.1.41 5: option remote-subvolume /mnt/ArmorA60/data 6: option transport-type socket 7: option transport.address-family inet 8: option transport.tcp-user-timeout 0 9: option transport.socket.keepalive-time 20 10: option transport.socket.keepalive-interval 2 11: option transport.socket.keepalive-count 9 12: option send-gids true 13: end-volume 14: 15: volume data-client-1 16: type protocol/client 17: option ping-timeout 42 18: option remote-host 192.168.1.20 19: option remote-subvolume /mnt/ArmorA60/data 20: option transport-type socket 21: option transport.address-family inet 22: option transport.tcp-user-timeout 0 23: option transport.socket.keepalive-time 20 24: option transport.socket.keepalive-interval 2 25: option transport.socket.keepalive-count 9 26: option send-gids true 27: end-volume 28: 29: volume data-stripe-0 30: type cluster/stripe 31: subvolumes data-client-0 data-client-1 32: end-volume 33: 34: volume data-dht 35: type cluster/distribute 36: option lock-migration off 37: option force-migration off 38: subvolumes data-stripe-0 39: end-volume 40: 41: volume data-write-behind 42: type performance/write-behind 43: subvolumes data-dht 44: end-volume 45: 46: volume data-read-ahead 47: type performance/read-ahead 48: subvolumes data-write-behind 49: end-volume 50: 51: volume data-readdir-ahead 52: type performance/readdir-ahead 53: option parallel-readdir off 54: option rda-request-size 131072 55: option rda-cache-limit 10MB 56: subvolumes data-read-ahead 57: end-volume 58: 59: volume data-io-cache 60: type performance/io-cache 61: subvolumes data-readdir-ahead 62: end-volume 63: 64: volume data-quick-read 65: type performance/quick-read 66: subvolumes data-io-cache 67: end-volume 68: 69: volume data-open-behind 70: type performance/open-behind 71: subvolumes data-quick-read 72: end-volume 73: 74: volume data-md-cache 75: type performance/md-cache 76: subvolumes data-open-behind 77: end-volume 78: 79: volume data-io-threads 80: type performance/io-threads 81: subvolumes data-md-cache 82: end-volume 83: 84: volume data 85: type debug/io-stats 86: option log-level INFO 87: option latency-measurement off 88: option count-fop-hits off 89: subvolumes data-io-threads 90: end-volume 91: 92: volume meta-autoload 93: type meta 94: subvolumes data 95: end-volume 96:+------------------------------------------------------------------------------+[2020-09-30 12:42:21.718355] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-data-client-1: changing port to 49152 (from 0)[2020-09-30 12:42:21.718726] I [rpc-clnt.c:2042:rpc_clnt_reconfig] 0-data-client-0: changing port to 49152 (from 0)[2020-09-30 12:42:21.725344] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-data-client-1: Connected to data-client-1, attached to remote volume '/mnt/ArmorA60/data'.[2020-09-30 12:42:21.731976] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-data-client-0: Connected to data-client-0, attached to remote volume '/mnt/ArmorA60/data'.[2020-09-30 12:42:21.747464] I [fuse-bridge.c:4267:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.31[2020-09-30 12:42:21.747593] I [fuse-bridge.c:4878:fuse_graph_sync] 0-fuse: switched to graph 0[2020-09-30 12:42:21.760574] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0[2020-09-30 12:42:21.776484] I [MSGID: 109092] [dht-layout.c:744:dht_layout_dir_mismatch] 0-data-dht: /: Disk layout missing, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:42:21.776598] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 0-data-dht: Mismatching layouts for /, gfid = 00000000-0000-0000-0000-000000000001[2020-09-30 12:42:21.778843] I [MSGID: 109063] [dht-layout.c:659:dht_layout_normalize] 0-data-dht: Found anomalies in / (gfid = 00000000-0000-0000-0000-000000000001). Holes=1 overlaps=0
9) glusterd log doesn't provide any additional details...[2020-09-30 12:09:07.963638] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2020-09-30 12:09:07.964701] I [MSGID: 106488] [glusterd-handler.c:1558:__glusterd_handle_cli_get_volume] 0-management: Received get vol req[2020-09-30 12:09:37.014817] I [MSGID: 106487] [glusterd-handler.c:1497:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req[2020-09-30 12:41:50.915448] W [MSGID: 100032] [glusterfsd.c:1500:cleanup_and_exit] 0-: received signum (15), shutting down[2020-09-30 12:42:10.042885] I [MSGID: 100030] [glusterfsd.c:2715:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 5.5 (args: /usr/sbin/glusterd -p /run/glusterd.pid --log-level INFO)[2020-09-30 12:42:10.247943] I [MSGID: 106478] [glusterd.c:1435:init] 0-management: Maximum allowed open file descriptors set to 65536[2020-09-30 12:42:10.248270] I [MSGID: 106479] [glusterd.c:1491:init] 0-management: Using /var/lib/glusterd as working directory[2020-09-30 12:42:10.248356] I [MSGID: 106479] [glusterd.c:1497:init] 0-management: Using /var/run/gluster as pid file working directory[2020-09-30 12:42:10.313231] W [MSGID: 103071] [rdma.c:4475:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device][2020-09-30 12:42:10.313405] W [MSGID: 103055] [rdma.c:4774:init] 0-rdma.management: Failed to initialize IB Device[2020-09-30 12:42:10.313500] W [rpc-transport.c:339:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed[2020-09-30 12:42:10.314354] W [rpcsvc.c:1789:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed[2020-09-30 12:42:10.314450] E [MSGID: 106244] [glusterd.c:1798:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport[2020-09-30 12:42:20.719035] I [MSGID: 106513] [glusterd-store.c:2282:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 50400[2020-09-30 12:42:20.734156] I [MSGID: 106544] [glusterd.c:152:glusterd_uuid_init] 0-management: retrieved UUID: 315aa9d5-76a8-4221-9187-4727b283406a[2020-09-30 12:42:20.783134] I [MSGID: 106498] [glusterd-handler.c:3647:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0The message "I [MSGID: 106498] [glusterd-handler.c:3647:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0" repeated 32 times between [2020-09-30 12:42:20.783134] and [2020-09-30 12:42:20.817711][2020-09-30 12:42:20.817930] W [MSGID: 106061] [glusterd-handler.c:3453:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
10) Gluster was installed via apt, not compiled from source
11) Running Raspbian GNU/Linux 10 (buster). 32 RPI-4 clients, 2RPI-3b+ servers. I'll upgrade to 4 RPI-4 servers when I'm confidentgluster is working correctly.
12) Clients see the 4 Tb distributed storage under /gfdatapi at pi00:~ $ ls /bin dev gfdata lib media opt root sbin ssh tmp varboot etc home lost+found mnt proc run srv sys usrpi at pi00:~ $ dfFilesystem 1K-blocks Used Available Use% Mounted on/dev/root 30491968 3982860 25237988 14% /devtmpfs 829976 0 829976 0% /devtmpfs 962072 0 962072 0% /dev/shmtmpfs 962072 9580 952492 1% /runtmpfs 5120 4 5116 1% /run/locktmpfs 962072 0 962072 0% /sys/fs/cgroup/dev/mmcblk0p1 258096 54612 203484 22% /boot192.168.1.41:/data 3907024444 39371016 3867653428 2% /gfdatatmpfs 192412 0 192412 0% /run/user/1000pi at pi00:~ $
13) At boot, clusterd isn't always ready when the clients get to the point of mounting disks, so I forcibly mount them with sudo mount -a
pi at pi00:~/gluster-build $ head check1echo pi00 ; ls /gfdata/testssh pi at 192.168.1.79 'echo pi01 ; ls /gfdata/test'ssh pi at 192.168.1.69 'echo pi02 ; ls /gfdata/test'ssh pi at 192.168.1.70 'echo pi03 ; ls /gfdata/test'ssh pi at 192.168.1.65 'echo pi04 ; ls /gfdata/test'ssh pi at 192.168.1.66 'echo pi05 ; ls /gfdata/test'ssh pi at 192.168.1.64 'echo pi06 ; ls /gfdata/test'...
pi at pi00:~/gluster-build $ ./check1pi00a.out checkout.txt reset_aout test_mkdir testmpi.f testmpi.f90...pi10ls: cannot access '/gfdata/test': No such file or directorypi11ls: cannot access '/gfdata/test': No such file or directorypi12...
14) The gluster volume looks sensiblepi at pi00:~ $ sudo gluster volume info
Volume Name: dataType: StripeVolume ID: ee3fc5a6-edae-4896-b3af-a7bf2a2d33c1Status: StartedSnapshot Count: 0Number of Bricks: 1 x 2 = 2Transport-type: tcpBricks:Brick1: 192.168.1.41:/mnt/ArmorA60/dataBrick2: 192.168.1.20:/mnt/ArmorA60/dataOptions Reconfigured:nfs.disable: ontransport.address-family: inet
15) The pool looks okay toopi at pi00:~ $ sudo gluster pool listUUID Hostname State9709d800-4920-4e1b-b531-b952b2520323 192.168.1.14 Connected75306501-6238-4f4e-b562-74c6b10a614a 192.168.1.61 Connected4c1e76fd-0458-46b4-bd4a-cbb8dd9c508b 192.168.1.66 Connectede29d6f3a-8a32-4d20-9388-e2392054557b 192.168.1.34 Connected03d20361-606d-4040-8b70-76473821706a 192.168.1.43 Connecteddc400830-fef2-4039-85d3-92a3085d528e 192.168.1.41 Connectedab97b2b5-2e15-4358-86d0-62324b93007a 192.168.1.65 Connectedc56df059-0f40-4138-91be-e410a9b776db 192.168.1.32 Connected8f357d63-f96f-4a09-b3a7-b128e4780419 192.168.1.70 Connected1e90f804-99fa-43b8-8df3-07d5da49c0da 192.168.1.27 Connected8004fcb7-2a9f-481a-8e2d-79409ea39375 192.168.1.33 Connected4abeda01-1ad6-4bc0-927d-c7d0f2cff3c9 192.168.1.13 Connectedb3d4b8f9-9a56-4bed-b10b-a9bb895ab746 192.168.1.54 Connected6beeff80-a573-4571-b83e-2c14e3add80a 192.168.1.52 Connecteda40cc7c3-ee91-4ea9-9059-a89a3bd4fad2 192.168.1.55 Connectedb93ce138-cce4-4a95-994a-319a0387efb3 192.168.1.68 Connected83800431-f92d-4c2b-a402-160bdd4a4935 192.168.1.16 Connected8b612b17-7f2b-47b5-a27a-c1e87b845241 192.168.1.15 Connectede71a2ffe-0463-492c-a367-e6dc7a95845c 192.168.1.81 Connected603f4f57-e270-4a7d-a29f-6e2376aee8a1 192.168.1.29 Connected95e08fe4-534e-4388-94f9-c4db97b51795 192.168.1.19 Connectedd98ebd55-514c-4c00-ab5b-7d06ba7b402d 192.168.1.69 Connected0fac620b-761c-49c9-a019-9c996dd5c4a8 192.168.1.64 Connected6ee2eb12-592f-486c-83dd-b9508066fe51 192.168.1.47 Connectede239fc30-5f0b-447a-9742-fd5512fc7eac 192.168.1.60 Connected1ba42937-5aae-4d03-a8a0-cbcb4ea98f4e 192.168.1.79 Connected99f71a38-0935-46a3-853e-ab09a1a4903a 192.168.1.35 Connected096a50a2-2d86-450f-9f7d-929e91444e2e 192.168.1.28 Connectedae87a0cd-ee9a-43e5-99e6-40fdbb5b79fd 192.168.1.20 Connected3d64bd36-4078-49af-b251-07d9cbfddf9c 192.168.1.53 Connecteddaf3673f-91e6-4b11-b119-8a3be918cca7 192.168.1.44 Connected92728258-7422-41f3-b095-b733f858d49c 192.168.1.23 Connected40e7d4fe-702d-4b5d-93f0-cf2ff4880f27 192.168.1.48 Connected315aa9d5-76a8-4221-9187-4727b283406a localhost Connected
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200930/e0b2e424/attachment.html>
More information about the Gluster-users
mailing list