<div dir="auto"><p dir="auto" style="margin-top:0px;margin-bottom:16px;background-color:rgb(255,255,255)"><span style="color:rgb(36,41,47);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji"">During a rebalance, errors occured for some files resulting in duplicate files on the volume.</span><br></p><div style="color:rgb(36,41,47);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji";font-size:14px;background-color:rgb(255,255,255)" dir="auto"><pre style="font-family:'ui-monospace','sfmono-regular','sf mono','menlo','consolas','liberation mono',monospace;font-size:11.9px;margin-top:0px;margin-bottom:16px;padding:16px;line-height:1.45;border-radius:6px"><code style="font-family:ui-monospace,sfmono-regular,"sf mono",menlo,consolas,"liberation mono",monospace;padding:0px;margin:0px;background:transparent;border-radius:6px;border:0px;display:inline;line-height:inherit">[2022-03-14 16:24:47.957556 +0000] I [dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht: /data/queues/label-full-defect-qa-20191107.json: attempting to move from dc4-readdir-ahead-28 to dc4-readdir-ahead-16
[2022-03-14 16:24:48.053314 +0000] W [MSGID: 0] [dht-rebalance.c:922:__dht_check_free_space] 0-dc4-dht: Write will cross min-free-disk for file - /data/queues/label-full-defect-qa-20191107.json on subvol - dc4-readdir-ahead-16. Looking for new subvol
[2022-03-14 16:24:48.053455 +0000] I [MSGID: 0] [dht-rebalance.c:979:__dht_check_free_space] 0-dc4-dht: new target found - dc4-readdir-ahead-47 for file - /data/queues/label-full-defect-qa-20191107.json
[2022-03-14 16:24:48.192489 +0000] I [MSGID: 0] [dht-rebalance.c:1741:dht_migrate_file] 0-dc4-dht: destination for file - /data/queues/label-full-defect-qa-20191107.json is changed to - dc4-readdir-ahead-47
[2022-03-14 16:24:48.385145 +0000] E [MSGID: 109023] [dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr on /data/queues/label-full-defect-qa-20191107.json in dc4-readdir-ahead-16 [Operation not supported]
[2022-03-14 16:24:48.388013 +0000] E [MSGID: 109023] [dht-rebalance.c:2859:gf_defrag_migrate_single_file] 0-dc4-dht: migrate-data failed for /data/queues/label-full-defect-qa-20191107.json [Operation not supported]
</code></pre></div><p dir="auto" style="margin-top:0px;margin-bottom:16px;color:rgb(36,41,47);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji";background-color:rgb(255,255,255)">With the corresponding messages on the bricks logs:</p><div style="color:rgb(36,41,47);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji";font-size:14px;background-color:rgb(255,255,255)" dir="auto"><pre style="font-family:'ui-monospace','sfmono-regular','sf mono','menlo','consolas','liberation mono',monospace;font-size:11.9px;margin-top:0px;margin-bottom:16px;padding:16px;line-height:1.45;border-radius:6px"><code style="font-family:ui-monospace,sfmono-regular,"sf mono",menlo,consolas,"liberation mono",monospace;padding:0px;margin:0px;background:transparent;border-radius:6px;border:0px;display:inline;line-height:inherit">/var/log/glusterfs/bricks/bricks-14c2df57-2751-4567-8849-5ae9786eb264-brick.log.1:[2022-03-14 16:24:47.687763 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/14c2df57-2751-4567-8849-5ae9786eb264/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-14c2df57-2751-4567-8849-5ae9786eb264-brick.log.1:[2022-03-14 16:24:48.567493 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/14c2df57-2751-4567-8849-5ae9786eb264/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]

/var/log/glusterfs/bricks/bricks-96870e54-4a0b-4ada-9339-fcde096c346a-brick.log.1:[2022-03-14 16:24:47.687588 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/96870e54-4a0b-4ada-9339-fcde096c346a/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-96870e54-4a0b-4ada-9339-fcde096c346a-brick.log.1:[2022-03-14 16:24:48.567201 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/96870e54-4a0b-4ada-9339-fcde096c346a/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]

/var/log/glusterfs/bricks/bricks-3ba68667-bd43-45e0-8e8e-b2538dcfeb96-brick.log.1:[2022-03-14 16:24:47.687312 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/3ba68667-bd43-45e0-8e8e-b2538dcfeb96/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-3ba68667-bd43-45e0-8e8e-b2538dcfeb96-brick.log.1:[2022-03-14 16:24:48.566871 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/3ba68667-bd43-45e0-8e8e-b2538dcfeb96/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]

/var/log/glusterfs/bricks/bricks-0e213dec-2b72-4dd1-94aa-85d94e5a3639-brick.log.1:[2022-03-14 16:24:47.686811 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/0e213dec-2b72-4dd1-94aa-85d94e5a3639/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-0e213dec-2b72-4dd1-94aa-85d94e5a3639-brick.log.1:[2022-03-14 16:24:48.566324 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/0e213dec-2b72-4dd1-94aa-85d94e5a3639/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]

/var/log/glusterfs/bricks/bricks-8e953fb6-c0d2-40e0-831d-f67965186915-brick.log.1:[2022-03-14 16:24:47.687322 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/8e953fb6-c0d2-40e0-831d-f67965186915/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-8e953fb6-c0d2-40e0-831d-f67965186915-brick.log.1:[2022-03-14 16:24:48.567089 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/8e953fb6-c0d2-40e0-831d-f67965186915/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]

/var/log/glusterfs/bricks/bricks-f84410a1-636b-48e5-b1c2-ff1ff87504a0-brick.log.1:[2022-03-14 16:24:47.687433 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/f84410a1-636b-48e5-b1c2-ff1ff87504a0/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
/var/log/glusterfs/bricks/bricks-f84410a1-636b-48e5-b1c2-ff1ff87504a0-brick.log.1:[2022-03-14 16:24:48.567062 +0000] E [MSGID: 113001] [posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix: /bricks/f84410a1-636b-48e5-b1c2-ff1ff87504a0/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d: key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not supported]
</code></pre></div><p dir="auto" style="margin-top:0px;margin-bottom:16px;color:rgb(36,41,47);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji";background-color:rgb(255,255,255)">My understanding is that this xattr (glusterfs.inodelk-dom-count) could not be written (because it doesnt start with trusted, user, system, or security) and that actually these should not be attempted as this is filtered:<br><a href="https://github.com/gluster/glusterfs/blob/v9.3/xlators/storage/posix/src/posix-helpers.c#L1853" style="background-color:transparent;text-decoration-line:none">https://github.com/gluster/glusterfs/blob/v9.3/xlators/storage/posix/src/posix-helpers.c#L1853</a></p><p dir="auto" style="margin-top:0px;margin-bottom:16px;color:rgb(36,41,47);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji";background-color:rgb(255,255,255)">I'm not sure how else the execution could have got to that point.</p><p dir="auto" style="margin-top:0px;margin-bottom:16px;color:rgb(36,41,47);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji";background-color:rgb(255,255,255)">We have noticed some issues with the volume previously. Some directories had inconsistent gfids (they were consistent across the subvolumes, but about 15 of the subvolumes differed for some directories)</p><p dir="auto" style="margin-top:0px;margin-bottom:16px;color:rgb(36,41,47);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji";background-color:rgb(255,255,255)">Any help appreciated</p><p dir="auto" style="margin-top:0px;margin-bottom:16px;color:rgb(36,41,47);font-family:-apple-system,blinkmacsystemfont,"segoe ui",helvetica,arial,sans-serif,"apple color emoji","segoe ui emoji";background-color:rgb(255,255,255)">More info here <a href="https://github.com/gluster/glusterfs/issues/3309">https://github.com/gluster/glusterfs/issues/3309</a></p></div>