[Gluster-users] Rebalanced causing duplicate files

[Stephen Remde]

During a rebalance, errors occured for some files resulting in duplicate
files on the volume.

[2022-03-14 16:24:47.957556 +0000] I
[dht-rebalance.c:1536:dht_migrate_file] 0-dc4-dht:
/data/queues/label-full-defect-qa-20191107.json: attempting to move
from dc4-readdir-ahead-28 to dc4-readdir-ahead-16
[2022-03-14 16:24:48.053314 +0000] W [MSGID: 0]
[dht-rebalance.c:922:__dht_check_free_space] 0-dc4-dht: Write will
cross min-free-disk for file -
/data/queues/label-full-defect-qa-20191107.json on subvol -
dc4-readdir-ahead-16. Looking for new subvol
[2022-03-14 16:24:48.053455 +0000] I [MSGID: 0]
[dht-rebalance.c:979:__dht_check_free_space] 0-dc4-dht: new target
found - dc4-readdir-ahead-47 for file -
/data/queues/label-full-defect-qa-20191107.json
[2022-03-14 16:24:48.192489 +0000] I [MSGID: 0]
[dht-rebalance.c:1741:dht_migrate_file] 0-dc4-dht: destination for
file - /data/queues/label-full-defect-qa-20191107.json is changed to -
dc4-readdir-ahead-47
[2022-03-14 16:24:48.385145 +0000] E [MSGID: 109023]
[dht-rebalance.c:2014:dht_migrate_file] 0-dc4-dht: failed to set xattr
on /data/queues/label-full-defect-qa-20191107.json in
dc4-readdir-ahead-16 [Operation not supported]
[2022-03-14 16:24:48.388013 +0000] E [MSGID: 109023]
[dht-rebalance.c:2859:gf_defrag_migrate_single_file] 0-dc4-dht:
migrate-data failed for
/data/queues/label-full-defect-qa-20191107.json [Operation not
supported]

With the corresponding messages on the bricks logs:

/var/log/glusterfs/bricks/bricks-14c2df57-2751-4567-8849-5ae9786eb264-brick.log.1:[2022-03-14
16:24:47.687763 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/14c2df57-2751-4567-8849-5ae9786eb264/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]
/var/log/glusterfs/bricks/bricks-14c2df57-2751-4567-8849-5ae9786eb264-brick.log.1:[2022-03-14
16:24:48.567493 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/14c2df57-2751-4567-8849-5ae9786eb264/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]

/var/log/glusterfs/bricks/bricks-96870e54-4a0b-4ada-9339-fcde096c346a-brick.log.1:[2022-03-14
16:24:47.687588 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/96870e54-4a0b-4ada-9339-fcde096c346a/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]
/var/log/glusterfs/bricks/bricks-96870e54-4a0b-4ada-9339-fcde096c346a-brick.log.1:[2022-03-14
16:24:48.567201 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/96870e54-4a0b-4ada-9339-fcde096c346a/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]

/var/log/glusterfs/bricks/bricks-3ba68667-bd43-45e0-8e8e-b2538dcfeb96-brick.log.1:[2022-03-14
16:24:47.687312 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/3ba68667-bd43-45e0-8e8e-b2538dcfeb96/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]
/var/log/glusterfs/bricks/bricks-3ba68667-bd43-45e0-8e8e-b2538dcfeb96-brick.log.1:[2022-03-14
16:24:48.566871 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/3ba68667-bd43-45e0-8e8e-b2538dcfeb96/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]

/var/log/glusterfs/bricks/bricks-0e213dec-2b72-4dd1-94aa-85d94e5a3639-brick.log.1:[2022-03-14
16:24:47.686811 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/0e213dec-2b72-4dd1-94aa-85d94e5a3639/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]
/var/log/glusterfs/bricks/bricks-0e213dec-2b72-4dd1-94aa-85d94e5a3639-brick.log.1:[2022-03-14
16:24:48.566324 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/0e213dec-2b72-4dd1-94aa-85d94e5a3639/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]

/var/log/glusterfs/bricks/bricks-8e953fb6-c0d2-40e0-831d-f67965186915-brick.log.1:[2022-03-14
16:24:47.687322 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/8e953fb6-c0d2-40e0-831d-f67965186915/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]
/var/log/glusterfs/bricks/bricks-8e953fb6-c0d2-40e0-831d-f67965186915-brick.log.1:[2022-03-14
16:24:48.567089 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/8e953fb6-c0d2-40e0-831d-f67965186915/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]

/var/log/glusterfs/bricks/bricks-f84410a1-636b-48e5-b1c2-ff1ff87504a0-brick.log.1:[2022-03-14
16:24:47.687433 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/f84410a1-636b-48e5-b1c2-ff1ff87504a0/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]
/var/log/glusterfs/bricks/bricks-f84410a1-636b-48e5-b1c2-ff1ff87504a0-brick.log.1:[2022-03-14
16:24:48.567062 +0000] E [MSGID: 113001]
[posix-helpers.c:1273:posix_handle_pair] 0-dc4-posix:
/bricks/f84410a1-636b-48e5-b1c2-ff1ff87504a0/brick/.glusterfs/4a/d4/4ad4d646-6f36-4849-bd17-0d12fd67152d:
key:glusterfs.inodelk-dom-countflags: 0 length:16 [Operation not
supported]

My understanding is that this xattr (glusterfs.inodelk-dom-count) could not
be written (because it doesnt start with trusted, user, system, or
security) and that actually these should not be attempted as this is
filtered:
https://github.com/gluster/glusterfs/blob/v9.3/xlators/storage/posix/src/posix-helpers.c#L1853

I'm not sure how else the execution could have got to that point.

We have noticed some issues with the volume previously. Some directories
had inconsistent gfids (they were consistent across the subvolumes, but
about 15 of the subvolumes differed for some directories)

Any help appreciated

More info here https://github.com/gluster/glusterfs/issues/3309
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20220321/533658ef/attachment.html>