<div dir="ltr"><div>Hello Marcus,</div><div>Please provide below info for understanding the problem,</div><div>1. getxattr -d -m. -e hex fileName --> for any 2 of the files from all 3 nodes</div><div> getxattr -d -m. -e hex parentDirectoryName --> from all 3 nodes</div><div><br></div><div>This will confirm whether it is a GFID split brain case.</div><div><br></div><div>Thanks & Regards,<br>Chetan.</div><div><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
Message: 3<br>
Date: Wed, 6 Oct 2021 10:58:24 +0200<br>
From: Marcus Peders?n <<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>><br>
To: <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
Subject: [Gluster-users] Gluster heal problem<br>
Message-ID: <<a href="mailto:20211006085824.GA6893@slu.se" target="_blank">20211006085824.GA6893@slu.se</a>><br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi all,<br>
I have a problem with heal, I have 995 files that fails with heal.<br>
<br>
Gluster version: 9.3<br>
OS: Debian Bullseye<br>
<br>
My setup is a replicate with an arbiter:<br>
Volume Name: gds-admin<br>
Type: Replicate<br>
Volume ID: f1f112f4-8cee-4c04-8ea5-c7d895c8c8d6<br>
Status: Started<br>
Snapshot Count: 8<br>
Number of Bricks: 1 x (2 + 1) = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: urd-gds-001:/urd-gds/gds-admin<br>
Brick2: urd-gds-002:/urd-gds/gds-admin<br>
Brick3: urd-gds-000:/urd-gds/gds-admin (arbiter)<br>
Options Reconfigured:<br>
storage.build-pgfid: off<br>
performance.client-io-threads: off<br>
nfs.disable: on<br>
transport.address-family: inet<br>
storage.fips-mode-rchecksum: on<br>
features.barrier: disable<br>
<br>
Gluster volume status:<br>
Status of volume: gds-admin<br>
Gluster process TCP Port RDMA Port Online Pid<br>
------------------------------------------------------------------------------<br>
Brick urd-gds-001:/urd-gds/gds-admin 49155 0 Y 6964<br>
Brick urd-gds-002:/urd-gds/gds-admin 49155 0 Y 4270<br>
Brick urd-gds-000:/urd-gds/gds-admin 49152 0 Y 1175<br>
Self-heal Daemon on localhost N/A N/A Y 7031<br>
Self-heal Daemon on urd-gds-002 N/A N/A Y 4281<br>
Self-heal Daemon on urd-gds-000 N/A N/A Y 1230<br>
<br>
Task Status of Volume gds-admin<br>
------------------------------------------------------------------------------<br>
There are no active volume tasks<br>
<br>
<br>
Gluster pool list:<br>
UUID Hostname State<br>
8823d0d9-5d02-4f47-86e9- urd-gds-000 Connected<br>
73139305-08f5-42c2-92b6- urd-gds-002 Connected<br>
d612a705-8493-474e-9fdc- localhost Connected<br>
<br>
<br>
<br>
<br>
info summary says:<br>
Brick urd-gds-001:/urd-gds/gds-admin<br>
Status: Connected<br>
Total Number of entries: 995<br>
Number of entries in heal pending: 995<br>
Number of entries in split-brain: 0<br>
Number of entries possibly healing: 0<br>
<br>
Brick urd-gds-002:/urd-gds/gds-admin<br>
Status: Connected<br>
Total Number of entries: 0<br>
Number of entries in heal pending: 0<br>
Number of entries in split-brain: 0<br>
Number of entries possibly healing: 0<br>
<br>
Brick urd-gds-000:/urd-gds/gds-admin<br>
Status: Connected<br>
Total Number of entries: 995<br>
Number of entries in heal pending: 995<br>
Number of entries in split-brain: 0<br>
Number of entries possibly healing: 0<br>
<br>
<br>
<br>
Statistics says (on both node urd-gds-000 and urd-gds-001):<br>
Starting time of crawl: Tue Oct 5 14:25:08 2021<br>
<br>
Ending time of crawl: Tue Oct 5 14:25:25 2021<br>
<br>
Type of crawl: INDEX<br>
No. of entries healed: 0<br>
No. of entries in split-brain: 0<br>
No. of heal failed entries: 995<br>
<br>
<br>
To me it seems as if node urd-gds-002 has old version of files.<br>
I tried 2 files that had filenames and both urd-gds-000 and urd-gds-001<br>
has the same gfid for the file and the same timestamp for the file.<br>
Node urd-gds-002 has a different gfid and an older timestamp.<br>
The client could not access the file.<br>
I manually removed the file and gfid file from urd-gds-002 and these files<br>
were healed.<br>
<br>
I have a long list of files with just gfids (995).<br>
I tried to get the file path with (example):<br>
getfattr -n trusted.glusterfs.pathinfo -e text /mnt/gds-admin/.gfid/4e203eb1-795e-433a-9403-753ba56575fd<br>
getfattr: Removing leading '/' from absolute path names<br>
# file: mnt/gds-admin/.gfid/4e203eb1-795e-433a-9403-753ba56575fd<br>
trusted.glusterfs.pathinfo="(<REPLICATE:gds-admin-replicate-0> <POSIX(/urd-gds/gds-admin):urd-gds-000:/urd-gds/gds-admin/.glusterfs/30/70/3070276f-1096-44c8-b9e9-62625620aba3/04> <POSIX(/urd-gds/gds-admin):urd-gds-001:/urd-gds/gds-admin/.glusterfs/30/70/3070276f-1096-44c8-b9e9-62625620aba3/04>)"<br>
<br>
This tells me that the file exists on node urd-gds-000 and urd-gds-001.<br>
<br>
I have been looking through the glustershd.log and I see the similar error<br>
over and over again on urd-gds-000 and urd-gds-001:<br>
[2021-10-05 12:46:01.095509 +0000] I [MSGID: 108026] [afr-self-heal-entry.c:1052:afr_selfheal_entry_do] 0-gds-admin-replicate-0: performing entry selfheal on d0d8b20e-c9df-4b8b-ac2e-24697fdf9201<br>
[2021-10-05 12:46:01.802920 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:46:01.803538 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:46:01.803612 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:46:01.908395 +0000] I [MSGID: 108026] [afr-self-heal-entry.c:1052:afr_selfheal_entry_do] 0-gds-admin-replicate-0: performing entry selfheal on 0e309af2-2538-440a-8fd0-392620e83d05<br>
[2021-10-05 12:46:01.914909 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:46:01.915225 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:46:01.915230 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
<br>
On urd-gds-002 I get same error over and over again:<br>
[2021-10-05 12:34:38.013434 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:34:38.013576 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:34:38.013948 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:44:39.011771 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:44:39.011825 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:44:39.012306 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:54:40.017676 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:54:40.018240 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
[2021-10-05 12:54:40.021305 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:211:client4_0_mkdir_cbk] 0-gds-admin-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br>
<br>
It seems to gradualy become less and less entries, over night it has been reduced from 995 to 972.<br>
<br>
If I do an ls from the client side in some direcotories some names of the files shows up in info summary<br>
and then dissapears after a while.<br>
<br>
I would really appreciate some help on how to resolve this issue!<br>
<br>
Many thanks!<br>
<br>
Best regards<br>
Marcus<br>
<br>
---<br>
N?r du skickar e-post till SLU s? inneb?r detta att SLU behandlar dina personuppgifter. F?r att l?sa mer om hur detta g?r till, klicka h?r <<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" rel="noreferrer" target="_blank">https://www.slu.se/om-slu/kontakta-slu/personuppgifter/</a>><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click here <<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" rel="noreferrer" target="_blank">https://www.slu.se/en/about-slu/contact-slu/personal-data/</a>><br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
------------------------------<br>
<br>
End of Gluster-users Digest, Vol 162, Issue 1<br>
*********************************************<br>
<br>
</blockquote></div></div></div>