<div dir="ltr"><div><div><div><div><div><div><div><div><div>Hey Matt,<br><br></div>From the xattr output, it looks like the files are not present on the arbiter brick & needs healing. But on the parent it does not have the pending markers set for those entries.<br></div>The workaround for this is you need to do a lookup on the file which needs heal from the mount, so it will create the entry on the arbiter brick and then run the volume heal to do the healing.<br></div>Follow these steps to resolve the issue: (first try this on one file and check whether it gets healed. If it gets healed then do this for all the remaining files)<br></div>1. Get the file path for the gfids you got from heal info output.<br> find <brickpath> -samefile <brickpath/.glusterfs/<first two bits of gfid>/<next 2 bits of gfid>/<full gfid><br></div>2. Do ls/stat on the file from mount.<br></div>3. Run volume heal.<br></div>4. Check the heal info output to see whether the file got healed.<br><br></div>If one file gets healed, then do step 1 & 2 for the rest of the files and do step 3 & 4 once at the end.<br></div><div>Let me know if that resolves the issue.<br></div><div><br></div><div>Thanks & Regards,<br></div><div>Karthik<br></div> </div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 17, 2017 at 8:04 PM, Matt Waymack <span dir="ltr"><<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Attached is the heal log for the volume as well as the shd log.<br>
<span class=""><br>
>> Run these commands on all the bricks of the replica pair to get the attrs set on the backend.<br>
<br>
</span>[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . /exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<br>
getfattr: Removing leading '/' from absolute path names<br>
# file: exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<br>
security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000<br>
trusted.afr.dirty=<wbr>0x000000000000000000000000<br>
trusted.afr.gv0-client-2=<wbr>0x000000000000000100000000<br>
trusted.gfid=<wbr>0x108694dbc0394b7cbd3dad6a15d8<wbr>11a2<br>
trusted.gfid2path.<wbr>9a2f5ada22eb9c45=<wbr>0x38633262623330322d323466332d<wbr>346463622d393630322d3839356136<wbr>396461363131662f435f564f4c2d62<wbr>3030312d693637342d63642d63772e<wbr>6d6435<br>
<br>
[root@tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m . /exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<br>
getfattr: Removing leading '/' from absolute path names<br>
# file: exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<br>
security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000<br>
trusted.afr.dirty=<wbr>0x000000000000000000000000<br>
trusted.afr.gv0-client-2=<wbr>0x000000000000000100000000<br>
trusted.gfid=<wbr>0x108694dbc0394b7cbd3dad6a15d8<wbr>11a2<br>
trusted.gfid2path.<wbr>9a2f5ada22eb9c45=<wbr>0x38633262623330322d323466332d<wbr>346463622d393630322d3839356136<wbr>396461363131662f435f564f4c2d62<wbr>3030312d693637342d63642d63772e<wbr>6d6435<br>
<br>
[root@tpc-arbiter1-100617 ~]# getfattr -d -e hex -m . /exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<br>
getfattr: /exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2: No such file or directory<br>
<br>
<br>
[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . /exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<br>
getfattr: Removing leading '/' from absolute path names<br>
# file: exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<br>
security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000<br>
trusted.afr.dirty=<wbr>0x000000000000000000000000<br>
trusted.afr.gv0-client-11=<wbr>0x000000000000000100000000<br>
trusted.gfid=<wbr>0xe0c56bf78bfe46cabde1e46b92d3<wbr>3df3<br>
trusted.gfid2path.<wbr>be3ba24c3ef95ff2=<wbr>0x63323366353834652d353566652d<wbr>343033382d393131622d3866373063<wbr>656334616136662f435f564f4c2d62<wbr>3030332d69313331342d63642d636d<wbr>2d63722e6d6435<br>
<br>
[root@tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m . /exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<br>
getfattr: Removing leading '/' from absolute path names<br>
# file: exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<br>
security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000<br>
trusted.afr.dirty=<wbr>0x000000000000000000000000<br>
trusted.afr.gv0-client-11=<wbr>0x000000000000000100000000<br>
trusted.gfid=<wbr>0xe0c56bf78bfe46cabde1e46b92d3<wbr>3df3<br>
trusted.gfid2path.<wbr>be3ba24c3ef95ff2=<wbr>0x63323366353834652d353566652d<wbr>343033382d393131622d3866373063<wbr>656334616136662f435f564f4c2d62<wbr>3030332d69313331342d63642d636d<wbr>2d63722e6d6435<br>
<br>
[root@tpc-arbiter1-100617 ~]# getfattr -d -e hex -m . /exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<br>
getfattr: /exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3: No such file or directory<br>
<span class=""><br>
>> And the output of "gluster volume heal <volname> info split-brain"<br>
<br>
</span>[root@tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info split-brain<br>
Brick tpc-cent-glus1-081017:/exp/b1/<wbr>gv0<br>
Status: Connected<br>
Number of entries in split-brain: 0<br>
<br>
Brick tpc-cent-glus2-081017:/exp/b1/<wbr>gv0<br>
Status: Connected<br>
Number of entries in split-brain: 0<br>
<span class=""><br>
Brick tpc-arbiter1-100617:/exp/b1/<wbr>gv0<br>
Status: Connected<br>
</span>Number of entries in split-brain: 0<br>
<span class=""><br>
Brick tpc-cent-glus1-081017:/exp/b2/<wbr>gv0<br>
Status: Connected<br>
</span>Number of entries in split-brain: 0<br>
<span class=""><br>
Brick tpc-cent-glus2-081017:/exp/b2/<wbr>gv0<br>
Status: Connected<br>
</span>Number of entries in split-brain: 0<br>
<span class=""><br>
Brick tpc-arbiter1-100617:/exp/b2/<wbr>gv0<br>
Status: Connected<br>
</span>Number of entries in split-brain: 0<br>
<span class=""><br>
Brick tpc-cent-glus1-081017:/exp/b3/<wbr>gv0<br>
Status: Connected<br>
</span>Number of entries in split-brain: 0<br>
<span class=""><br>
Brick tpc-cent-glus2-081017:/exp/b3/<wbr>gv0<br>
Status: Connected<br>
</span>Number of entries in split-brain: 0<br>
<span class=""><br>
Brick tpc-arbiter1-100617:/exp/b3/<wbr>gv0<br>
Status: Connected<br>
</span>Number of entries in split-brain: 0<br>
<br>
Brick tpc-cent-glus1-081017:/exp/b4/<wbr>gv0<br>
Status: Connected<br>
Number of entries in split-brain: 0<br>
<br>
Brick tpc-cent-glus2-081017:/exp/b4/<wbr>gv0<br>
Status: Connected<br>
Number of entries in split-brain: 0<br>
<span class=""><br>
Brick tpc-arbiter1-100617:/exp/b4/<wbr>gv0<br>
Status: Connected<br>
</span>Number of entries in split-brain: 0<br>
<br>
-Matt<br>
<span class=""><br>
From: Karthik Subrahmanya [mailto:<a href="mailto:ksubrahm@redhat.com">ksubrahm@redhat.com</a>]<br>
</span>Sent: Tuesday, October 17, 2017 1:26 AM<br>
<span class="">To: Matt Waymack <<a href="mailto:mwaymack@nsgdv.com">mwaymack@nsgdv.com</a>><br>
Cc: gluster-users <<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>><br>
Subject: Re: [Gluster-users] gfid entries in volume heal info that do not heal<br>
<br>
Hi Matt,<br>
<br>
</span><span class="">Run these commands on all the bricks of the replica pair to get the attrs set on the backend.<br>
<br>
On the bricks of first replica set:<br>
getfattr -d -e hex -m . <brick path>/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<br>
On the fourth replica set:<br>
getfattr -d -e hex -m . <brick path>/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<br>
Also run the "gluster volume heal <volname>" once and send the shd log.<br>
And the output of "gluster volume heal <volname> info split-brain"<br>
Regards,<br>
Karthik<br>
<br>
</span><div><div class="h5">On Mon, Oct 16, 2017 at 9:51 PM, Matt Waymack <mailto:<a href="mailto:mwaymack@nsgdv.com">mwaymack@nsgdv.com</a>> wrote:<br>
OK, so here’s my output of the volume info and the heal info. I have not yet tracked down physical location of these files, any tips to finding them would be appreciated, but I’m definitely just wanting them gone. I forgot to mention earlier that the cluster is running 3.12 and was upgraded from 3.10; these files were likely stuck like this when it was on 3.10.<br>
<br>
[root@tpc-cent-glus1-081017 ~]# gluster volume info gv0<br>
<br>
Volume Name: gv0<br>
Type: Distributed-Replicate<br>
Volume ID: 8f07894d-e3ab-4a65-bda1-<wbr>9d9dd46db007<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 4 x (2 + 1) = 12<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: tpc-cent-glus1-081017:/exp/b1/<wbr>gv0<br>
Brick2: tpc-cent-glus2-081017:/exp/b1/<wbr>gv0<br>
Brick3: tpc-arbiter1-100617:/exp/b1/<wbr>gv0 (arbiter)<br>
Brick4: tpc-cent-glus1-081017:/exp/b2/<wbr>gv0<br>
Brick5: tpc-cent-glus2-081017:/exp/b2/<wbr>gv0<br>
Brick6: tpc-arbiter1-100617:/exp/b2/<wbr>gv0 (arbiter)<br>
Brick7: tpc-cent-glus1-081017:/exp/b3/<wbr>gv0<br>
Brick8: tpc-cent-glus2-081017:/exp/b3/<wbr>gv0<br>
Brick9: tpc-arbiter1-100617:/exp/b3/<wbr>gv0 (arbiter)<br>
Brick10: tpc-cent-glus1-081017:/exp/b4/<wbr>gv0<br>
Brick11: tpc-cent-glus2-081017:/exp/b4/<wbr>gv0<br>
Brick12: tpc-arbiter1-100617:/exp/b4/<wbr>gv0 (arbiter)<br>
Options Reconfigured:<br>
nfs.disable: on<br>
transport.address-family: inet<br>
<br>
[root@tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info<br>
Brick tpc-cent-glus1-081017:/exp/b1/<wbr>gv0<br>
<gfid:108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2><br>
<gfid:6d5ade20-8996-4de2-95d5-<wbr>20ef98004742><br>
<gfid:bc6cdc3d-5c46-4597-a7eb-<wbr>282b21e9bdd5><br>
<gfid:3c2ff4d1-3662-4214-8f21-<wbr>f8f47dbdbf06><br>
<gfid:053e2fb1-bc89-476e-a529-<wbr>90dffa39963c><br>
<br>
<removed to save scrolling><br>
<br>
Status: Connected<br>
Number of entries: 118<br>
<br>
Brick tpc-cent-glus2-081017:/exp/b1/<wbr>gv0<br>
<gfid:108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2><br>
<gfid:6d5ade20-8996-4de2-95d5-<wbr>20ef98004742><br>
<gfid:bc6cdc3d-5c46-4597-a7eb-<wbr>282b21e9bdd5><br>
<gfid:3c2ff4d1-3662-4214-8f21-<wbr>f8f47dbdbf06><br>
<gfid:053e2fb1-bc89-476e-a529-<wbr>90dffa39963c><br>
<br>
<removed to save scrolling><br>
<br>
Status: Connected<br>
Number of entries: 118<br>
<br>
Brick tpc-arbiter1-100617:/exp/b1/<wbr>gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick tpc-cent-glus1-081017:/exp/b2/<wbr>gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick tpc-cent-glus2-081017:/exp/b2/<wbr>gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick tpc-arbiter1-100617:/exp/b2/<wbr>gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick tpc-cent-glus1-081017:/exp/b3/<wbr>gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick tpc-cent-glus2-081017:/exp/b3/<wbr>gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick tpc-arbiter1-100617:/exp/b3/<wbr>gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Brick tpc-cent-glus1-081017:/exp/b4/<wbr>gv0<br>
<gfid:e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3><br>
<gfid:6f0a0549-8669-46de-8823-<wbr>d6677fdca8e3><br>
<gfid:d0e2fb2a-21b5-4ea8-a578-<wbr>0801280b2530><br>
<gfid:48bff79c-7bc2-4dc5-8b7f-<wbr>4401b27fdf5a><br>
<gfid:5902593d-a059-4ec7-b18b-<wbr>7a2ab5c49a50><br>
<gfid:cb821178-4621-4fcf-90f3-<wbr>5b5c2ad7f756><br>
<gfid:6aea0805-8dd1-437c-b922-<wbr>52c9d11e488a><br>
<gfid:f4076a37-2e2f-4d7a-90dd-<wbr>0a3560a4bdff><br>
<gfid:51ff7386-a550-4971-957c-<wbr>b42c4d915e9f><br>
<gfid:4309f7b8-3a9d-4bc8-ba2b-<wbr>799f8a02611b><br>
<gfid:b76746ec-6d7d-4ea3-a001-<wbr>c96672a4d47e><br>
<gfid:f8de26e7-d17d-41e0-adcd-<wbr>e7d24ed74ac8><br>
<gfid:8e2c4540-e0b4-4006-bb5d-<wbr>aacd57f8f21b><br>
<gfid:183ebefb-b827-4cbc-b42b-<wbr>bfd136d5cabb><br>
<gfid:88d492fe-bfbd-4463-ba55-<wbr>0582d0ad671b><br>
<gfid:e3a6c068-d48b-44b5-9480-<wbr>245a69648a9b><br>
<gfid:4aab9c6a-22d2-469a-a688-<wbr>7b0a8784f4b1><br>
<gfid:c6d182f2-7e46-4502-a0d2-<wbr>b92824caa4de><br>
<gfid:eb546f93-e9d6-4a59-ac35-<wbr>6139b5c40919><br>
<gfid:6043e381-7edf-4569-bc37-<wbr>e27dd13549d2><br>
<gfid:52090dc7-7a3c-40f9-9c54-<wbr>3395f5158eab><br>
<gfid:ecceee46-4310-421e-b56e-<wbr>5fe46bd5263c><br>
<gfid:354aea57-4b40-47fc-8ede-<wbr>1d7e3b7501b4><br>
<gfid:d43284d4-86aa-42ff-98b8-<wbr>f6340b407d9d><br>
Status: Connected<br>
Number of entries: 24<br>
<br>
Brick tpc-cent-glus2-081017:/exp/b4/<wbr>gv0<br>
<gfid:e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3><br>
<gfid:6f0a0549-8669-46de-8823-<wbr>d6677fdca8e3><br>
<gfid:d0e2fb2a-21b5-4ea8-a578-<wbr>0801280b2530><br>
<gfid:48bff79c-7bc2-4dc5-8b7f-<wbr>4401b27fdf5a><br>
<gfid:5902593d-a059-4ec7-b18b-<wbr>7a2ab5c49a50><br>
<gfid:cb821178-4621-4fcf-90f3-<wbr>5b5c2ad7f756><br>
<gfid:6aea0805-8dd1-437c-b922-<wbr>52c9d11e488a><br>
<gfid:f4076a37-2e2f-4d7a-90dd-<wbr>0a3560a4bdff><br>
<gfid:51ff7386-a550-4971-957c-<wbr>b42c4d915e9f><br>
<gfid:4309f7b8-3a9d-4bc8-ba2b-<wbr>799f8a02611b><br>
<gfid:b76746ec-6d7d-4ea3-a001-<wbr>c96672a4d47e><br>
<gfid:f8de26e7-d17d-41e0-adcd-<wbr>e7d24ed74ac8><br>
<gfid:8e2c4540-e0b4-4006-bb5d-<wbr>aacd57f8f21b><br>
<gfid:183ebefb-b827-4cbc-b42b-<wbr>bfd136d5cabb><br>
<gfid:88d492fe-bfbd-4463-ba55-<wbr>0582d0ad671b><br>
<gfid:e3a6c068-d48b-44b5-9480-<wbr>245a69648a9b><br>
<gfid:4aab9c6a-22d2-469a-a688-<wbr>7b0a8784f4b1><br>
<gfid:c6d182f2-7e46-4502-a0d2-<wbr>b92824caa4de><br>
<gfid:eb546f93-e9d6-4a59-ac35-<wbr>6139b5c40919><br>
<gfid:6043e381-7edf-4569-bc37-<wbr>e27dd13549d2><br>
<gfid:52090dc7-7a3c-40f9-9c54-<wbr>3395f5158eab><br>
<gfid:ecceee46-4310-421e-b56e-<wbr>5fe46bd5263c><br>
<gfid:354aea57-4b40-47fc-8ede-<wbr>1d7e3b7501b4><br>
<gfid:d43284d4-86aa-42ff-98b8-<wbr>f6340b407d9d><br>
Status: Connected<br>
Number of entries: 24<br>
<br>
Brick tpc-arbiter1-100617:/exp/b4/<wbr>gv0<br>
Status: Connected<br>
Number of entries: 0<br>
<br>
Thank you for your help!<br>
<br>
</div></div>From: Karthik Subrahmanya [mailto:<a href="mailto:mailto">mailto</a>:<a href="mailto:ksubrahm@redhat.com">ksubrahm@<wbr>redhat.com</a>]<br>
<span class="">Sent: Monday, October 16, 2017 10:27 AM<br>
</span>To: Matt Waymack <mailto:<a href="mailto:mwaymack@nsgdv.com">mwaymack@nsgdv.com</a>><br>
Cc: gluster-users <mailto:<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.<wbr>org</a>><br>
<span class="">Subject: Re: [Gluster-users] gfid entries in volume heal info that do not heal<br>
<br>
Hi Matt, <br>
<br>
The files might be in split brain. Could you please send the outputs of these? <br>
gluster volume info <volname><br>
gluster volume heal <volname> info<br>
And also the getfattr output of the files which are in the heal info output from all the bricks of that replica pair.<br>
getfattr -d -e hex -m . <file path on brick><br>
<br>
Thanks & Regards<br>
Karthik<br>
<br>
</span><span class="">On 16-Oct-2017 8:16 PM, "Matt Waymack" <mailto:<a href="mailto:mwaymack@nsgdv.com">mwaymack@nsgdv.com</a>> wrote:<br>
Hi all,<br>
<br>
I have a volume where the output of volume heal info shows several gfid entries to be healed, but they’ve been there for weeks and have not healed. Any normal file that shows up on the heal info does get healed as expected, but these gfid entries do not. Is there any way to remove these orphaned entries from the volume so they are no longer stuck in the heal process?<br>
<br>
Thank you!<br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
</span>mailto:<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.<wbr>org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
<br>
<br>
</blockquote></div><br></div>