Can you share the output of 'ls -l' and 'getfattr -d -e hex -m . <gfid/file> ?<div><br></div><div>Best Regards,</div><div>Strahil Nikolov<br> <br> <blockquote style="margin: 0 0 20px 0;"> <div style="font-family:Roboto, sans-serif; color:#6D00F6;"> <div>On Fri, Mar 25, 2022 at 20:17, Collin Strassburger</div><div><cstrassburger@bihrle.com> wrote:</div> </div> <div style="padding: 10px 0 0 20px; margin: 10px 0 0 0; border-left: 1px solid #6D00F6;"> <div id="yiv2527564451"><style><!--
#yiv2527564451
_filtered {}
_filtered {}
_filtered {}
#yiv2527564451
#yiv2527564451 p.yiv2527564451MsoNormal, #yiv2527564451 li.yiv2527564451MsoNormal, #yiv2527564451 div.yiv2527564451MsoNormal
{margin:0in;font-size:11.0pt;font-family:"Calibri", sans-serif;}
#yiv2527564451 a:link, #yiv2527564451 span.yiv2527564451MsoHyperlink
{color:blue;text-decoration:underline;}
#yiv2527564451 p.yiv2527564451msonormal, #yiv2527564451 li.yiv2527564451msonormal, #yiv2527564451 div.yiv2527564451msonormal
{margin-right:0in;margin-left:0in;font-size:11.0pt;font-family:"Calibri", sans-serif;}
#yiv2527564451 span.yiv2527564451EmailStyle27
{font-family:"Calibri", sans-serif;color:windowtext;}
#yiv2527564451 .yiv2527564451MsoChpDefault
{font-family:"Calibri", sans-serif;}
_filtered {}
#yiv2527564451 div.yiv2527564451WordSection1
{}
--></style><div>
<div class="yiv2527564451WordSection1">
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Thank you for the suggestion!</span></font></p>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">I mounted the volume on a new mount point, retrieved the gfid of another file, and was able to obtain the file information through Method2.</span></font></p>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">However, when I used the gfids listed in heal, I get a “transport endpoint is not connected” error, despite them showing as “connected” in heal info (and volume status).</span></font></p>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Thanks,</span></font></p>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Collin</span></font></p>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<div id="yiv2527564451yqt19081" class="yiv2527564451yqt8517964934"><div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in;">
<p class="yiv2527564451MsoNormal"><b><font size="2" face="Calibri"><span style="font-size:11.0pt;font-weight:bold;">From:</span></font></b> Strahil Nikolov <hunter86_bg@yahoo.com>
<br clear="none">
<b><span style="font-weight:bold;">Sent:</span></b> Friday, March 25, 2022 1:47 PM<br clear="none">
<b><span style="font-weight:bold;">To:</span></b> Collin Strassburger <cstrassburger@bihrle.com>; gluster-users@gluster.org<br clear="none">
<b><span style="font-weight:bold;">Subject:</span></b> Re: [Gluster-users] gfid entries failing to heal</p>
</div>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<div style="border:solid black 1.5pt;padding:0in 0in 0in 0in;">
<p style="margin:2.25pt;background:#FFFFCC;"><b><font size="1" color="red" face="Calibri"><span style="font-size:9.0pt;color:red;font-weight:bold;">CAUTION - EXTERNAL EMAIL:</span></font></b><font size="1" color="black"><span style="font-size:9.0pt;color:black;">
Do not click any links or open any attachments unless you trust the sender and know the content is safe.
</span></font><font size="1"><span style="font-size:9.0pt;"></span></font></p>
</div>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">To find gfid path, you can use <a rel="nofollow noopener noreferrer" shape="rect" id="yiv2527564451linkextractor__1648230314675" target="_blank" href="https://docs.gluster.org/en/main/Troubleshooting/gfid-to-path/">https://docs.gluster.org/en/main/Troubleshooting/gfid-to-path/</a></span></font></p>
<div>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
</div>
<div>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Usually, I prefer to mount and then use Method2 to retrieve the path.</span></font></p>
</div>
<div>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
</div>
<div>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Then, you can getfattr the file/dir to get a clue.</span></font></p>
</div>
<div>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
</div>
<div>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Best Regards,</span></font></p>
</div>
<div>
<p style="margin-bottom:12.0pt;" class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Strahil Nikolov</span></font></p>
<blockquote style="margin-left:0in;margin-right:0in;margin-bottom:15.0pt;">
<div>
<div>
<p class="yiv2527564451MsoNormal"><font size="2" color="#6d00f6" face="Roboto"><span style="font-size:11.0pt;font-family:Roboto;color:#6D00F6;">On Fri, Mar 25, 2022 at 18:51, Collin Strassburger</span></font></p>
</div>
<div>
<p class="yiv2527564451MsoNormal"><font size="2" color="#6d00f6" face="Roboto"><span style="font-size:11.0pt;font-family:Roboto;color:#6D00F6;"><<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:cstrassburger@bihrle.com" target="_blank" href="mailto:cstrassburger@bihrle.com">cstrassburger@bihrle.com</a>> wrote:</span></font></p>
</div>
</div>
<div style="border:none;border-left:solid #6D00F6 1.0pt;padding:0in 0in 0in 15.0pt;margin-top:7.5pt;">
<div id="yiv2527564451">
<div>
<div>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Hello,</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">I am having a problem with a replica 3 volume.</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">When I run: gluster volume heal hydra_pbs_vol info</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">It returns:</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Brick hydra1:/data/glusterfs/PBS/NonMountDir</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"><gfid:78ce1382-4ac8-4c2e-94af-45b5abd055c5></span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Status: Connected</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Number of entries: 1</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Brick hydra2:/data/glusterfs/PBS/NonMountDir</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"><gfid:c3e7dc8e-111c-4ab2-9c10-ea5c1fd11223></span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"><gfid:96a72cf8-025d-4127-a216-4429dd1f59c6></span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"><gfid:8abbef15-929b-450f-9528-cb37dfc40bde></span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"><gfid:825bffa1-c727-4c2e-8c21-4da563851d7b></span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Status: Connected</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Number of entries: 4</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Brick viz1:/data/glusterfs/PBS/NonMountDir</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Status: Connected</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Number of entries: 0</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">The items have been present for some time and do not appear to be healing.</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">As shown above, the items are not labeled as split-brain and they do not have path information to do a manual delete-and-heal.</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Content of /var/log/glusterfs/glfsheal-hydra_pbs_vol.log is attached</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Info:</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">gluster volume info hydra_pbs_vol</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Volume Name: hydra_pbs_vol</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Type: Replicate</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Volume ID: efb30804-1c08-4ef6-a579-a2f77d5049e0</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Status: Started</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Snapshot Count: 0</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Number of Bricks: 1 x 3 = 3</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Transport-type: tcp</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Bricks:</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Brick1: hydra1:/data/glusterfs/PBS/NonMountDir</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Brick2: hydra2:/data/glusterfs/PBS/NonMountDir</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Brick3: viz1:/data/glusterfs/PBS/NonMountDir</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Options Reconfigured:</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">performance.client-io-threads: off</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">nfs.disable: on</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">transport.address-family: inet</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">storage.fips-mode-rchecksum: on</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">cluster.granular-entry-heal: on</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">features.bitrot: on</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">features.scrub: Active</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Status:</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Status of volume: hydra_pbs_vol</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Gluster process TCP Port RDMA Port Online Pid</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">------------------------------------------------------------------------------</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Brick hydra1:/data/glusterfs/PBS/NonMountDi</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">r 49152 0 Y 1518</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Brick hydra2:/data/glusterfs/PBS/NonMountDi</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">r 49152 0 Y 1438</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Brick viz1:/data/glusterfs/PBS/NonMountDir 49152 0 Y 2991</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Self-heal Daemon on localhost N/A N/A Y 1942</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Bitrot Daemon on localhost N/A N/A Y 1563</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Scrubber Daemon on localhost N/A N/A Y 1738</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Self-heal Daemon on viz1 N/A N/A Y 3491</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Bitrot Daemon on viz1 N/A N/A Y 3203</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Scrubber Daemon on viz1 N/A N/A Y 3261</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Self-heal Daemon on hydra2 N/A N/A Y 1843</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Bitrot Daemon on hydra2 N/A N/A Y 1475</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Scrubber Daemon on hydra2 N/A N/A Y 1651</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Task Status of Volume hydra_pbs_vol</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">------------------------------------------------------------------------------</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">There are no active volume tasks</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">How can I resolve these entries/issues?</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;"> </span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Thanks,</span></font></p>
<p class="yiv2527564451msonormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">Collin Strassburger (he/him)</span></font></p>
</div>
</div>
</div>
<p class="yiv2527564451MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt;">________<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
Community Meeting Calendar:<br clear="none">
<br clear="none">
Schedule -<br clear="none">
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br clear="none">
Bridge: <a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a><br clear="none">
Gluster-users mailing list<br clear="none">
<a rel="nofollow noopener noreferrer" shape="rect" ymailto="mailto:Gluster-users@gluster.org" target="_blank" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br clear="none">
<a rel="nofollow noopener noreferrer" shape="rect" target="_blank" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></span></font></p>
</div>
</blockquote>
</div></div>
</div>
</div></div> </div> </blockquote></div>