<html><head></head><body><div>That took a while!</div><div><br></div><div>I have the following stats:</div><div><br></div><div>4085169 files in both bricks</div><div>3162940 files only have a single hard link.</div><div><br></div><div>All of the files exist on both servers. bmidata2 (below) WAS running when bmidata1 died.</div><div><br></div><div>gluster volume heal clifford statistics heal-count</div><div>Gathering count of entries to be healed on volume clifford has been successful </div><div><br></div><div>Brick bmidata1:/data/glusterfs/clifford/brick/brick</div><div>Number of entries: 0</div><div><br></div><div>Brick bmidata2:/data/glusterfs/clifford/brick/brick</div><div>Number of entries: 296252</div><div><br></div><div>Brick bmidata1:/data/glusterfs/clifford3/brick/brick</div><div>Number of entries: 1</div><div><br></div><div>Brick bmidata2:/data/glusterfs/clifford3/brick/brick</div><div>Number of entries: 182407</div><div></div><div><br></div><div>Why those numbers are so much smaller than the data from a stat run through the entire brick, I have no idea.</div><div><br></div><div>The 20TB of space for this mount point is composed of 4 10TB brick in a 2x2.</div><div><br></div><div>As this is all from a large copy in from a backup source, I'm thinking of rerunning rsync to overwrite files with same create/modify times on the mount to realign things (maybe?)</div><div><br></div><div>I ran a giant ls/stat on the mount but nothing changed. Ran it again with no changes.</div><div><br></div><div>gluster-health-report </div><div><br></div><div>Loaded reports: glusterd-op-version, georep, gfid-mismatch-dht-report, glusterd-peer-disconnect, disk_usage, errors_in_logs, coredump, glusterd, glusterd_volume_version_cksum_errors, kernel_issues, errors_in_logs, ifconfig, nic-health, process_status</div><div><br></div><div>[ OK] Disk used percentage path=/ percentage=7</div><div>[ OK] Disk used percentage path=/var percentage=7</div><div>[ OK] Disk used percentage path=/tmp percentage=7</div><div>[ ERROR] Report failure report=report_check_errors_in_glusterd_log</div><div>[ OK] All peers are in connected state connected_count=1 total_peer_count=1</div><div>[ OK] no gfid mismatch</div><div>[ NOT OK] Failed to check op-version</div><div>[ NOT OK] The maximum size of core files created is NOT set to unlimited.</div><div>[ ERROR] Report failure report=report_check_worker_restarts</div><div>[ ERROR] Report failure report=report_non_participating_bricks</div><div>[ OK] Glusterd is running uptime_sec=5177509</div><div>[ ERROR] Report failure report=report_check_version_or_cksum_errors_in_glusterd_log</div><div>[ ERROR] Report failure report=report_check_errors_in_glusterd_log</div><div>[ NOT OK] Recieve errors in "ifconfig enp131s0" output</div><div>[ NOT OK] Recieve errors in "ifconfig eth0" output</div><div>[ NOT OK] Recieve errors in "ifconfig eth3" output</div><div>[ NOT OK] Recieve errors in "ifconfig mlx_ib0" output</div><div>[ NOT OK] Transmission errors in "ifconfig mlx_ib0" output</div><div>[ NOT OK] Errors seen in "cat /proc/net/dev -- eth0" output</div><div>[ NOT OK] Errors seen in "cat /proc/net/dev -- eth3" output</div><div>[ NOT OK] Errors seen in "cat /proc/net/dev -- mlx_ib0" output</div><div>[ NOT OK] Errors seen in "cat /proc/net/dev -- enp131s0" output</div><div>High CPU usage by Self-heal</div><div><br></div><div></div><div>NOTE: Bmidata2 up for over 300 days. due for reboot. </div><div><br></div><div>On Tue, 2017-10-24 at 12:35 +0530, Karthik Subrahmanya wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Hi Jim,<br><br>Can you check whether the same hardlinks are present on both the bricks & both of them have the link count 2?<br></div>If the link count is 2 then "find <brickpath> -samefile <brickpath/.glusterfs/<first two
bits of gfid>/<next 2 bits of gfid>/<full gfid>"<br></div></div></div>should give you the file path.<br><br></div>Regards,<br></div>Karthik<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 24, 2017 at 3:28 AM, Jim Kinney <span dir="ltr"><<a href="mailto:jim.kinney@gmail.com" target="_blank">jim.kinney@gmail.com</a>></span> wrote:<br><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div lang="EN-US"><div>I'm not so lucky. ALL of mine show 2 links and none have the attr data that supplies the path to the original.</div><div><br></div><div>I have the inode from stat. Looking now to dig out the path/filename from xfs_db on the specific inodes individually.</div><div><br></div><div>Is the hash of the filename or <path>/filename and if so relative to where? /, <path from top of brick>, ?</div><div><div class="h5"><div><br></div><div>On Mon, 2017-10-23 at 18:54 +0000, Matt Waymack wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div class="m_-8984230724855213427WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">In my case I was able to delete the hard links in the .glusterfs folders of the bricks and it seems to have done the trick, thanks!<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_-8984230724855213427__MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></a></p>
<span></span>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Karthik Subrahmanya [mailto:<a href="mailto:ksubrahm@redhat.com" target="_blank">ksubrahm@redhat.com</a>]
<br>
<b>Sent:</b> Monday, October 23, 2017 1:52 AM<br>
<b>To:</b> Jim Kinney <<a href="mailto:jim.kinney@gmail.com" target="_blank">jim.kinney@gmail.com</a>>; Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>><br>
<b>Cc:</b> gluster-users <<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><br>
<b>Subject:</b> Re: [Gluster-users] gfid entries in volume heal info that do not heal<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Jim & Matt,<u></u><u></u></p>
</div>
<p class="MsoNormal">Can you also check for the link count in the stat output of those hardlink entries in the .glusterfs folder on the bricks.<br>
If the link count is 1 on all the bricks for those entries, then they are orphaned entries and you can delete those hardlinks.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">To be on the safer side have a backup before deleting any of the entries.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Regards,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Karthik<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Oct 20, 2017 at 3:18 AM, Jim Kinney <<a href="mailto:jim.kinney@gmail.com" target="_blank">jim.kinney@gmail.com</a>> wrote:<u></u><u></u></p>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div>
<div>
<p class="MsoNormal">I've been following this particular thread as I have a similar issue (RAID6 array failed out with 3 dead drives at once while a 12 TB load was being copied into one mounted space - what a mess)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I have >700K GFID entries that have no path data:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Example:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">getfattr -d -e hex -m . .glusterfs/00/00/0000a5ef-<wbr>5af7-401b-84b5-ff2a51c10421<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"># file: .glusterfs/00/00/0000a5ef-<wbr>5af7-401b-84b5-ff2a51c10421<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">trusted.bit-rot.version=<wbr>0x020000000000000059b1b3160002<wbr>70e7<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">trusted.gfid=<wbr>0x0000a5ef5af7401b84b5ff2a51c1<wbr>0421<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">[<a href="mailto:root@bmidata1" target="_blank">root@bmidata1</a> brick]# getfattr -d -n trusted.glusterfs.pathinfo -e hex -m . .glusterfs/00/00/0000a5ef-<wbr>5af7-401b-84b5-ff2a51c10421<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">.glusterfs/00/00/0000a5ef-<wbr>5af7-401b-84b5-ff2a51c10421: trusted.glusterfs.pathinfo: No such attribute<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I had to totally rebuild the dead RAID array and did a copy from the live one before activating gluster on the rebuilt system. I accidentally copied over the .glusterfs folder from the working side<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">(replica 2 only for now - adding arbiter node as soon as I can get this one cleaned up).
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've run the methods from "<a href="http://docs.gluster.org/en/latest/Troubleshooting/gfid-to-path/" target="_blank">http://docs.gluster.org/en/<wbr>latest/Troubleshooting/gfid-<wbr>to-path/</a>" with no results using random GFIDs. A full systemic
run using the script from method 3 crashes with "too many nested links" error (or something similar).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">When I run gluster volume heal volname info, I get 700K+ GFIDs. Oh. gluster 3.8.4 on Centos 7.3<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Should I just remove the contents of the .glusterfs folder on both and restart gluster and run a ls/stat on every file?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">When I run a heal, it no longer has a decreasing number of files to heal so that's an improvement over the last 2-3 weeks :-)<u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">On Tue, 2017-10-17 at 14:34 +0000, Matt Waymack wrote:<u></u><u></u></p>
</div>
</div>
</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div>
<div>
<pre>Attached is the heal log for the volume as well as the shd log. <u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<pre><u></u> <u></u></pre>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<pre><u></u> <u></u></pre>
<pre>Run these commands on all the bricks of the replica pair to get the attrs set on the backend.<u></u><u></u></pre>
</blockquote>
<pre><u></u> <u></u></pre>
</blockquote>
<pre><u></u> <u></u></pre>
<pre><u></u> <u></u></pre>
<pre>[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . /exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<u></u><u></u></pre>
<pre>getfattr: Removing leading '/' from absolute path names<u></u><u></u></pre>
<pre># file: exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<u></u><u></u></pre>
<pre>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000<u></u><u></u></pre>
<pre>trusted.afr.dirty=<wbr>0x000000000000000000000000<u></u><u></u></pre>
<pre>trusted.afr.gv0-client-2=<wbr>0x000000000000000100000000<u></u><u></u></pre>
<pre>trusted.gfid=<wbr>0x108694dbc0394b7cbd3dad6a15d8<wbr>11a2<u></u><u></u></pre>
<pre>trusted.gfid2path.<wbr>9a2f5ada22eb9c45=<wbr>0x38633262623330322d323466332d<wbr>346463622d393630322d3839356136<wbr>396461363131662f435f564f4c2d62<wbr>3030312d693637342d63642d63772e<wbr>6d6435<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>[root@tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m . /exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<u></u><u></u></pre>
<pre>getfattr: Removing leading '/' from absolute path names<u></u><u></u></pre>
<pre># file: exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<u></u><u></u></pre>
<pre>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000<u></u><u></u></pre>
<pre>trusted.afr.dirty=<wbr>0x000000000000000000000000<u></u><u></u></pre>
<pre>trusted.afr.gv0-client-2=<wbr>0x000000000000000100000000<u></u><u></u></pre>
<pre>trusted.gfid=<wbr>0x108694dbc0394b7cbd3dad6a15d8<wbr>11a2<u></u><u></u></pre>
<pre>trusted.gfid2path.<wbr>9a2f5ada22eb9c45=<wbr>0x38633262623330322d323466332d<wbr>346463622d393630322d3839356136<wbr>396461363131662f435f564f4c2d62<wbr>3030312d693637342d63642d63772e<wbr>6d6435<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>[root@tpc-arbiter1-100617 ~]# getfattr -d -e hex -m . /exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<u></u><u></u></pre>
<pre>getfattr: /exp/b1/gv0/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2: No such file or directory<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre><u></u> <u></u></pre>
<pre>[root@tpc-cent-glus1-081017 ~]# getfattr -d -e hex -m . /exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<u></u><u></u></pre>
<pre>getfattr: Removing leading '/' from absolute path names<u></u><u></u></pre>
<pre># file: exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<u></u><u></u></pre>
<pre>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000<u></u><u></u></pre>
<pre>trusted.afr.dirty=<wbr>0x000000000000000000000000<u></u><u></u></pre>
<pre>trusted.afr.gv0-client-11=<wbr>0x000000000000000100000000<u></u><u></u></pre>
<pre>trusted.gfid=<wbr>0xe0c56bf78bfe46cabde1e46b92d3<wbr>3df3<u></u><u></u></pre>
<pre>trusted.gfid2path.<wbr>be3ba24c3ef95ff2=<wbr>0x63323366353834652d353566652d<wbr>343033382d393131622d3866373063<wbr>656334616136662f435f564f4c2d62<wbr>3030332d69313331342d63642d636d<wbr>2d63722e6d6435<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>[root@tpc-cent-glus2-081017 ~]# getfattr -d -e hex -m . /exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<u></u><u></u></pre>
<pre>getfattr: Removing leading '/' from absolute path names<u></u><u></u></pre>
<pre># file: exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<u></u><u></u></pre>
<pre>security.selinux=<wbr>0x73797374656d5f753a6f626a6563<wbr>745f723a756e6c6162656c65645f74<wbr>3a733000<u></u><u></u></pre>
<pre>trusted.afr.dirty=<wbr>0x000000000000000000000000<u></u><u></u></pre>
<pre>trusted.afr.gv0-client-11=<wbr>0x000000000000000100000000<u></u><u></u></pre>
<pre>trusted.gfid=<wbr>0xe0c56bf78bfe46cabde1e46b92d3<wbr>3df3<u></u><u></u></pre>
<pre>trusted.gfid2path.<wbr>be3ba24c3ef95ff2=<wbr>0x63323366353834652d353566652d<wbr>343033382d393131622d3866373063<wbr>656334616136662f435f564f4c2d62<wbr>3030332d69313331342d63642d636d<wbr>2d63722e6d6435<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>[root@tpc-arbiter1-100617 ~]# getfattr -d -e hex -m . /exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<u></u><u></u></pre>
<pre>getfattr: /exp/b4/gv0/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3: No such file or directory<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<pre><u></u> <u></u></pre>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<pre><u></u> <u></u></pre>
<pre>And the output of "gluster volume heal <volname> info split-brain"<u></u><u></u></pre>
</blockquote>
<pre><u></u> <u></u></pre>
</blockquote>
<pre><u></u> <u></u></pre>
<pre><u></u> <u></u></pre>
<pre>[root@tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info split-brain<u></u><u></u></pre>
<pre>Brick tpc-cent-glus1-081017:/exp/b1/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-cent-glus2-081017:/exp/b1/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-arbiter1-100617:/exp/b1/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-cent-glus1-081017:/exp/b2/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-cent-glus2-081017:/exp/b2/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-arbiter1-100617:/exp/b2/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-cent-glus1-081017:/exp/b3/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-cent-glus2-081017:/exp/b3/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-arbiter1-100617:/exp/b3/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-cent-glus1-081017:/exp/b4/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-cent-glus2-081017:/exp/b4/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Brick tpc-arbiter1-100617:/exp/b4/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries in split-brain: 0<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>-Matt<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>From: Karthik Subrahmanya [<a href="mailto:ksubrahm@redhat.com" target="_blank">mailto:ksubrahm@redhat.com</a>] <u></u><u></u></pre>
<pre>Sent: Tuesday, October 17, 2017 1:26 AM<u></u><u></u></pre>
<pre>To: Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>><u></u><u></u></pre>
<pre>Cc: gluster-users <<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>><u></u><u></u></pre>
<pre>Subject: Re: [Gluster-users] gfid entries in volume heal info that do not heal<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Hi Matt,<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>Run these commands on all the bricks of the replica pair to get the attrs set on the backend.<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>On the bricks of first replica set:<u></u><u></u></pre>
<pre>getfattr -d -e hex -m . <brick path>/.glusterfs/10/86/<wbr>108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2<u></u><u></u></pre>
<pre>On the fourth replica set:<u></u><u></u></pre>
<pre>getfattr -d -e hex -m . <brick path>/.glusterfs/e0/c5/<wbr>e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3<u></u><u></u></pre>
<pre>Also run the "gluster volume heal <volname>" once and send the shd log.<u></u><u></u></pre>
<pre>And the output of "gluster volume heal <volname> info split-brain"<u></u><u></u></pre>
<pre>Regards,<u></u><u></u></pre>
<pre>Karthik<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>On Mon, Oct 16, 2017 at 9:51 PM, Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mailto:mwaymack@nsgdv.com</a>> wrote:<u></u><u></u></pre>
<pre>OK, so here’s my output of the volume info and the heal info. I have not yet tracked down physical location of these files, any tips to finding them would be appreciated, but I’m definitely just wanting them gone. I forgot to mention earlier that the cluster is running 3.12 and was upgraded from 3.10; these files were likely stuck like this when it was on 3.10.<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>[root@tpc-cent-glus1-081017 ~]# gluster volume info gv0<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Volume Name: gv0<u></u><u></u></pre>
<pre>Type: Distributed-Replicate<u></u><u></u></pre>
<pre>Volume ID: 8f07894d-e3ab-4a65-bda1-<wbr>9d9dd46db007<u></u><u></u></pre>
<pre>Status: Started<u></u><u></u></pre>
<pre>Snapshot Count: 0<u></u><u></u></pre>
<pre>Number of Bricks: 4 x (2 + 1) = 12<u></u><u></u></pre>
<pre>Transport-type: tcp<u></u><u></u></pre>
<pre>Bricks:<u></u><u></u></pre>
<pre>Brick1: tpc-cent-glus1-081017:/exp/b1/<wbr>gv0<u></u><u></u></pre>
<pre>Brick2: tpc-cent-glus2-081017:/exp/b1/<wbr>gv0<u></u><u></u></pre>
<pre>Brick3: tpc-arbiter1-100617:/exp/b1/<wbr>gv0 (arbiter)<u></u><u></u></pre>
<pre>Brick4: tpc-cent-glus1-081017:/exp/b2/<wbr>gv0<u></u><u></u></pre>
<pre>Brick5: tpc-cent-glus2-081017:/exp/b2/<wbr>gv0<u></u><u></u></pre>
<pre>Brick6: tpc-arbiter1-100617:/exp/b2/<wbr>gv0 (arbiter)<u></u><u></u></pre>
<pre>Brick7: tpc-cent-glus1-081017:/exp/b3/<wbr>gv0<u></u><u></u></pre>
<pre>Brick8: tpc-cent-glus2-081017:/exp/b3/<wbr>gv0<u></u><u></u></pre>
<pre>Brick9: tpc-arbiter1-100617:/exp/b3/<wbr>gv0 (arbiter)<u></u><u></u></pre>
<pre>Brick10: tpc-cent-glus1-081017:/exp/b4/<wbr>gv0<u></u><u></u></pre>
<pre>Brick11: tpc-cent-glus2-081017:/exp/b4/<wbr>gv0<u></u><u></u></pre>
<pre>Brick12: tpc-arbiter1-100617:/exp/b4/<wbr>gv0 (arbiter)<u></u><u></u></pre>
<pre>Options Reconfigured:<u></u><u></u></pre>
<pre>nfs.disable: on<u></u><u></u></pre>
<pre>transport.address-family: inet<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>[root@tpc-cent-glus1-081017 ~]# gluster volume heal gv0 info<u></u><u></u></pre>
<pre>Brick tpc-cent-glus1-081017:/exp/b1/<wbr>gv0<u></u><u></u></pre>
<pre><gfid:108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2><u></u><u></u></pre>
<pre><gfid:6d5ade20-8996-4de2-95d5-<wbr>20ef98004742><u></u><u></u></pre>
<pre><gfid:bc6cdc3d-5c46-4597-a7eb-<wbr>282b21e9bdd5><u></u><u></u></pre>
<pre><gfid:3c2ff4d1-3662-4214-8f21-<wbr>f8f47dbdbf06><u></u><u></u></pre>
<pre><gfid:053e2fb1-bc89-476e-a529-<wbr>90dffa39963c><u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre><removed to save scrolling><u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 118<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-cent-glus2-081017:/exp/b1/<wbr>gv0<u></u><u></u></pre>
<pre><gfid:108694db-c039-4b7c-bd3d-<wbr>ad6a15d811a2><u></u><u></u></pre>
<pre><gfid:6d5ade20-8996-4de2-95d5-<wbr>20ef98004742><u></u><u></u></pre>
<pre><gfid:bc6cdc3d-5c46-4597-a7eb-<wbr>282b21e9bdd5><u></u><u></u></pre>
<pre><gfid:3c2ff4d1-3662-4214-8f21-<wbr>f8f47dbdbf06><u></u><u></u></pre>
<pre><gfid:053e2fb1-bc89-476e-a529-<wbr>90dffa39963c><u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre><removed to save scrolling><u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 118<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-arbiter1-100617:/exp/b1/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 0<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-cent-glus1-081017:/exp/b2/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 0<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-cent-glus2-081017:/exp/b2/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 0<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-arbiter1-100617:/exp/b2/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 0<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-cent-glus1-081017:/exp/b3/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 0<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-cent-glus2-081017:/exp/b3/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 0<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-arbiter1-100617:/exp/b3/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 0<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-cent-glus1-081017:/exp/b4/<wbr>gv0<u></u><u></u></pre>
<pre><gfid:e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3><u></u><u></u></pre>
<pre><gfid:6f0a0549-8669-46de-8823-<wbr>d6677fdca8e3><u></u><u></u></pre>
<pre><gfid:d0e2fb2a-21b5-4ea8-a578-<wbr>0801280b2530><u></u><u></u></pre>
<pre><gfid:48bff79c-7bc2-4dc5-8b7f-<wbr>4401b27fdf5a><u></u><u></u></pre>
<pre><gfid:5902593d-a059-4ec7-b18b-<wbr>7a2ab5c49a50><u></u><u></u></pre>
<pre><gfid:cb821178-4621-4fcf-90f3-<wbr>5b5c2ad7f756><u></u><u></u></pre>
<pre><gfid:6aea0805-8dd1-437c-b922-<wbr>52c9d11e488a><u></u><u></u></pre>
<pre><gfid:f4076a37-2e2f-4d7a-90dd-<wbr>0a3560a4bdff><u></u><u></u></pre>
<pre><gfid:51ff7386-a550-4971-957c-<wbr>b42c4d915e9f><u></u><u></u></pre>
<pre><gfid:4309f7b8-3a9d-4bc8-ba2b-<wbr>799f8a02611b><u></u><u></u></pre>
<pre><gfid:b76746ec-6d7d-4ea3-a001-<wbr>c96672a4d47e><u></u><u></u></pre>
<pre><gfid:f8de26e7-d17d-41e0-adcd-<wbr>e7d24ed74ac8><u></u><u></u></pre>
<pre><gfid:8e2c4540-e0b4-4006-bb5d-<wbr>aacd57f8f21b><u></u><u></u></pre>
<pre><gfid:183ebefb-b827-4cbc-b42b-<wbr>bfd136d5cabb><u></u><u></u></pre>
<pre><gfid:88d492fe-bfbd-4463-ba55-<wbr>0582d0ad671b><u></u><u></u></pre>
<pre><gfid:e3a6c068-d48b-44b5-9480-<wbr>245a69648a9b><u></u><u></u></pre>
<pre><gfid:4aab9c6a-22d2-469a-a688-<wbr>7b0a8784f4b1><u></u><u></u></pre>
<pre><gfid:c6d182f2-7e46-4502-a0d2-<wbr>b92824caa4de><u></u><u></u></pre>
<pre><gfid:eb546f93-e9d6-4a59-ac35-<wbr>6139b5c40919><u></u><u></u></pre>
<pre><gfid:6043e381-7edf-4569-bc37-<wbr>e27dd13549d2><u></u><u></u></pre>
<pre><gfid:52090dc7-7a3c-40f9-9c54-<wbr>3395f5158eab><u></u><u></u></pre>
<pre><gfid:ecceee46-4310-421e-b56e-<wbr>5fe46bd5263c><u></u><u></u></pre>
<pre><gfid:354aea57-4b40-47fc-8ede-<wbr>1d7e3b7501b4><u></u><u></u></pre>
<pre><gfid:d43284d4-86aa-42ff-98b8-<wbr>f6340b407d9d><u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 24<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-cent-glus2-081017:/exp/b4/<wbr>gv0<u></u><u></u></pre>
<pre><gfid:e0c56bf7-8bfe-46ca-bde1-<wbr>e46b92d33df3><u></u><u></u></pre>
<pre><gfid:6f0a0549-8669-46de-8823-<wbr>d6677fdca8e3><u></u><u></u></pre>
<pre><gfid:d0e2fb2a-21b5-4ea8-a578-<wbr>0801280b2530><u></u><u></u></pre>
<pre><gfid:48bff79c-7bc2-4dc5-8b7f-<wbr>4401b27fdf5a><u></u><u></u></pre>
<pre><gfid:5902593d-a059-4ec7-b18b-<wbr>7a2ab5c49a50><u></u><u></u></pre>
<pre><gfid:cb821178-4621-4fcf-90f3-<wbr>5b5c2ad7f756><u></u><u></u></pre>
<pre><gfid:6aea0805-8dd1-437c-b922-<wbr>52c9d11e488a><u></u><u></u></pre>
<pre><gfid:f4076a37-2e2f-4d7a-90dd-<wbr>0a3560a4bdff><u></u><u></u></pre>
<pre><gfid:51ff7386-a550-4971-957c-<wbr>b42c4d915e9f><u></u><u></u></pre>
<pre><gfid:4309f7b8-3a9d-4bc8-ba2b-<wbr>799f8a02611b><u></u><u></u></pre>
<pre><gfid:b76746ec-6d7d-4ea3-a001-<wbr>c96672a4d47e><u></u><u></u></pre>
<pre><gfid:f8de26e7-d17d-41e0-adcd-<wbr>e7d24ed74ac8><u></u><u></u></pre>
<pre><gfid:8e2c4540-e0b4-4006-bb5d-<wbr>aacd57f8f21b><u></u><u></u></pre>
<pre><gfid:183ebefb-b827-4cbc-b42b-<wbr>bfd136d5cabb><u></u><u></u></pre>
<pre><gfid:88d492fe-bfbd-4463-ba55-<wbr>0582d0ad671b><u></u><u></u></pre>
<pre><gfid:e3a6c068-d48b-44b5-9480-<wbr>245a69648a9b><u></u><u></u></pre>
<pre><gfid:4aab9c6a-22d2-469a-a688-<wbr>7b0a8784f4b1><u></u><u></u></pre>
<pre><gfid:c6d182f2-7e46-4502-a0d2-<wbr>b92824caa4de><u></u><u></u></pre>
<pre><gfid:eb546f93-e9d6-4a59-ac35-<wbr>6139b5c40919><u></u><u></u></pre>
<pre><gfid:6043e381-7edf-4569-bc37-<wbr>e27dd13549d2><u></u><u></u></pre>
<pre><gfid:52090dc7-7a3c-40f9-9c54-<wbr>3395f5158eab><u></u><u></u></pre>
<pre><gfid:ecceee46-4310-421e-b56e-<wbr>5fe46bd5263c><u></u><u></u></pre>
<pre><gfid:354aea57-4b40-47fc-8ede-<wbr>1d7e3b7501b4><u></u><u></u></pre>
<pre><gfid:d43284d4-86aa-42ff-98b8-<wbr>f6340b407d9d><u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 24<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Brick tpc-arbiter1-100617:/exp/b4/<wbr>gv0<u></u><u></u></pre>
<pre>Status: Connected<u></u><u></u></pre>
<pre>Number of entries: 0<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Thank you for your help!<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>From: Karthik Subrahmanya [<a href="mailto:mailto:ksubrahm@redhat.com" target="_blank">mailto:mailto:ksubrahm@<wbr>redhat.com</a>] <u></u><u></u></pre>
<pre>Sent: Monday, October 16, 2017 10:27 AM<u></u><u></u></pre>
<pre>To: Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mailto:mwaymack@nsgdv.com</a>><u></u><u></u></pre>
<pre>Cc: gluster-users <<a href="mailto:Gluster-users@gluster.org" target="_blank">mailto:Gluster-users@gluster.<wbr>org</a>><u></u><u></u></pre>
<pre>Subject: Re: [Gluster-users] gfid entries in volume heal info that do not heal<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Hi Matt, <u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>The files might be in split brain. Could you please send the outputs of these? <u></u><u></u></pre>
<pre>gluster volume info <volname><u></u><u></u></pre>
<pre>gluster volume heal <volname> info<u></u><u></u></pre>
<pre>And also the getfattr output of the files which are in the heal info output from all the bricks of that replica pair.<u></u><u></u></pre>
<pre>getfattr -d -e hex -m . <file path on brick><u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Thanks & Regards<u></u><u></u></pre>
<pre>Karthik<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>On 16-Oct-2017 8:16 PM, "Matt Waymack" <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mailto:mwaymack@nsgdv.com</a>> wrote:<u></u><u></u></pre>
<pre>Hi all,<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>I have a volume where the output of volume heal info shows several gfid entries to be healed, but they’ve been there for weeks and have not healed. Any normal file that shows up on the heal info does get healed as expected, but these gfid entries do not. Is there any way to remove these orphaned entries from the volume so they are no longer stuck in the heal process?<u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre>Thank you!<u></u><u></u></pre>
<pre><u></u> <u></u></pre>
<pre>______________________________<wbr>_________________<u></u><u></u></pre>
<pre>Gluster-users mailing list<u></u><u></u></pre>
<pre><a href="mailto:Gluster-users@gluster.org" target="_blank">mailto:Gluster-users@gluster.<wbr>org</a><u></u><u></u></pre>
<pre><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><u></u><u></u></pre>
<pre> <u></u><u></u></pre>
<pre><u></u> <u></u></pre>
</div>
</div>
<pre>______________________________<wbr>_________________<u></u><u></u></pre>
<pre>Gluster-users mailing list<u></u><u></u></pre>
<pre><a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><u></u><u></u></pre>
<pre><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><u></u><u></u></pre>
</blockquote>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</blockquote></div></div></div></blockquote></div><br></div>
</blockquote></body></html>