<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
Hi Strahil,<br>
<br>
Thanks for the input. It worked flawlesly!<br>
I copy/paste the process here (maybe useful for someone).<br>
<br>
### entering the "corrupt" gluster brick folder<tt><br>
</tt><tt>[root@node1 data]# cd
7ac28c32-947b-4ad5-8d69-213a205f06e8/images/d7c11f2e-58e4-4fe1-8236-6ded0f4dd757/</tt><tt><br>
</tt><tt>[root@node1 d7c11f2e-58e4-4fe1-8236-6ded0f4dd757]# ls -la</tt><tt><br>
</tt><tt>total 1080</tt><tt><br>
</tt><tt>drwxr-xr-x. 2 vdsm kvm 8192 Jan 1 1970 .</tt><tt><br>
</tt><tt>drwxr-xr-x. 4 vdsm kvm 94 Dec 18 11:11 ..</tt><tt><br>
</tt><tt>-rw-rw----. 2 vdsm kvm 30720 Dec 17 13:17
e566f230-df72-4073-aecf-7e5a8d6b569b</tt><tt><br>
</tt><tt>-rw-rw----. 2 vdsm kvm 1048576 Dec 2 14:55
e566f230-df72-4073-aecf-7e5a8d6b569b.lease</tt><tt><br>
</tt><tt>-rw-r--r--. 2 vdsm kvm 429 Dec 17 13:17
e566f230-df72-4073-aecf-7e5a8d6b569b.meta</tt><tt><br>
</tt>### making a backup of the files (did it on all the nodes)<tt><br>
</tt><tt>[root@node1 d7c11f2e-58e4-4fe1-8236-6ded0f4dd757]# mkdir -p
/root/save/gluster_bricks/data/data/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/d7c11f2e-58e4-4fe1-8236-6ded0f4dd757</tt><tt><br>
</tt><tt>[root@node1 d7c11f2e-58e4-4fe1-8236-6ded0f4dd757]# cp *
/root/save/gluster_bricks/data/data/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/d7c11f2e-58e4-4fe1-8236-6ded0f4dd757/</tt><tt><br>
</tt>### rsyncing the files from the selected source (i choosen
node2 as soruce, and did this on node3 also)<tt><br>
</tt><tt>[root@node1 d7c11f2e-58e4-4fe1-8236-6ded0f4dd757]# rsync
-avh
<a class="moz-txt-link-abbreviated" href="mailto:root@node2.ovirt.local:/gluster_bricks/data/data/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/d7c11f2e-58e4-4fe1-8236-6ded0f4dd757/e566f230-df72-4073-aecf-7e5a8d6b569b*">root@node2.ovirt.local:/gluster_bricks/data/data/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/d7c11f2e-58e4-4fe1-8236-6ded0f4dd757/e566f230-df72-4073-aecf-7e5a8d6b569b*</a>
.</tt><tt><br>
</tt><tt>receiving incremental file list</tt><tt><br>
</tt><tt><br>
</tt><tt>sent 20 bytes received 129 bytes 99.33 bytes/sec</tt><tt><br>
</tt><tt>total size is 1.08M speedup is 7,246.48</tt><tt><br>
</tt>### started the healing<tt><br>
</tt><tt>[root@node1 d7c11f2e-58e4-4fe1-8236-6ded0f4dd757]# gluster
volume heal data</tt><tt><br>
</tt><tt>Launching heal operation to perform index self heal on
volume data has been successful</tt><tt><br>
</tt><tt>Use heal info commands to check status.</tt><tt><br>
</tt><tt>### checking result<tt><br>
</tt></tt><tt>[root@node1 d7c11f2e-58e4-4fe1-8236-6ded0f4dd757]#
gluster volume heal data info</tt><tt><br>
</tt><tt>Brick node1storage.ovirt.local:/gluster_bricks/data/data</tt><tt><br>
</tt><tt>/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/079904a4-71af-492c-bb2f-b45a918e8a2e/fce4b64d-2444-4f11-b226-db75bb2960c2.meta</tt><tt><br>
</tt><tt>Status: Connected</tt><tt><br>
</tt><tt>Number of entries: 1</tt><tt><br>
</tt><tt><br>
</tt><tt>Brick node2storage.ovirt.local:/gluster_bricks/data/data</tt><tt><br>
</tt><tt>/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/079904a4-71af-492c-bb2f-b45a918e8a2e/fce4b64d-2444-4f11-b226-db75bb2960c2.meta</tt><tt><br>
</tt><tt>Status: Connected</tt><tt><br>
</tt><tt>Number of entries: 1</tt><tt><br>
</tt><tt><br>
</tt><tt>Brick node3storage.ovirt.local:/gluster_bricks/data/data</tt><tt><br>
</tt><tt>Status: Connected</tt><tt><br>
</tt><tt>Number of entries: 0</tt><tt><br>
</tt><tt><br>
</tt>### As you can see the only one pending entry left. I did the
above fix for the other files, and it also healed successfully.<tt><br>
</tt><br>
Best Regards,<br>
Balazs Szilagyi<br>
<br>
2019.12.18. 20:07 keltezéssel, Strahil Nikolov írta:<br>
<blockquote type="cite"
cite="mid:1090341855.606785.1576696028536@mail.yahoo.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div class="ydp338460b0yahoo-style-wrap"
style="font-family:courier new, courier, monaco, monospace,
sans-serif;font-size:16px;">
<div dir="ltr" data-setdir="false">You are the second person
(excluding me) who observes this behaviour.</div>
<div dir="ltr" data-setdir="false">The easiest way to resolve
this is to:</div>
<div dir="ltr" data-setdir="false">1. Check which file is newest
(there is a timestamp in the file) for :</div>
<div dir="ltr" data-setdir="false">
<div>
<div dir="ltr" style="color: rgb(38, 40, 42); font-family:
Helvetica Neue, Helvetica, Arial, sans-serif;"
data-setdir="false">/gluster_bricks/data/data<span><span
style="color: rgb(38, 40, 42); font-family: Helvetica
Neue, Helvetica, Arial, sans-serif;">/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/d7c11f2e-58e4-4fe1-8236-6ded0f4dd757/e566f230-df72-4073-aecf-7e5a8d6b569b.meta</span></span></div>
</div>
<div dir="ltr" style="color: rgb(38, 40, 42); font-family:
Helvetica Neue, Helvetica, Arial, sans-serif;"
data-setdir="false"><span><span style="color: rgb(38, 40,
42); font-family: Helvetica Neue, Helvetica, Arial,
sans-serif;"><span><span style="color: rgb(38, 40, 42);
font-family: Helvetica Neue, Helvetica, Arial,
sans-serif;"><span><span style="color: rgb(38, 40,
42); font-family: Helvetica Neue, Helvetica,
Arial, sans-serif; font-size: 16px;">And for :</span></span></span></span></span></span></div>
<div dir="ltr" style="color: rgb(38, 40, 42); font-family:
Helvetica Neue, Helvetica, Arial, sans-serif;"
data-setdir="false"><span><span style="color: rgb(38, 40,
42); font-family: Helvetica Neue, Helvetica, Arial,
sans-serif;"><span><span style="color: rgb(38, 40, 42);
font-family: Helvetica Neue, Helvetica, Arial,
sans-serif;"><span><span style="color: rgb(38, 40,
42); font-family: Helvetica Neue, Helvetica,
Arial, sans-serif; font-size: 16px;">/gluster_bricks/data/data</span><span
style="color: rgb(38, 40, 42); font-family:
Helvetica Neue, Helvetica, Arial, sans-serif;
font-size: 16px;">/</span></span>7ac28c32-947b-4ad5-8d69-213a205f06e8/images/079904a4-71af-492c-bb2f-b45a918e8a2e/fce4b64d-2444-4f11-b226-db75bb2960c2.meta</span></span><br>
</span></span></div>
<br>
</div>
<div dir="ltr" data-setdir="false">Let's assume
node3storage.ovirt.local has the newest data.</div>
<div dir="ltr" data-setdir="false"><br>
</div>
<div dir="ltr" data-setdir="false">2.Then you can just backup
(just in case you change your mind) the files locally and
rsync from node3storage.ovirt.local (actually replace with the
node with newest timestamp in the file) to the other bricks.</div>
<div dir="ltr" data-setdir="false"><br>
</div>
<div dir="ltr" data-setdir="false">3. Run a gluster heal just to
notify gluster that the issue is resolved.</div>
<div dir="ltr" data-setdir="false"><br>
</div>
<div dir="ltr" data-setdir="false">In my case one of the nodes
had a newer version of the file (I am using replica 2 arbiter
1 volume) and the gfid was different and this prevented
Gluster from healing that.</div>
<div dir="ltr" data-setdir="false"><br>
</div>
<div dir="ltr" data-setdir="false">Usually , oVirt just updates
the timestamp in the meta files - so even an older version is
not a problem.</div>
<div dir="ltr" data-setdir="false"><br>
</div>
<div dir="ltr" data-setdir="false">P.S.: What version of gluster
are you using. I suppose v6.5 or v6.6 , right ?</div>
<div dir="ltr" data-setdir="false"><br>
</div>
<div dir="ltr" data-setdir="false">Best Regards,</div>
<div dir="ltr" data-setdir="false">Strahil Nikolov</div>
<div dir="ltr" data-setdir="false"><br>
</div>
<div dir="ltr" data-setdir="false"><br>
</div>
<div dir="ltr" data-setdir="false"><br>
</div>
<div><br>
</div>
</div>
<div id="ydp22192ad3yahoo_quoted_7165734837"
class="ydp22192ad3yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial,
sans-serif;font-size:13px;color:#26282a;">
<div> В сряда, 18 декември 2019 г., 16:18:12 ч. Гринуич+2,
Szilágyi Balázs <a class="moz-txt-link-rfc2396E" href="mailto:szilagyi.balazs@pcvilag.hu"><szilagyi.balazs@pcvilag.hu></a> написа: </div>
<div><br>
</div>
<div><br>
</div>
<div>
<div dir="ltr">Dear Gluster Users,<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">I'm a newbie to gluster storage and during
the stability testing I made <br>
</div>
<div dir="ltr">a node reboot and I've got some heal issues
after that, that i'm unable <br>
</div>
<div dir="ltr">to fix.<br>
</div>
<div dir="ltr">Anyway the vm's are running fine from the
storage, and I did not <br>
</div>
<div dir="ltr">discovered any data corruption.<br>
</div>
<div dir="ltr">The system is oVirt version 4.3.7. I have 3
nodes with a replica 3. <br>
</div>
<div dir="ltr">Please let me know what to do with the
pending heals that are unable to <br>
</div>
<div dir="ltr">finish.<br>
</div>
<div dir="ltr">Also let me know if some more details are
necessary.<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Thanks,<br>
</div>
<div dir="ltr"> Balazs<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">[<a href="mailto:root@node2" rel="nofollow"
target="_blank" moz-do-not-send="true">root@node2</a>
~]# gluster volume status data<br>
</div>
<div dir="ltr">Status of volume: data<br>
</div>
<div dir="ltr">Gluster process
TCP Port RDMA Port Online Pid<br>
</div>
<div dir="ltr">------------------------------------------------------------------------------<br>
</div>
<div dir="ltr">Brick node1storage.ovirt.local:/gluster_bri<br>
</div>
<div dir="ltr">cks/data/data
49152 0 Y 4187<br>
</div>
<div dir="ltr">Brick node2storage.ovirt.local:/gluster_bri<br>
</div>
<div dir="ltr">cks/data/data
49154 0 Y 6163<br>
</div>
<div dir="ltr">Brick node3storage.ovirt.local:/gluster_bri<br>
</div>
<div dir="ltr">cks/data/data
49154 0 Y 19439<br>
</div>
<div dir="ltr">Self-heal Daemon on localhost
N/A N/A Y 3136<br>
</div>
<div dir="ltr">Self-heal Daemon on node3storage.ovirt.loca<br>
</div>
<div dir="ltr">l
N/A N/A Y 5876<br>
</div>
<div dir="ltr">Self-heal Daemon on node1storage.ovirt.loca<br>
</div>
<div dir="ltr">l
N/A N/A Y 15479<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Task Status of Volume data<br>
</div>
<div dir="ltr">------------------------------------------------------------------------------<br>
</div>
<div dir="ltr">There are no active volume tasks<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">[<a href="mailto:root@node2" rel="nofollow"
target="_blank" moz-do-not-send="true">root@node2</a>
~]#<br>
</div>
<div dir="ltr">[<a href="mailto:root@node2" rel="nofollow"
target="_blank" moz-do-not-send="true">root@node2</a>
~]# gluster volume heal data info summary<br>
</div>
<div dir="ltr">Brick
node1storage.ovirt.local:/gluster_bricks/data/data<br>
</div>
<div dir="ltr">Status: Connected<br>
</div>
<div dir="ltr">Total Number of entries: 2<br>
</div>
<div dir="ltr">Number of entries in heal pending: 2<br>
</div>
<div dir="ltr">Number of entries in split-brain: 0<br>
</div>
<div dir="ltr">Number of entries possibly healing: 0<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Brick
node2storage.ovirt.local:/gluster_bricks/data/data<br>
</div>
<div dir="ltr">Status: Connected<br>
</div>
<div dir="ltr">Total Number of entries: 2<br>
</div>
<div dir="ltr">Number of entries in heal pending: 2<br>
</div>
<div dir="ltr">Number of entries in split-brain: 0<br>
</div>
<div dir="ltr">Number of entries possibly healing: 0<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Brick
node3storage.ovirt.local:/gluster_bricks/data/data<br>
</div>
<div dir="ltr">Status: Connected<br>
</div>
<div dir="ltr">Total Number of entries: 0<br>
</div>
<div dir="ltr">Number of entries in heal pending: 0<br>
</div>
<div dir="ltr">Number of entries in split-brain: 0<br>
</div>
<div dir="ltr">Number of entries possibly healing: 0<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">[<a href="mailto:root@node2" rel="nofollow"
target="_blank" moz-do-not-send="true">root@node2</a>
~]# gluster volume heal data info<br>
</div>
<div dir="ltr">Brick
node1storage.ovirt.local:/gluster_bricks/data/data<br>
</div>
<div dir="ltr">/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/d7c11f2e-58e4-4fe1-8236-6ded0f4dd757/e566f230-df72-4073-aecf-7e5a8d6b569b.meta<br>
</div>
<div dir="ltr">/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/079904a4-71af-492c-bb2f-b45a918e8a2e/fce4b64d-2444-4f11-b226-db75bb2960c2.meta<br>
</div>
<div dir="ltr">Status: Connected<br>
</div>
<div dir="ltr">Number of entries: 2<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Brick
node2storage.ovirt.local:/gluster_bricks/data/data<br>
</div>
<div dir="ltr">/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/d7c11f2e-58e4-4fe1-8236-6ded0f4dd757/e566f230-df72-4073-aecf-7e5a8d6b569b.meta<br>
</div>
<div dir="ltr">/7ac28c32-947b-4ad5-8d69-213a205f06e8/images/079904a4-71af-492c-bb2f-b45a918e8a2e/fce4b64d-2444-4f11-b226-db75bb2960c2.meta<br>
</div>
<div dir="ltr">Status: Connected<br>
</div>
<div dir="ltr">Number of entries: 2<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Brick
node3storage.ovirt.local:/gluster_bricks/data/data<br>
</div>
<div dir="ltr">Status: Connected<br>
</div>
<div dir="ltr">Number of entries: 0<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">________<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Community Meeting Calendar:<br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">APAC Schedule -<br>
</div>
<div dir="ltr">Every 2nd and 4th Tuesday at 11:30 AM IST<br>
</div>
<div dir="ltr">Bridge: <a
href="https://bluejeans.com/441850968" rel="nofollow"
target="_blank" moz-do-not-send="true">https://bluejeans.com/441850968</a><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">NA/EMEA Schedule -<br>
</div>
<div dir="ltr">Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
</div>
<div dir="ltr">Bridge: <a
href="https://bluejeans.com/441850968" rel="nofollow"
target="_blank" moz-do-not-send="true">https://bluejeans.com/441850968</a><br>
</div>
<div dir="ltr"><br>
</div>
<div dir="ltr">Gluster-users mailing list<br>
</div>
<div dir="ltr"><a href="mailto:Gluster-users@gluster.org"
rel="nofollow" target="_blank" moz-do-not-send="true">Gluster-users@gluster.org</a><br>
</div>
<div dir="ltr"><a
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
rel="nofollow" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>