<html><head></head><body><div class="yahoo-style-wrap" style="font-family:courier new, courier, monaco, monospace, sans-serif;font-size:16px;"><div dir="ltr" data-setdir="false">Hello Colleagues,</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">I want to share my experience and especially how I have recovered after a situation where gluster refuses to heal several files of my oVirt Lab.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Most probably the situation was caused by the fact that I didn't check if all files were healed before I started the upgrade on the next node , which in a 'replica 2 arbiter1' setup has caused multiple files missing/conflicting and heal fails to happen.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Background:</div><div dir="ltr" data-setdir="false">1. I have powered off my HostedEngine VM and made a snapshot of the gluster volume</div><div dir="ltr" data-setdir="false">2. Started the update , but without screen - I messed it up and decided to revert from the snapshot</div><div dir="ltr" data-setdir="false">3. Powered off the volume , restored from snapshot and then started (and again snapshoted the volume)</div><div dir="ltr" data-setdir="false">4. Upgrade of the HostedEngine was successfull</div><div dir="ltr" data-setdir="false">5. Upgraded the arbiter (ovirt3)</div><div dir="ltr" data-setdir="false">6. I forgot to check the heal status on the arbiter and upgraded ovirt1/gluster1 (which maybe was the reason for the issue)</div><div dir="ltr" data-setdir="false">7. After gluster1 was healed I saw that some files are left for healing , but expected it will finish till ovirt2/gluster2 is patched</div><div dir="ltr" data-setdir="false">8. Sadly my assumption was not right and after the reboot of ovirt2/gluster2 I noticed that some files never heal.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Symptoms:</div><div dir="ltr" data-setdir="false">2 files per volume never heal (used 'full' mode) even after I 'stat'-ed every file/dir in the volume.</div><div dir="ltr" data-setdir="false">Ovirt Dashboard reported multiple errors (50+) that it cannot update the OVF metadata for volume/VM.</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false">Here are my notes while I was recovering from the situation. As this is my LAB, I shutdown all VMs (including the HostedEngine) as downtime was not an issue:</div><div dir="ltr" data-setdir="false"><br></div><div dir="ltr" data-setdir="false"><div><span style="font-family:monospace"><span style="color: rgb(0, 0, 0);">Heals never complete:
</span><br>
<br># gluster volume heal data_fast4 info
<br>Brick gluster1:/gluster_bricks/data_fast4/data_fast4
<br>Status: Connected
<br>Number of entries: 0
<br>
<br>Brick gluster2:/gluster_bricks/data_fast4/data_fast4
<br><gfid:d21a6512-eaf6-4859-90cf-eeef2cc0cab8> <br><gfid:95bc2cd2-8a1e-464e-a384-6b128780d370> <br>Status: Connected
<br>Number of entries: 2
<br>
<br>Brick ovirt3:/gluster_bricks/data_fast4/data_fast4
<br><gfid:d21a6512-eaf6-4859-90cf-eeef2cc0cab8> <br><gfid:95bc2cd2-8a1e-464e-a384-6b128780d370> <br>Status: Connected
<br>Number of entries: 2
<br>
<br>
<br>Mount to get the gfid to path relationship:
<br>mount -t glusterfs -o aux-gfid-mount gluster1:/data_fast4 /mnt
<br>
<br># getfattr -n trusted.glusterfs.pathinfo -e text /mnt/.gfid/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
<br>getfattr: Removing leading '/' from absolute path names
<br># file: mnt/.gfid/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
<br> trusted.glusterfs.pathinfo="(<REPLICATE:data_fast4-replicate-0> <POSIX(/gluster_bricks/data_fast4/data_fast4):ovirt2.localdomain:/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a651<br>2-eaf6-4859-90cf-eeef2cc0cab8> <POSIX(/gluster_bricks/data_fast4/data_fast4):ovirt3.localdomain:/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8>)"
<br>
<br>The local brick (that is supposed to be healed) is missing some data:
<br>
<br># ls -l /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
<br>ls: няма достъп до /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8: Няма такъв файл или директория
<br>
<br>Remote is OK:
<br># ssh gluster2 'ls -l /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8'
<br>-rw-r--r--. 2 vdsm kvm 436 9 ное 19,34 /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
<br>
<br>Arbiter is also OK:
<br># ssh ovirt3 'ls -l /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8'
<br>-rw-r--r--. 2 vdsm kvm 0 9 ное 19,34 /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
<br>
<br>
<br>Rsync the file/directory from a good brick to broken one:
<br>
<br># rsync -avP gluster2:/gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/ /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/
<br># ls -l /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
<br>-rw-r--r--. 1 vdsm kvm 436 9 ное 19,34 /gluster_bricks/data_fast4/data_fast4/.glusterfs/d2/1a/d21a6512-eaf6-4859-90cf-eeef2cc0cab8
<br>
<br>After a full heal we see our problematic file:
<br>
<br># gluster volume heal data_fast4 full
<br>Launching heal operation to perform full self heal on volume data_fast4 has been successful <br>Use heal info commands to check status.
<br>
<br># gluster volume heal data_fast4 info
<br>Brick gluster1:/gluster_bricks/data_fast4/data_fast4
<br>Status: Connected
<br>Number of entries: 0
<br>
<br>Brick gluster2:/gluster_bricks/data_fast4/data_fast4
<br>/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta <br>Status: Connected
<br>Number of entries: 1
<br>
<br>Brick ovirt3:/gluster_bricks/data_fast4/data_fast4
<br>/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta <br>Status: Connected
<br>Number of entries: 1
<br>
<br>
<br>This time the file is missing on both gluster1 (older version) and arbiter:
<br>
<br># cat /gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
<br>CTIME=1558265783
<br>DESCRIPTION={"Updated":true,"Size":256000,"Last Updated":"Sat Nov 09 19:24:08 EET 2019","Storage Domains":[{"uuid":"578bca3d-6540-41cd-8e0e-9e3047026484"}],"Disk Description":"OVF_STORE"}
<br>DISKTYPE=OVFS
<br>DOMAIN=578bca3d-6540-41cd-8e0e-9e3047026484
<br>FORMAT=RAW
<br>GEN=0
<br>IMAGE=58e197a6-12df-4432-a643-298d40e44130
<br>LEGALITY=LEGAL
<br>MTIME=0
<br>PUUID=00000000-0000-0000-0000-000000000000
<br>SIZE=262144
<br>TYPE=PREALLOCATED
<br>VOLTYPE=LEAF
<br>EOF
<br>
<br># ssh gluster2 cat /gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
<br>CTIME=1558265783
<br>DESCRIPTION={"Updated":true,"Size":256000,"Last Updated":"Sat Nov 09 19:34:33 EET 2019","Storage Domains":[{"uuid":"578bca3d-6540-41cd-8e0e-9e3047026484"}],"Disk Description":"OVF_STORE"}
<br>DISKTYPE=OVFS
<br>DOMAIN=578bca3d-6540-41cd-8e0e-9e3047026484
<br>FORMAT=RAW
<br>GEN=0
<br>IMAGE=58e197a6-12df-4432-a643-298d40e44130
<br>LEGALITY=LEGAL
<br>MTIME=0
<br>PUUID=00000000-0000-0000-0000-000000000000
<br>SIZE=262144
<br>TYPE=PREALLOCATED
<br>VOLTYPE=LEAF
<br>EOF
<br>
<br># ssh ovirt3 cat /578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
<br>cat: /578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta: Няма такъв файл или директория
<br>
<br>As we miss the same file on both a brick and arbiter we take another approach:
<br>First remove on gluster1 the file to another name:
<br>
<br># mv /gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta /gluster_bricks/data_fast4<br>/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta_old
<br>
<br>Then we copy the file from gluster2 (good brick)
<br># rsync -avP gluster2:/gluster_bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta /gluster_<br>bricks/data_fast4/data_fast4/578bca3d-6540-41cd-8e0e-9e3047026484/images/58e197a6-12df-4432-a643-298d40e44130/535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
<br>receiving incremental file list
<br>535ec7f7-f4d1-4d1e-a988-c1e95b4a38ca.meta
<br> 436 100% 425.78kB/s 0:00:00 (xfr#1, to-chk=0/1)
<br>
<br>sent 43 bytes received 571 bytes 1,228.00 bytes/sec
<br>total size is 436 speedup is 0.71
<br>
<br>Run full heal:
<br># gluster volume heal data_fast4 full
<br>Launching heal operation to perform full self heal on volume data_fast4 has been successful <br>Use heal info commands to check status
<br>
<br># gluster volume heal data_fast4 info
<br>Brick gluster1:/gluster_bricks/data_fast4/data_fast4
<br>Status: Connected
<br>Number of entries: 0
<br>
<br>Brick gluster2:/gluster_bricks/data_fast4/data_fast4
<br>Status: Connected
<br>Number of entries: 0
<br>
<br>Brick ovirt3:/gluster_bricks/data_fast4/data_fast4
<br>Status: Connected
<br>Number of entries: 0
<br>
<br>
<br>And ofcourse umount /mnt</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace"><br></span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace">I did the above for all oVirt Storage domains and rebooted all nodes (simultaneously) after stopping the whole stack. This one should not be neccessary , but I wanted to be sure that after power outage the cluster will be operational again:</span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><br></span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace">systemctl stop ovirt-ha-agent ovirt-ha-broker vdsmd supervdsmd sanlock glusterd</span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><span><span style="font-family:monospace"><span style="color: rgb(0, 0, 0);">/usr/share/glusterfs/scripts/stop-all-gluster-processes.sh</span><br></span></span></span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><span><span style="font-family:monospace"><span style="color: rgb(0, 0, 0);"><br></span></span></span></span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><span><span style="font-family:monospace"><span style="color: rgb(0, 0, 0);">Verification:</span></span></span></span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><span><span style="font-family:monospace"><span style="color: rgb(0, 0, 0);">After the reboot , I have tried to set each oVirt storage domain in 'Maintenance' which confirms that engine can update the OVMF meta and then set it back to Active. Without downtime , this will not be possible.</span></span></span></span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><span><span style="font-family:monospace"><span style="color: rgb(0, 0, 0);"><br></span></span></span></span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><br></span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace">I hope this long post will help anyone .</span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><br></span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace">PS: I have collected some data for some of the files, that I have ommited as this e-mail is very long.</span></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><br></span></div><div dir="ltr" data-setdir="false"><font face="monospace">Best Regards,</font></div><div dir="ltr" data-setdir="false"><font face="monospace">Strahil Nikolov</font></div><div dir="ltr" data-setdir="false"><span style="font-family:monospace"><br>
<br></span></div><br></div></div></body></html>