<div dir="ltr"><div dir="ltr">Hi Dmitry,<div><br></div><div>Answers inline.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 29, 2019 at 6:26 PM Dmitry Antipov <<a href="mailto:dmantipov@yandex.ru">dmantipov@yandex.ru</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I'm trying to manually garbage data on bricks </blockquote><div>First of all changing data directly on the backend is not recommended and is not supported. All the operations needs to be done from the client mount point.</div><div>Only few special cases needs changing few data about the file directly on the backend.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">(when the volume is<br>
stopped) and then check whether healing is possible. For example:<br>
<br>
Start:<br>
<br>
# glusterd --debug<br>
<br>
Bricks (on EXT4 mounted with 'rw,realtime'):<br>
<br>
# mkdir /root/data0<br>
# mkdir /root/data1<br>
# mkdir /root/data2<br>
<br>
Volume:<br>
<br>
# gluster volume create gv0 replica 3 [local-ip]:/root/data0 [local-ip]:/root/data1 [local-ip]:/root/data2 force<br>
volume create: gv0: success: please start the volume to access data<br>
# gluster volume start gv0<br>
volume start: gv0: success<br>
<br>
Mount:<br>
<br>
# mkdir /mnt/gv0<br>
# mount -t glusterfs [local-ip]:/gv0 /mnt/gv0<br>
WARNING: getfattr not found, certain checks will be skipped..<br>
<br>
Create file:<br>
<br>
# openssl rand 65536 > /mnt/gv0/64K<br>
# md5sum /mnt/gv0/64K<br>
ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K<br>
<br>
Umount and down the volume:<br>
<br>
# umount /mnt/gv0<br>
# gluster volume stop gv0<br>
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y<br>
volume stop: gv0: success<br>
<br>
Check data on bricks:<br>
<br>
# md5sum /root/data[012]/64K<br>
ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data0/64K<br>
ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data1/64K<br>
ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data2/64K<br>
<br>
Seems OK. Then garbage all:<br>
<br>
# openssl rand 65536 > /root/data0/64K<br>
# openssl rand 65536 > /root/data1/64K<br>
# openssl rand 65536 > /root/data2/64K<br>
# md5sum /root/data[012]/64K<br>
c69096d15007578dab95d9940f89e167 /root/data0/64K<br>
b85292fb60f1a1d27f1b0e3bc6bfdfae /root/data1/64K<br>
c2e90335cc2f600ddab5c53a992b2bb6 /root/data2/64K<br>
<br>
Restart the volume and start full heal:<br>
<br>
# gluster volume start gv0<br>
volume start: gv0: success<br>
# /usr/glusterfs/sbin/gluster volume heal gv0 full<br>
Launching heal operation to perform full self heal on volume gv0 has been successful<br>
Use heal info commands to check status.<br>
<br>
Finally:<br>
<br>
# gluster volume heal gv0 info summary<br>
<br>
Brick [local-ip]:/root/data0<br>
Status: Connected<br>
Total Number of entries: 0<br>
Number of entries in heal pending: 0<br>
Number of entries in split-brain: 0<br>
Number of entries possibly healing: 0<br>
<br>
Brick [local-ip]:/root/data1<br>
Status: Connected<br>
Total Number of entries: 0<br>
Number of entries in heal pending: 0<br>
Number of entries in split-brain: 0<br>
Number of entries possibly healing: 0<br>
<br>
Brick [local-ip]:/root/data2<br>
Status: Connected<br>
Total Number of entries: 0<br>
Number of entries in heal pending: 0<br>
Number of entries in split-brain: 0<br>
Number of entries possibly healing: 0<br>
<br>
Since all 3 copies are different from each other, majority voting is useless<br>
and data (IIUC) should be marked as split-brain at least. But I'm seeing just<br>
zeroes everywhere above. Why it is so?<br></blockquote><div>Since the data is changed directly on the backend, gluster will not be knowing these changes. If the changes done from the client mount fails on some bricks, only those will be recognized and marked by gluster so that it can heal those when possible. Since this is a replica 3 volume and if you end up in split-brain when you are doing the operations on the mount pint, then that will be a bug. As far as this is considered it is not a bug or issue on the gluster side.</div><div><br></div><div>HTH,</div><div>Karthik</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks in advance,<br>
Dmitry<br>
________<br>
<br>
Community Meeting Calendar:<br>
<br>
APAC Schedule -<br>
Every 2nd and 4th Tuesday at 11:30 AM IST<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
NA/EMEA Schedule -<br>
Every 1st and 3rd Tuesday at 01:00 PM EDT<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
</blockquote></div></div>