<html><head></head><body><div class="ydp2aae5160yahoo-style-wrap" style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;"><div style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;"></div>
<div dir="ltr" data-setdir="false" style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;">Hi Dmitry,</div><div dir="ltr" data-setdir="false" style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;"><br></div><div dir="ltr" data-setdir="false" style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;">keep in mind that gluster doesn't run md5sum (or any SHAsum) against the local copies on the bricks , in order to verify your data.</div><div dir="ltr" data-setdir="false" style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;"><br></div><div dir="ltr" data-setdir="false" style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;">The simplest said, it relies on entries in gluster's "/brick/.glusterfs" folder and the metadata (<span><code style="color: rgb(55, 71, 79); font-family: Roboto Mono, Courier New, Courier, monospace; font-size: 13.6px; direction: ltr; margin: 0px 0.29412em; padding: 0.07353em 0px;">getfattr -d -m . -e hex <file-path-on-brick></code><span style="font-family: Roboto, Helvetica Neue, Helvetica, Arial, sans-serif; font-size: 16px;">.) differences in order to detect and fight split brain.</span></span></div><div dir="ltr" data-setdir="false" style=""><font face="Roboto, Helvetica Neue, Helvetica, Arial, sans-serif">In your test , you have overwritten the contents of the file , but metadata is the old one and also there are no entries in </font>"/brick/.glusterfs" - as last operation was successful (all bricks were online).</div><div dir="ltr" data-setdir="false" style=""><br></div><div dir="ltr" data-setdir="false" style="">Your test is not properly testing the software and as far as I know it's not expecting that someone will mess up with all 3 bricks :D.</div><div dir="ltr" data-setdir="false" style=""><br></div><div dir="ltr" data-setdir="false" style=""><br></div><div dir="ltr" data-setdir="false" style="">Best Regards,</div><div dir="ltr" data-setdir="false" style="">Strahil Nikolov</div><div style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;"><br></div>
</div><div id="ydp4213d278yahoo_quoted_5457364751" class="ydp4213d278yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
В петък, 29 ноември 2019 г., 14:55:56 ч. Гринуич+2, Dmitry Antipov <dmantipov@yandex.ru> написа:
</div>
<div><br></div>
<div><br></div>
<div><div dir="ltr">I'm trying to manually garbage data on bricks (when the volume is<br></div><div dir="ltr">stopped) and then check whether healing is possible. For example:<br></div><div dir="ltr"><br></div><div dir="ltr">Start:<br></div><div dir="ltr"><br></div><div dir="ltr"># glusterd --debug<br></div><div dir="ltr"><br></div><div dir="ltr">Bricks (on EXT4 mounted with 'rw,realtime'):<br></div><div dir="ltr"><br></div><div dir="ltr"># mkdir /root/data0<br></div><div dir="ltr"># mkdir /root/data1<br></div><div dir="ltr"># mkdir /root/data2<br></div><div dir="ltr"><br></div><div dir="ltr">Volume:<br></div><div dir="ltr"><br></div><div dir="ltr"># gluster volume create gv0 replica 3 [local-ip]:/root/data0 [local-ip]:/root/data1 [local-ip]:/root/data2 force<br></div><div dir="ltr">volume create: gv0: success: please start the volume to access data<br></div><div dir="ltr"># gluster volume start gv0<br></div><div dir="ltr">volume start: gv0: success<br></div><div dir="ltr"><br></div><div dir="ltr">Mount:<br></div><div dir="ltr"><br></div><div dir="ltr"># mkdir /mnt/gv0<br></div><div dir="ltr"># mount -t glusterfs [local-ip]:/gv0 /mnt/gv0<br></div><div dir="ltr">WARNING: getfattr not found, certain checks will be skipped..<br></div><div dir="ltr"><br></div><div dir="ltr">Create file:<br></div><div dir="ltr"><br></div><div dir="ltr"># openssl rand 65536 > /mnt/gv0/64K<br></div><div dir="ltr"># md5sum /mnt/gv0/64K<br></div><div dir="ltr">ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K<br></div><div dir="ltr"><br></div><div dir="ltr">Umount and down the volume:<br></div><div dir="ltr"><br></div><div dir="ltr"># umount /mnt/gv0<br></div><div dir="ltr"># gluster volume stop gv0<br></div><div dir="ltr">Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y<br></div><div dir="ltr">volume stop: gv0: success<br></div><div dir="ltr"><br></div><div dir="ltr">Check data on bricks:<br></div><div dir="ltr"><br></div><div dir="ltr"># md5sum /root/data[012]/64K<br></div><div dir="ltr">ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data0/64K<br></div><div dir="ltr">ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data1/64K<br></div><div dir="ltr">ca53c9c1b6cd78f59a91cd1b0b866ed9 /root/data2/64K<br></div><div dir="ltr"><br></div><div dir="ltr">Seems OK. Then garbage all:<br></div><div dir="ltr"><br></div><div dir="ltr"># openssl rand 65536 > /root/data0/64K<br></div><div dir="ltr"># openssl rand 65536 > /root/data1/64K<br></div><div dir="ltr"># openssl rand 65536 > /root/data2/64K<br></div><div dir="ltr"># md5sum /root/data[012]/64K<br></div><div dir="ltr">c69096d15007578dab95d9940f89e167 /root/data0/64K<br></div><div dir="ltr">b85292fb60f1a1d27f1b0e3bc6bfdfae /root/data1/64K<br></div><div dir="ltr">c2e90335cc2f600ddab5c53a992b2bb6 /root/data2/64K<br></div><div dir="ltr"><br></div><div dir="ltr">Restart the volume and start full heal:<br></div><div dir="ltr"><br></div><div dir="ltr"># gluster volume start gv0<br></div><div dir="ltr">volume start: gv0: success<br></div><div dir="ltr"># /usr/glusterfs/sbin/gluster volume heal gv0 full<br></div><div dir="ltr">Launching heal operation to perform full self heal on volume gv0 has been successful<br></div><div dir="ltr">Use heal info commands to check status.<br></div><div dir="ltr"><br></div><div dir="ltr">Finally:<br></div><div dir="ltr"><br></div><div dir="ltr"># gluster volume heal gv0 info summary<br></div><div dir="ltr"><br></div><div dir="ltr">Brick [local-ip]:/root/data0<br></div><div dir="ltr">Status: Connected<br></div><div dir="ltr">Total Number of entries: 0<br></div><div dir="ltr">Number of entries in heal pending: 0<br></div><div dir="ltr">Number of entries in split-brain: 0<br></div><div dir="ltr">Number of entries possibly healing: 0<br></div><div dir="ltr"><br></div><div dir="ltr">Brick [local-ip]:/root/data1<br></div><div dir="ltr">Status: Connected<br></div><div dir="ltr">Total Number of entries: 0<br></div><div dir="ltr">Number of entries in heal pending: 0<br></div><div dir="ltr">Number of entries in split-brain: 0<br></div><div dir="ltr">Number of entries possibly healing: 0<br></div><div dir="ltr"><br></div><div dir="ltr">Brick [local-ip]:/root/data2<br></div><div dir="ltr">Status: Connected<br></div><div dir="ltr">Total Number of entries: 0<br></div><div dir="ltr">Number of entries in heal pending: 0<br></div><div dir="ltr">Number of entries in split-brain: 0<br></div><div dir="ltr">Number of entries possibly healing: 0<br></div><div dir="ltr"><br></div><div dir="ltr">Since all 3 copies are different from each other, majority voting is useless<br></div><div dir="ltr">and data (IIUC) should be marked as split-brain at least. But I'm seeing just<br></div><div dir="ltr">zeroes everywhere above. Why it is so?<br></div><div dir="ltr"><br></div><div dir="ltr">Thanks in advance,<br></div><div dir="ltr">Dmitry<br></div><div dir="ltr">________<br></div><div dir="ltr"><br></div><div dir="ltr">Community Meeting Calendar:<br></div><div dir="ltr"><br></div><div dir="ltr">APAC Schedule -<br></div><div dir="ltr">Every 2nd and 4th Tuesday at 11:30 AM IST<br></div><div dir="ltr">Bridge: <a href="https://bluejeans.com/441850968" rel="nofollow" target="_blank">https://bluejeans.com/441850968</a><br></div><div dir="ltr"><br></div><div dir="ltr">NA/EMEA Schedule -<br></div><div dir="ltr">Every 1st and 3rd Tuesday at 01:00 PM EDT<br></div><div dir="ltr">Bridge: <a href="https://bluejeans.com/441850968" rel="nofollow" target="_blank">https://bluejeans.com/441850968</a><br></div><div dir="ltr"><br></div><div dir="ltr">Gluster-users mailing list<br></div><div dir="ltr"><a href="mailto:Gluster-users@gluster.org" rel="nofollow" target="_blank">Gluster-users@gluster.org</a><br></div><div dir="ltr"><a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="nofollow" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></div></div>
</div>
</div></body></html>