[Gluster-users] Healing completely loss file on replica 3 volume

[Strahil Nikolov]

 Hi Dmitry,
keep in mind that gluster doesn't run md5sum (or any SHAsum) against the local copies on the bricks , in order to verify your data.
The simplest said, it relies on entries in gluster's "/brick/.glusterfs" folder and the metadata (getfattr -d -m . -e hex <file-path-on-brick>.) differences in order to detect and fight split brain.In your test , you have overwritten the contents of the file , but metadata is the old one and also there are no entries in "/brick/.glusterfs" - as last operation was successful (all bricks were online).
Your test is not properly testing the software and as far as I know it's not expecting that someone will mess up with all 3 bricks :D.

Best Regards,Strahil Nikolov
    В петък, 29 ноември 2019 г., 14:55:56 ч. Гринуич+2, Dmitry Antipov <dmantipov at yandex.ru> написа:  
 
 I'm trying to manually garbage data on bricks (when the volume is
stopped) and then check whether healing is possible. For example:

Start:

# glusterd --debug

Bricks (on EXT4 mounted with 'rw,realtime'):

# mkdir /root/data0
# mkdir /root/data1
# mkdir /root/data2

Volume:

# gluster volume create gv0 replica 3 [local-ip]:/root/data0  [local-ip]:/root/data1  [local-ip]:/root/data2 force
volume create: gv0: success: please start the volume to access data
# gluster volume start gv0
volume start: gv0: success

Mount:

# mkdir /mnt/gv0
# mount -t glusterfs [local-ip]:/gv0 /mnt/gv0
WARNING: getfattr not found, certain checks will be skipped..

Create file:

# openssl rand 65536 > /mnt/gv0/64K
# md5sum /mnt/gv0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K

Umount and down the volume:

# umount /mnt/gv0
# gluster volume stop gv0
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: gv0: success

Check data on bricks:

# md5sum /root/data[012]/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data1/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data2/64K

Seems OK. Then garbage all:

# openssl rand 65536 > /root/data0/64K
# openssl rand 65536 > /root/data1/64K
# openssl rand 65536 > /root/data2/64K
# md5sum /root/data[012]/64K
c69096d15007578dab95d9940f89e167  /root/data0/64K
b85292fb60f1a1d27f1b0e3bc6bfdfae  /root/data1/64K
c2e90335cc2f600ddab5c53a992b2bb6  /root/data2/64K

Restart the volume and start full heal:

# gluster volume start gv0
volume start: gv0: success
# /usr/glusterfs/sbin/gluster volume heal gv0 full
Launching heal operation to perform full self heal on volume gv0 has been successful
Use heal info commands to check status.

Finally:

# gluster volume heal gv0 info summary

Brick [local-ip]:/root/data0
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick [local-ip]:/root/data1
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick [local-ip]:/root/data2
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Since all 3 copies are different from each other, majority voting is useless
and data (IIUC) should be marked as split-brain at least. But I'm seeing just
zeroes everywhere above. Why it is so?

Thanks in advance,
Dmitry
________

Community Meeting Calendar:

APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: https://bluejeans.com/441850968

NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191130/31144ca4/attachment-0001.html>