[Gluster-users] Healing completely loss file on replica 3 volume

[Dmitry Antipov]

I'm trying to manually garbage data on bricks (when the volume is
stopped) and then check whether healing is possible. For example:

Start:

# glusterd --debug

Bricks (on EXT4 mounted with 'rw,realtime'):

# mkdir /root/data0
# mkdir /root/data1
# mkdir /root/data2

Volume:

# gluster volume create gv0 replica 3 [local-ip]:/root/data0  [local-ip]:/root/data1  [local-ip]:/root/data2 force
volume create: gv0: success: please start the volume to access data
# gluster volume start gv0
volume start: gv0: success

Mount:

# mkdir /mnt/gv0
# mount -t glusterfs [local-ip]:/gv0 /mnt/gv0
WARNING: getfattr not found, certain checks will be skipped..

Create file:

# openssl rand 65536 > /mnt/gv0/64K
# md5sum /mnt/gv0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K

Umount and down the volume:

# umount /mnt/gv0
# gluster volume stop gv0
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: gv0: success

Check data on bricks:

# md5sum /root/data[012]/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data0/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data1/64K
ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data2/64K

Seems OK. Then garbage all:

# openssl rand 65536 > /root/data0/64K
# openssl rand 65536 > /root/data1/64K
# openssl rand 65536 > /root/data2/64K
# md5sum /root/data[012]/64K
c69096d15007578dab95d9940f89e167  /root/data0/64K
b85292fb60f1a1d27f1b0e3bc6bfdfae  /root/data1/64K
c2e90335cc2f600ddab5c53a992b2bb6  /root/data2/64K

Restart the volume and start full heal:

# gluster volume start gv0
volume start: gv0: success
# /usr/glusterfs/sbin/gluster volume heal gv0 full
Launching heal operation to perform full self heal on volume gv0 has been successful
Use heal info commands to check status.

Finally:

# gluster volume heal gv0 info summary

Brick [local-ip]:/root/data0
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick [local-ip]:/root/data1
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick [local-ip]:/root/data2
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Since all 3 copies are different from each other, majority voting is useless
and data (IIUC) should be marked as split-brain at least. But I'm seeing just
zeroes everywhere above. Why it is so?

Thanks in advance,
Dmitry