[Gluster-users] Healing completely loss file on replica 3 volume

[Karthik Subrahmanya]

Hi Dmitry,

Answers inline.

On Fri, Nov 29, 2019 at 6:26 PM Dmitry Antipov <dmantipov at yandex.ru> wrote:

> I'm trying to manually garbage data on bricks

First of all changing data directly on the backend is not recommended and
is not supported.  All the operations needs to be done from the client
mount point.
Only few special cases needs changing few data about the file directly on
the backend.

> (when the volume is
> stopped) and then check whether healing is possible. For example:
>
> Start:
>
> # glusterd --debug
>
> Bricks (on EXT4 mounted with 'rw,realtime'):
>
> # mkdir /root/data0
> # mkdir /root/data1
> # mkdir /root/data2
>
> Volume:
>
> # gluster volume create gv0 replica 3 [local-ip]:/root/data0
> [local-ip]:/root/data1  [local-ip]:/root/data2 force
> volume create: gv0: success: please start the volume to access data
> # gluster volume start gv0
> volume start: gv0: success
>
> Mount:
>
> # mkdir /mnt/gv0
> # mount -t glusterfs [local-ip]:/gv0 /mnt/gv0
> WARNING: getfattr not found, certain checks will be skipped..
>
> Create file:
>
> # openssl rand 65536 > /mnt/gv0/64K
> # md5sum /mnt/gv0/64K
> ca53c9c1b6cd78f59a91cd1b0b866ed9 /mnt/gv0/64K
>
> Umount and down the volume:
>
> # umount /mnt/gv0
> # gluster volume stop gv0
> Stopping volume will make its data inaccessible. Do you want to continue?
> (y/n) y
> volume stop: gv0: success
>
> Check data on bricks:
>
> # md5sum /root/data[012]/64K
> ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data0/64K
> ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data1/64K
> ca53c9c1b6cd78f59a91cd1b0b866ed9  /root/data2/64K
>
> Seems OK. Then garbage all:
>
> # openssl rand 65536 > /root/data0/64K
> # openssl rand 65536 > /root/data1/64K
> # openssl rand 65536 > /root/data2/64K
> # md5sum /root/data[012]/64K
> c69096d15007578dab95d9940f89e167  /root/data0/64K
> b85292fb60f1a1d27f1b0e3bc6bfdfae  /root/data1/64K
> c2e90335cc2f600ddab5c53a992b2bb6  /root/data2/64K
>
> Restart the volume and start full heal:
>
> # gluster volume start gv0
> volume start: gv0: success
> # /usr/glusterfs/sbin/gluster volume heal gv0 full
> Launching heal operation to perform full self heal on volume gv0 has been
> successful
> Use heal info commands to check status.
>
> Finally:
>
> # gluster volume heal gv0 info summary
>
> Brick [local-ip]:/root/data0
> Status: Connected
> Total Number of entries: 0
> Number of entries in heal pending: 0
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> Brick [local-ip]:/root/data1
> Status: Connected
> Total Number of entries: 0
> Number of entries in heal pending: 0
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> Brick [local-ip]:/root/data2
> Status: Connected
> Total Number of entries: 0
> Number of entries in heal pending: 0
> Number of entries in split-brain: 0
> Number of entries possibly healing: 0
>
> Since all 3 copies are different from each other, majority voting is
> useless
> and data (IIUC) should be marked as split-brain at least. But I'm seeing
> just
> zeroes everywhere above. Why it is so?
>
Since the data is changed directly on the backend, gluster will not be
knowing these changes. If the changes done from the client mount fails on
some bricks, only those will be recognized and marked by gluster so that it
can heal those when possible. Since this is a replica 3 volume and if you
end up in split-brain when you are doing the operations on the mount pint,
then that will be a bug. As far as this is considered it is not a bug or
issue on the gluster side.

HTH,
Karthik

>
> Thanks in advance,
> Dmitry
> ________
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191202/638e3941/attachment.html>