[Bugs] [Bug 1741899] New: the volume of occupied space in the bricks of gluster volume (3 nodes replica) differs on nodes and the healing does not fix it

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1741899

            Bug ID: 1741899
           Summary: the volume of occupied space in the bricks of gluster
                    volume (3 nodes replica) differs on nodes and the
                    healing does not fix it
           Product: GlusterFS
           Version: 5
          Hardware: x86_64
                OS: Linux
            Status: NEW
         Component: glusterd
          Severity: medium
          Assignee: bugs at gluster.org
          Reporter: s.pleshkov at hostco.ru
                CC: bugs at gluster.org
  Target Milestone: ---
    Classification: Community



Description of problem:
I have a gluster volume on 3 nodes (replicate) with following configuration

[root at LSY-GL-0(1,2,3) /]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.6 (Maipo)

[root at LSY-GL-02 host]# gluster volume info TST

Volume Name: TST
Type: Replicate
Volume ID: a96c7b8c-61ec-4a4d-b47e-b445faf6c39b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: lsy-gl-01:/diskForTestData/tst
Brick2: lsy-gl-02:/diskForTestData/tst
Brick3: lsy-gl-03:/diskForTestData/tst
Options Reconfigured:
cluster.favorite-child-policy: size
features.shard-block-size: 64MB
features.shard: on
performance.io-thread-count: 24
client.event-threads: 24
server.event-threads: 24
server.allow-insecure: on
network.ping-timeout: 5
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
cluster.heal-timeout: 120

Recently this volume been moved to other disk by command
gluster volume replace-brick TST lsy-gl-0(1,2,3):/diskForData/tst
lsy-gl-0(1,2,3):/diskForTestData/tst commit force

sequentialy, started with lsy-gl-03 node, all nodes been online

And now i have this state

[root at LSY-GL-02 host]# gluster volume status TST detail
Status of volume: TST
------------------------------------------------------------------------------
Brick                : Brick lsy-gl-01:/diskForTestData/tst
TCP Port             : 49154
RDMA Port            : 0
Online               : Y
Pid                  : 7555
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 399.9GB
Total Disk Space     : 499.8GB
Inode Count          : 262143488
Free Inodes          : 261684925
------------------------------------------------------------------------------
Brick                : Brick lsy-gl-02:/diskForTestData/tst
TCP Port             : 49154
RDMA Port            : 0
Online               : Y
Pid                  : 25732
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 399.9GB
Total Disk Space     : 499.8GB
Inode Count          : 262143488
Free Inodes          : 261684925
------------------------------------------------------------------------------
Brick                : Brick lsy-gl-03:/diskForTestData/tst
TCP Port             : 49154
RDMA Port            : 0
Online               : Y
Pid                  : 25243
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size           : 512
Disk Space Free      : 357.6GB
Total Disk Space     : 499.8GB
Inode Count          : 262143488
Free Inodes          : 261684112

[root at LSY-GL-02 host]# gluster volume heal TST full
Launching heal operation to perform full self heal on volume TST has been
successful
Use heal info commands to check status.
[root at LSY-GL-02 host]# gluster volume heal TST info
Brick lsy-gl-01:/diskForTestData/tst
Status: Connected
Number of entries: 0

Brick lsy-gl-02:/diskForTestData/tst
Status: Connected
Number of entries: 0

Brick lsy-gl-03:/diskForTestData/tst
Status: Connected
Number of entries: 0


[root at LSY-GL-01 /]# df -Th
Filesystem                        Type            Size  Used Avail Use% Mounted
on
LSY-GL-01:/TST                    fuse.glusterfs  500G  148G  353G  30%
/mnt/tst
/dev/sdc1                         xfs             500G  100G  400G  20%
/diskForTestData

[root at LSY-GL-02 host]# df -Th
Filesystem                        Type            Size  Used Avail Use% Mounted
on
LSY-GL-02:/TST                    fuse.glusterfs  500G  148G  353G  30%
/mnt/tst
/dev/sdc1                         xfs             500G  100G  400G  20%
/diskForTestData


[root at LSY-GL-03 host]# df -Th
Filesystem                        Type            Size  Used Avail Use% Mounted
on
/dev/sdc1                         xfs             500G  143G  358G  29%
/diskForTestData
LSY-GL-03:/TST                    fuse.glusterfs  500G  148G  353G  30%
/mnt/tst

Version-Release number of selected component (if applicable):

[root at LSY-GL-0(1,2,3) /]# rpm -qa | grep gluster*
glusterfs-libs-5.5-1.el7.x86_64
glusterfs-fuse-5.5-1.el7.x86_64
glusterfs-client-xlators-5.5-1.el7.x86_64
centos-release-gluster5-1.0-1.el7.centos.noarch
glusterfs-api-5.5-1.el7.x86_64
glusterfs-cli-5.5-1.el7.x86_64
nfs-ganesha-gluster-2.7.1-1.el7.x86_64
glusterfs-5.5-1.el7.x86_64
glusterfs-server-5.5-1.el7.x86_64

How reproducible:

Umm, I will test it again soon and do comment

Steps to Reproduce:
1.
2.
3.

Actual results:
Size of brick on lsy-gl-01, and lsy-gl-02 differ from size brick on lsy-gl-03.
Healing full not fixed this situation

Expected results:
What things I should do to fix it ?

Additional info:

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.