[Gluster-users] df reports wrong full capacity for distributed volumes (Glusterfs 3.12.6-1)

[Jose V. Carrión]

Hi,

Some days ago all my glusterfs configuration was working fine. Today I
realized that the total size reported by df command was changed and is
smaller than the aggregated capacity of all the bricks in the volume.

I checked that all the volumes status are fine, all the glusterd daemons
are running, there is no error in logs,  however df shows a bad total size.

My configuration for one volume: volumedisk1
[root at stor1 ~]# gluster volume status volumedisk1  detail

Status of volume: volumedisk1
------------------------------------------------------------------------------
Brick                : Brick stor1data:/mnt/glusterfs/vol1/brick1
TCP Port             : 49153
RDMA Port            : 0
Online               : Y
Pid                  : 13579
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw,noatime
Inode Size           : 512
Disk Space Free      : 35.0TB
Total Disk Space     : 49.1TB
Inode Count          : 5273970048
Free Inodes          : 5273123069
------------------------------------------------------------------------------
Brick                : Brick stor2data:/mnt/glusterfs/vol1/brick1
TCP Port             : 49153
RDMA Port            : 0
Online               : Y
Pid                  : 13344
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw,noatime
Inode Size           : 512
Disk Space Free      : 35.0TB
Total Disk Space     : 49.1TB
Inode Count          : 5273970048
Free Inodes          : 5273124718
------------------------------------------------------------------------------
Brick                : Brick stor3data:/mnt/disk_c/glusterfs/vol1/brick1
TCP Port             : 49154
RDMA Port            : 0
Online               : Y
Pid                  : 17439
File System          : xfs
Device               : /dev/sdc1
Mount Options        : rw,noatime
Inode Size           : 512
Disk Space Free      : 35.7TB
Total Disk Space     : 49.1TB
Inode Count          : 5273970048
Free Inodes          : 5273125437
------------------------------------------------------------------------------
Brick                : Brick stor3data:/mnt/disk_d/glusterfs/vol1/brick1
TCP Port             : 49155
RDMA Port            : 0
Online               : Y
Pid                  : 17459
File System          : xfs
Device               : /dev/sdd1
Mount Options        : rw,noatime
Inode Size           : 512
Disk Space Free      : 35.6TB
Total Disk Space     : 49.1TB
Inode Count          : 5273970048
Free Inodes          : 5273127036


Then full size for volumedisk1 should be: 49.1TB + 49.1TB + 49.1TB +49.1TB
= *196,4 TB  *but df shows:

[root at stor1 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2              48G   21G   25G  46% /
tmpfs                  32G   80K   32G   1% /dev/shm
/dev/sda1             190M   62M  119M  35% /boot
/dev/sda4             395G  251G  124G  68% /data
/dev/sdb1              26T  601G   25T   3% /mnt/glusterfs/vol0
/dev/sdc1              50T   15T   36T  29% /mnt/glusterfs/vol1
stor1data:/volumedisk0
                       76T  1,6T   74T   3% /volumedisk0
stor1data:/volumedisk1
                      *148T*   42T  106T  29% /volumedisk1

Exactly 1 brick minus: 196,4 TB - 49,1TB = 148TB

It's a production system so I hope you can help me.

Thanks in advance.

Jose V.


Below some other data of my configuration:

[root at stor1 ~]# gluster volume info

Volume Name: volumedisk0
Type: Distribute
Volume ID: 0ee52d94-1131-4061-bcef-bd8cf898da10
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol0/brick1
Brick2: stor2data:/mnt/glusterfs/vol0/brick1
Brick3: stor3data:/mnt/disk_b1/glusterfs/vol0/brick1
Brick4: stor3data:/mnt/disk_b2/glusterfs/vol0/brick1
Options Reconfigured:
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on

Volume Name: volumedisk1
Type: Distribute
Volume ID: 591b7098-800e-4954-82a9-6b6d81c9e0a2
Status: Started
Snapshot Count: 0
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: stor1data:/mnt/glusterfs/vol1/brick1
Brick2: stor2data:/mnt/glusterfs/vol1/brick1
Brick3: stor3data:/mnt/disk_c/glusterfs/vol1/brick1
Brick4: stor3data:/mnt/disk_d/glusterfs/vol1/brick1
Options Reconfigured:
cluster.min-free-inodes: 6%
performance.cache-size: 4GB
cluster.min-free-disk: 1%
performance.io-thread-count: 16
performance.readdir-ahead: on

[root at stor1 ~]# grep -n "share" /var/lib/glusterd/vols/volumedisk1/*
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol:3:
option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor1data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:
option shared-brick-count 1
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor2data.mnt-glusterfs-vol1-brick1.vol.rpmsave:3:
option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_c-glusterfs-vol1-brick1.vol.rpmsave:3:
option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol:3:
option shared-brick-count 0
/var/lib/glusterd/vols/volumedisk1/volumedisk1.stor3data.mnt-disk_d-glusterfs-vol1-brick1.vol.rpmsave:3:
option shared-brick-count 0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180227/623ee52a/attachment.html>