[Gluster-users] GlusterFS 3.1.2 - df and du irregularities

[Nic von Waltsleben]

Hi Amar

Thanks for clarifying that for me however I don't think this applies in our scenario.
On 04 Mar 2011, at 11:00 AM, Amar Tumballi wrote:
> 
> I'm having a weird issue where my gluster volumes are being reported as full by df but du shows only 18GB of actual data in them (which is less than 1% of the capacity of the volumes).  The client shows 2/3 of the volume is full, only replicate-3 is currently not "full" and given how this problem started I expect it to be full by the time I come into the office tomorrow morning.  I am runnung glusterfs 3.1.2 on Ubuntu 10.10. Does anyone have any idea why this might be 
> happening?
> 
> 
> GlusterFS exports in a thin provision way by design, which causes this behavior. You are seeing this behavior because of the fundamental difference in the implementation of the two commands. (df internally uses 'statvfs(2)', and du uses 'readdir(), stat()' recursively to find out the size under a given directory).
> 
Here is the df output on one of our Gluster hosts, notice that the exported volume is a dedicated filesystem.  

root at gluster01.jnb1:/export/glusterfs# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2               50G  1.6G   46G   4% /
none                  997M  200K  997M   1% /dev
none                 1004M     0 1004M   0% /dev/shm
none                 1004M   72K 1004M   1% /var/run
none                 1004M     0 1004M   0% /var/lock
/dev/md3              1.7T  1.7T     0 100% /export/glusterfs
/dev/md0              230M   26M  193M  12% /boot


> That means the export directory used for the volume resides on a filesystem which is already 72% full. So, when the filesystem is mounted, the 'df' of the filesystem shows 72% full, and there are no files in glusterfs volume.
> 
> Hope that explains. So, to set up glusterfs properly, make sure your backend exports are dedicated to GlusterFS and some other process is not filling it up.
> 
> Regards,
> Amar
> 
> PS: this should be to our FAQ in wiki, so people will be aware of these type of issues due to thin provision factors.
> 
> 

du only finds about 23GB of data on the filesystem.  After rebooting the Gluster server disk usage is reported accurately by df.

nic at gluster01:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md2               50G  1.6G   46G   4% /
none                  997M  200K  997M   1% /dev
none                 1004M     0 1004M   0% /dev/shm
none                 1004M   68K 1004M   1% /var/run
none                 1004M     0 1004M   0% /var/lock
none                   50G  1.6G   46G   4% /var/lib/ureadahead/debugfs
/dev/md0              230M   26M  193M  12% /boot
/dev/md3              1.7T   23G  1.6T   2% /export/glusterfs

Do you have any idea what would cause this to occur?

Regards,

Nic von Waltsleben
nic at clue.co.za