[Bugs] [Bug 1542601] New: The used space in the volume increases when the volume is expanded

bugzilla at redhat.com bugzilla at redhat.com
Tue Feb 6 16:05:38 UTC 2018


https://bugzilla.redhat.com/show_bug.cgi?id=1542601

            Bug ID: 1542601
           Summary: The used space in the volume increases when the volume
                    is expanded
           Product: GlusterFS
           Version: 3.12
         Component: distribute
          Assignee: bugs at gluster.org
          Reporter: nbalacha at redhat.com
                CC: bugs at gluster.org, hchiramm at redhat.com,
                    madam at redhat.com, mliyazud at redhat.com,
                    nbalacha at redhat.com, pprakash at redhat.com,
                    rgeorge at redhat.com, rhs-bugs at redhat.com,
                    rtalur at redhat.com, sankarshan at redhat.com,
                    storage-qa-internal at redhat.com, tdesala at redhat.com
        Depends On: 1540961, 1541916
            Blocks: 1535332



+++ This bug was initially created as a clone of Bug #1541916 +++

+++ This bug was initially created as a clone of Bug #1540961 +++

+++ This bug was initially created as a clone of Bug #1535332 +++

Description of problem:

A volume of size 40GB was created and a file of 39GB was created on it. The
volume was expanded by 30GB and a rebalance was performed. The total size of
the volume became 70GB. But the used space on the volume increased from 39GB to
61.5GB

Version-Release number of selected component (if applicable):

cns-deploy-5.0.0-57.el7rhgs.x86_64
heketi-client-5.0.0-19.el7rhgs.x86_64

How reproducible:


Expected results:
The used space should stay the same i.e., 39GB

Additional info:



** gluster volume info after expansion:
sh-4.2# gluster vol info vol_14a05e3c1f728afdebc484c15f7c6cad                  

Volume Name: vol_14a05e3c1f728afdebc484c15f7c6cad                              
Type: Distributed-Replicate            
Volume ID: 15550f1f-743b-494d-b761-81d03c8bb24f                                
Status: Started                        
Snapshot Count: 0                      
Number of Bricks: 2 x 3 = 6            
Transport-type: tcp                    
Bricks:                                
Brick1:
server1:/var/lib/heketi/mounts/vg_02b16cdadce2aaf1356c37599dadf3f4/brick_a99f4919f14c0ab20b667a0269adeff5/brick 
Brick2:
server2:/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_1b0082ad01b71b87e16686c5667cab6e/brick 
Brick3:
server3:/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_35a93294d10a783faf85aca0d8ced048/brick 
Brick4:
server3:/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_7542d4af880f8f2b501180e0a2e1f117/brick 
Brick5:
server1:/var/lib/heketi/mounts/vg_bf6b608031932669bb05cc8430948619/brick_490a6de2e0e5b67e7c9495b3d57bb9e0/brick 
Brick6:
server2:/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_7c97d7dc6c2f0a3d77bffd3e2c95bd7e/brick 
Options Reconfigured:                  
nfs.disable: on                        
transport.address-family: inet         
cluster.brick-multiplex: on      



** ls on the client after expansion:
# ls -lash /usr/share/busybox/
total 40894472                          
     4 drwxrwsr-x    4 root     2000        4.0K Jan 16 14:25 .                 
     0 drwxr-xr-x    3 root     root          21 Jan 18 07:12 ..                
     4 drwxr-sr-x    3 root     2000        4.0K Jan 16 14:25 .trashcan         
40894464 -rw-r--r--    1 root     2000       39.0G Jan 16 14:23 39gfile         

** ls on the bricks after expansion
* node 1
sh-4.2# ls -lash
/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_7c97d7dc6c2f0a3d77bffd3e2c95bd7e/brick/ 
total 23G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:25 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:25 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
23G ---------T. 2 root 2000   0 Jan 16 14:25 39gfile                           
sh-4.2# ls -lash
/var/lib/heketi/mounts/vg_cec08acd1c8b4e8aa15c61a97d89181c/brick_1b0082ad01b71b87e16686c5667cab6e/brick/ 
total 40G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:23 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:14 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile    

* node 2
sh-4.2# ls -lash
/var/lib/heketi/mounts/vg_02b16cdadce2aaf1356c37599dadf3f4/brick_a99f4919f14c0ab20b667a0269adeff5/brick/ 
total 40G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:23 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:14 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile                           
sh-4.2# ls -lash
/var/lib/heketi/mounts/vg_bf6b608031932669bb05cc8430948619/brick_490a6de2e0e5b67e7c9495b3d57bb9e0/brick/ 
total 23G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:25 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:25 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
23G ---------T. 2 root 2000   0 Jan 16 14:25 39gfile         

* node 3
sh-4.2# ls -lash
/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_35a93294d10a783faf85aca0d8ced048/brick/ 
total 40G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:23 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:14 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:23 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
40G -rw-r--r--. 2 root 2000 39G Jan 16 14:23 39gfile                           
sh-4.2# ls -lash
/var/lib/heketi/mounts/vg_159d40a7085c9372b8c33572d14eba7b/brick_7542d4af880f8f2b501180e0a2e1f117/brick/ 
total 23G                              
  0 drwxrwsr-x. 4 root 2000  56 Jan 16 14:25 .                                 
  0 drwxr-xr-x. 3 root root  19 Jan 16 14:25 ..                                
  0 drw---S---. 9 root 2000 184 Jan 16 14:26 .glusterfs                        
  0 drwxr-sr-x. 3 root 2000  25 Jan 16 14:25 .trashcan                         
23G ---------T. 2 root 2000   0 Jan 16 14:25 39gfile

--- Additional comment from Raghavendra Talur on 2018-01-18 13:21:19 IST ---

df output seems to showing sum of space occupied by real file and link to file 

10.70.46.153:vol_14a05e3c1f728afdebc484c15f7c6cad
                         70.0G     61.5G      8.4G  88% /usr/share/busybox

--- Additional comment from Nithya Balachandran on 2018-01-22 10:56:42 IST ---

[root at dhcp46-2 ~]# oc rsh glusterfs-j7wkd

sh-4.2# grep 39gfile *rebalance.log
vol_14a05e3c1f728afdebc484c15f7c6cad-rebalance.log:[2018-01-16 14:25:57.154717]
I [dht-rebalance.c:1570:dht_migrate_file]
0-vol_14a05e3c1f728afdebc484c15f7c6cad-dht: /39gfile: attempting to move from
vol_14a05e3c1f728afdebc484c15f7c6cad-replicate-0 to
vol_14a05e3c1f728afdebc484c15f7c6cad-replicate-1
vol_14a05e3c1f728afdebc484c15f7c6cad-rebalance.log:[2018-01-16 14:25:57.221248]
E [MSGID: 109023] [dht-rebalance.c:847:__dht_rebalance_create_dst_file]
0-vol_14a05e3c1f728afdebc484c15f7c6cad-dht: fallocate failed for /39gfile on
vol_14a05e3c1f728afdebc484c15f7c6cad-replicate-1 (No space left on device)
vol_14a05e3c1f728afdebc484c15f7c6cad-rebalance.log:[2018-01-16 14:25:57.221456]
E [MSGID: 0] [dht-rebalance.c:1688:dht_migrate_file]
0-vol_14a05e3c1f728afdebc484c15f7c6cad-dht: Create dst failed on -
vol_14a05e3c1f728afdebc484c15f7c6cad-replicate-1 for file - /39gfile
vol_14a05e3c1f728afdebc484c15f7c6cad-rebalance.log:[2018-01-16 14:25:57.230524]
E [MSGID: 109023] [dht-rebalance.c:2757:gf_defrag_migrate_single_file]
0-vol_14a05e3c1f728afdebc484c15f7c6cad-dht: migrate-data failed for /39gfile


Need to check if the fallocate call which creates the dst file does not clean
up the allocated space on failure.


--- Additional comment from Nithya Balachandran on 2018-02-01 14:53:03 IST ---

This is probably the root cause (found by vbellur): 

https://stackoverflow.com/questions/32799252/why-fallocate-on-linux-creates-a-non-empty-file-when-it-has-not-enough-space

--- Additional comment from Worker Ant on 2018-02-05 03:13:50 EST ---

REVIEW: https://review.gluster.org/19495 (cluster/dht: Cleanup on fallocate
failure) posted (#1) for review on master by N Balachandran


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1535332
[Bug 1535332] The used space in the volume increases when the volume is
expanded
https://bugzilla.redhat.com/show_bug.cgi?id=1540961
[Bug 1540961] The used space in the volume increases when the volume is
expanded
https://bugzilla.redhat.com/show_bug.cgi?id=1541916
[Bug 1541916] The used space in the volume increases when the volume is
expanded
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list