[Gluster-users] memory leak in 3.3.1 rebalance?

Pierre-Francois Laquerre pierre.francois at nec-labs.com
Tue Mar 5 21:33:43 UTC 2013

I started rebalancing my 25x2 distributed-replicate volume two days ago. 
Since then, the memory usage of the rebalance processes has been 
steadily climbing by 1-2 megabytes per minute. Following 
I tried "echo 2 > /proc/sys/vm/drop_caches". This had no effect on the 
processes' memory usage. Some of the servers are already eating >10G of 
memory. At this rate, I will have to cancel this rebalance, even though 
brick usage is heavily skewed right now (most bricks are at 87% 
capacity, but recently added ones are at 18-28%).

Any ideas what might be causing this? The only references to rebalance 
memory leaks I could find were related to 3.2.x, not 3.3.1.

gluster volume info:
Volume Name: bigdata
Type: Distributed-Replicate
Volume ID: 56498956-7b4b-4ee3-9d2b-4c8cfce26051
Status: Started
Number of Bricks: 25 x 2 = 50
Transport-type: tcp
Brick1: ml43:/mnt/donottouch/localb
Brick2: ml44:/mnt/donottouch/localb
Brick3: ml43:/mnt/donottouch/localc
Brick4: ml44:/mnt/donottouch/localc
Brick5: ml45:/mnt/donottouch/localb
Brick6: ml46:/mnt/donottouch/localb
Brick7: ml45:/mnt/donottouch/localc
Brick8: ml46:/mnt/donottouch/localc
Brick9: ml47:/mnt/donottouch/localb
Brick10: ml48:/mnt/donottouch/localb
Brick11: ml47:/mnt/donottouch/localc
Brick12: ml48:/mnt/donottouch/localc
Brick13: ml45:/mnt/donottouch/locald
Brick14: ml46:/mnt/donottouch/locald
Brick15: ml47:/mnt/donottouch/locald
Brick16: ml48:/mnt/donottouch/locald
Brick17: ml51:/mnt/donottouch/localb
Brick18: ml52:/mnt/donottouch/localb
Brick19: ml51:/mnt/donottouch/localc
Brick20: ml52:/mnt/donottouch/localc
Brick21: ml51:/mnt/donottouch/locald
Brick22: ml52:/mnt/donottouch/locald
Brick23: ml59:/mnt/donottouch/locald
Brick24: ml54:/mnt/donottouch/locald
Brick25: ml59:/mnt/donottouch/localc
Brick26: ml54:/mnt/donottouch/localc
Brick27: ml59:/mnt/donottouch/localb
Brick28: ml54:/mnt/donottouch/localb
Brick29: ml55:/mnt/donottouch/localb
Brick30: ml29:/mnt/donottouch/localb
Brick31: ml55:/mnt/donottouch/localc
Brick32: ml29:/mnt/donottouch/localc
Brick33: ml30:/mnt/donottouch/localc
Brick34: ml31:/mnt/donottouch/localc
Brick35: ml30:/mnt/donottouch/localb
Brick36: ml31:/mnt/donottouch/localb
Brick37: ml40:/mnt/donottouch/localb
Brick38: ml41:/mnt/donottouch/localb
Brick39: ml40:/mnt/donottouch/localc
Brick40: ml41:/mnt/donottouch/localc
Brick41: ml56:/mnt/donottouch/localb
Brick42: ml57:/mnt/donottouch/localb
Brick43: ml56:/mnt/donottouch/localc
Brick44: ml57:/mnt/donottouch/localc
Brick45: ml25:/mnt/donottouch/localb
Brick46: ml26:/mnt/donottouch/localb
Brick47: ml01:/mnt/donottouch/localb/brick
Brick48: ml25:/mnt/donottouch/localc/brick
Brick49: ml01:/mnt/donottouch/localc/brick
Brick50: ml26:/mnt/donottouch/localc/brick
Options Reconfigured:
nfs.register-with-portmap: OFF
nfs.disable: on
performance.quick-read: on

The majority of these bricks are ext4, except for the latest 7 which are 
xfs. Each is backed by a 2T hard drive.

gluster --version:
glusterfs 3.3.1 built on Oct 11 2012 21:49:37

cat /etc/system-release:
Scientific Linux release 6.1 (Carbon)

uname -a:
Linux ml01 2.6.32-131.17.1.el6.x86_64 #1 SMP Wed Oct 5 17:19:54 CDT 2011 
x86_64 x86_64 x86_64 GNU/Linux

df -i /mnt/bigdata:
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
ml43:/bigdata        3272292992 114236317 3158056675    4% /mnt/bigdata

df /mnt/bigdata:
Filesystem           1K-blocks      Used Available Use% Mounted on
ml43:/bigdata        48160570880 33787913600 12223793792  74% /mnt/bigdata

the process I am referring to:

/usr/sbin/glusterfs -s localhost --volfile-id bigdata --xlator-option 
*dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes 
--xlator-option *dht.assert-no-child-down=yes --xlator-option 
*replicate*.data-self-heal=off --xlator-option 
*replicate*.metadata-self-heal=off --xlator-option 
*replicate*.entry-self-heal=off --xlator-option *dht.rebalance-cmd=1 
--xlator-option *dht.node-uuid=5c338e03-28ff-429b-b702-0a04e25565f8 
-l /var/log/glusterfs/bigdata-rebalance.log

I am seeing a lot of:
[2013-03-05 10:59:51.170051] I [dht-rebalance.c:647:dht_migrate_file] 
0-bigdata-dht: /a/c/lu/lu-nod/_4.nrm: attempting to move from 
bigdata-replicate-23 to bigdata-replicate-17
[2013-03-05 10:59:51.296487] W 
[dht-rebalance.c:361:__dht_check_free_space] 0-bigdata-dht: data 
movement attempted from node (bigdata-replicate-23) with higher disk 
space to a node (bigdata-replicate-17) with lesser disk space 
[2013-03-05 10:59:51.296604] E 
[dht-rebalance.c:1202:gf_defrag_migrate_data] 0-bigdata-dht: 
migrate-data failed for /a/c/lu/lu-nod/_4.nrm

in the rebalance log, and a lot of:

[2013-03-05 11:00:02.609033] I [server3_1-fops.c:576:server_mknod_cbk] 
0-bigdata-server: 586514: MKNOD (null) (--) ==> -1 (File exists)


[2013-03-05 10:59:58.755505] E [posix-helpers.c:701:posix_handle_pair] 
0-bigdata-posix: /mnt/donottouch/localb/brick/b/somedir/foobar: 
key:trusted.glusterfs.dht.linkto error:File exists
[2013-03-05 10:59:58.755576] E [posix.c:1755:posix_create] 
0-bigdata-posix: setting xattrs on 
/mnt/donottouch/localb/brick/b/somedir/foobar failed (File exists)

in the brick logs.

gluster volume rebalance bigdata status:
                                     Node, Rebalanced-files, size, 
scanned, failures, status
                                     ml01              296 279.1MB       
6761635         29006    in progress
                                     ml25             1208 1.1GB       
6167501         75131    in progress
                                     ml26                0 0Bytes       
7197961             0    in progress
                                     ml29                0 0Bytes       
5892140             0    in progress
                                     ml30           118486 74.3GB       
5867068        107288    in progress
                                     ml31                0 0Bytes       
6802084             0    in progress
                                     ml40            55652 41.2GB       
6100163         94717    in progress
                                     ml41                0 0Bytes       
6151999             0    in progress
                                     ml43           124390 53.3GB       
5470703        101802    in progress
                                     ml44                0 0Bytes       
6789294             0    in progress
                                     ml45           146811 61.1GB       
4829884        242848    in progress
                                     ml46                0 0Bytes       
7184315             0    in progress
                                     ml47            82719 67.5GB       
5954615        153226    in progress
                                     ml48                0 0Bytes       
7826333             0    in progress
                                     ml51           156903 76.2GB       
4681106        146970    in progress
                                     ml52                0 0Bytes       
6154015             0    in progress
                                     ml54                0 
0Bytes         42773             0    in progress
                                     ml55            44194 33.4GB       
6113086         82874    in progress
                                     ml56            16060 16.5GB       
5949356        136277    in progress
                                     ml57                0 0Bytes       
7845347             0    in progress
                                     ml59           128070 76.7GB       
5847966        145739    in progress

More information about the Gluster-users mailing list