[Gluster-users] Memory leak in 3.6.*?

[Yannick Perret]

Hello,
some times ago I posted about a memory leak in client process, but it 
was on a very old 32bit machine (both kernel and OS) and I don't found 
evidences about a similar problem on our recent machines.
But I performed more tests and I have the same problem.

Clients are 64bit Debian 8.2 machines. Glusterfs client on these 
machines is compiled from sources with activated stuff:
FUSE client          : yes
Infiniband verbs     : no
epoll IO multiplex   : yes
argp-standalone      : no
fusermount           : yes
readline             : yes
georeplication       : yes
Linux-AIO            : no
Enable Debug         : no
systemtap            : no
Block Device xlator  : no
glupy                : no
Use syslog           : yes
XML output           : yes
QEMU Block formats   : no
Encryption xlator    : yes
Erasure Code xlator  : yes

I tested both 3.6.7 and 3.6.9 version on client (3.6.7 is the one 
installed on our machines, even on servers, 3.6.9 is for testing with 
last 3.6 version).

Here are the operations on the client (also performed with similar 
results with 3.6.7 version):
# /usr/local/sbin/glusterfs --version
glusterfs 3.6.9 built on Jul 22 2016 13:27:42
(…)
# mount -t glusterfs sto1.my.domain:BACKUP-ADMIN-DATA  /zog/
# cd /usr/
# cp -Rp * /zog/TEMP/
Then monitoring memory used by glusterfs process while 'cp' is running 
(resp. VSZ and RSS from 'ps'):
284740 70232
284740 70232
284876 71704
285000 72684
285136 74008
285416 75940
(…)
368684 151980
369324 153768
369836 155576
370092 156192
370092 156192
Here both sizes are stable and correspond to the end of 'cp' command.
If I restart an other 'cp' (even on the same directories) size starts 
again to increase.
If I perform a 'ls -lR' in the directory size also increase:
370756 192488
389964 212148
390948 213232
(here I ^C the 'ls')

When doing nothing the size don't increase but never decrease (calling 
'sync' don't change the situation).
Sending a HUP signal to glusterfs process also increases memory (390948 
213324 → 456484 213320).
Changing volume configuration (changing diagnostics.client-sys-log-level 
value) don't change anything.

Here the actual ps:
root     17041  4.9  5.2 456484 213320 ?       Ssl  13:29   1:21 
/usr/local/sbin/glusterfs --volfile-server=sto1.my.domain 
--volfile-id=BACKUP-ADMIN-DATA /zog

Of course umouting/remounting fall back to "start" size:
# umount /zog
# mount -t glusterfs sto1.my.domain:BACKUP-ADMIN-DATA  /zog/
→ root     28741  0.3  0.7 273320 30484 ?        Ssl  13:57   0:00 
/usr/local/sbin/glusterfs --volfile-server=sto1.my.domain 
--volfile-id=BACKUP-ADMIN-DATA /zog


I didn't saw this before because most of our volumes are mounted "on 
demand" for some storage activities or are permanently mounted but with 
very few activity.
But clearly this memory usage driff is a long-term problem. On the old 
32bit machine I had this problem ("solved" by using NFS mounts in order 
to wait for this old machine to be replaced) and it lead to glusterfs 
being killed by OS when out of free memory. It was faster than what I 
describe here but it's just a question of time.


Thanks for any help about that.

Regards,
--
Y.


The corresponding volume on servers is (if it can help):
Volume Name: BACKUP-ADMIN-DATA
Type: Replicate
Volume ID: 306d57f3-fb30-4bcc-8687-08bf0a3d7878
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: sto1.my.domain:/glusterfs/backup-admin/data
Brick2: sto2.my.domain:/glusterfs/backup-admin/data
Options Reconfigured:
diagnostics.client-sys-log-level: WARNING




-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3369 bytes
Desc: Signature cryptographique S/MIME
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160722/ecba239c/attachment.p7s>