[Gluster-users] Memory leak in 3.6.*?

[Mykola Ulianytskyi]

Hi

>  3.7 clients are not compatible with 3.6 servers

Can you provide more info?

I use some 3.7 clients with 3.6 servers and don't see issues.

Thank you

--
With best regards,
Mykola


On Fri, Jul 22, 2016 at 4:31 PM, Yannick Perret
<yannick.perret at liris.cnrs.fr> wrote:
> Note: I'm have a dev client machine so I can perform tests or recompile
> glusterfs client if it can helps getting data about that.
>
> I did not test this problem against 3.7.x version as my 2 servers are in use
> and I can't upgrade them at this time, and 3.7 clients are not compatible
> with 3.6 servers (as far as I can see from my tests).
>
> --
> Y.
>
>
> Le 22/07/2016 14:06, Yannick Perret a écrit :
>
> Hello,
> some times ago I posted about a memory leak in client process, but it was on
> a very old 32bit machine (both kernel and OS) and I don't found evidences
> about a similar problem on our recent machines.
> But I performed more tests and I have the same problem.
>
> Clients are 64bit Debian 8.2 machines. Glusterfs client on these machines is
> compiled from sources with activated stuff:
> FUSE client          : yes
> Infiniband verbs     : no
> epoll IO multiplex   : yes
> argp-standalone      : no
> fusermount           : yes
> readline             : yes
> georeplication       : yes
> Linux-AIO            : no
> Enable Debug         : no
> systemtap            : no
> Block Device xlator  : no
> glupy                : no
> Use syslog           : yes
> XML output           : yes
> QEMU Block formats   : no
> Encryption xlator    : yes
> Erasure Code xlator  : yes
>
> I tested both 3.6.7 and 3.6.9 version on client (3.6.7 is the one installed
> on our machines, even on servers, 3.6.9 is for testing with last 3.6
> version).
>
> Here are the operations on the client (also performed with similar results
> with 3.6.7 version):
> # /usr/local/sbin/glusterfs --version
> glusterfs 3.6.9 built on Jul 22 2016 13:27:42
> (…)
> # mount -t glusterfs sto1.my.domain:BACKUP-ADMIN-DATA  /zog/
> # cd /usr/
> # cp -Rp * /zog/TEMP/
> Then monitoring memory used by glusterfs process while 'cp' is running
> (resp. VSZ and RSS from 'ps'):
> 284740 70232
> 284740 70232
> 284876 71704
> 285000 72684
> 285136 74008
> 285416 75940
> (…)
> 368684 151980
> 369324 153768
> 369836 155576
> 370092 156192
> 370092 156192
> Here both sizes are stable and correspond to the end of 'cp' command.
> If I restart an other 'cp' (even on the same directories) size starts again
> to increase.
> If I perform a 'ls -lR' in the directory size also increase:
> 370756 192488
> 389964 212148
> 390948 213232
> (here I ^C the 'ls')
>
> When doing nothing the size don't increase but never decrease (calling
> 'sync' don't change the situation).
> Sending a HUP signal to glusterfs process also increases memory (390948
> 213324 → 456484 213320).
> Changing volume configuration (changing diagnostics.client-sys-log-level
> value) don't change anything.
>
> Here the actual ps:
> root     17041  4.9  5.2 456484 213320 ?       Ssl  13:29   1:21
> /usr/local/sbin/glusterfs --volfile-server=sto1.my.domain
> --volfile-id=BACKUP-ADMIN-DATA /zog
>
> Of course umouting/remounting fall back to "start" size:
> # umount /zog
> # mount -t glusterfs sto1.my.domain:BACKUP-ADMIN-DATA  /zog/
> → root     28741  0.3  0.7 273320 30484 ?        Ssl  13:57   0:00
> /usr/local/sbin/glusterfs --volfile-server=sto1.my.domain
> --volfile-id=BACKUP-ADMIN-DATA /zog
>
>
> I didn't saw this before because most of our volumes are mounted "on demand"
> for some storage activities or are permanently mounted but with very few
> activity.
> But clearly this memory usage driff is a long-term problem. On the old 32bit
> machine I had this problem ("solved" by using NFS mounts in order to wait
> for this old machine to be replaced) and it lead to glusterfs being killed
> by OS when out of free memory. It was faster than what I describe here but
> it's just a question of time.
>
>
> Thanks for any help about that.
>
> Regards,
> --
> Y.
>
>
> The corresponding volume on servers is (if it can help):
> Volume Name: BACKUP-ADMIN-DATA
> Type: Replicate
> Volume ID: 306d57f3-fb30-4bcc-8687-08bf0a3d7878
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: sto1.my.domain:/glusterfs/backup-admin/data
> Brick2: sto2.my.domain:/glusterfs/backup-admin/data
> Options Reconfigured:
> diagnostics.client-sys-log-level: WARNING
>
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users