[Gluster-users] Memory leak in 3.6.9

[Alessandro Ipe]

Hi,


Apparently, version 3.6.9 is suffering from a SERIOUS memory leak as illustrated in the following logs:
2016-04-26T11:54:27.971564+00:00 tsunami1 kernel: [698635.210069] glusterfsd invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
2016-04-26T11:54:27.974133+00:00 tsunami1 kernel: [698635.210076] Pid: 28111, comm: glusterfsd Tainted: G        W  O 3.7.10-1.1-desktop #1
2016-04-26T11:54:27.974136+00:00 tsunami1 kernel: [698635.210077] Call Trace:
2016-04-26T11:54:27.974137+00:00 tsunami1 kernel: [698635.210090]  [<ffffffff81004818>] dump_trace+0x88/0x300
2016-04-26T11:54:27.974137+00:00 tsunami1 kernel: [698635.210096]  [<ffffffff8158b033>] dump_stack+0x69/0x6f
2016-04-26T11:54:27.974138+00:00 tsunami1 kernel: [698635.210101]  [<ffffffff8158db39>] dump_header+0x70/0x200
2016-04-26T11:54:27.974139+00:00 tsunami1 kernel: [698635.210105]  [<ffffffff81112ad4>] oom_kill_process+0x244/0x390
2016-04-26T11:54:28.113125+00:00 tsunami1 kernel: [698635.210111]  [<ffffffff81113211>] out_of_memory+0x451/0x490
2016-04-26T11:54:28.113142+00:00 tsunami1 kernel: [698635.210116]  [<ffffffff81118afe>] __alloc_pages_nodemask+0x8ae/0x9f0
2016-04-26T11:54:28.113143+00:00 tsunami1 kernel: [698635.210122]  [<ffffffff81152fb7>] alloc_pages_current+0xb7/0x130
2016-04-26T11:54:28.113144+00:00 tsunami1 kernel: [698635.210127]  [<ffffffff81111673>] filemap_fault+0x283/0x440
2016-04-26T11:54:28.113144+00:00 tsunami1 kernel: [698635.210131]  [<ffffffff811345ee>] __do_fault+0x6e/0x560
2016-04-26T11:54:28.113145+00:00 tsunami1 kernel: [698635.210136]  [<ffffffff81137cf7>] handle_pte_fault+0x97/0x490
2016-04-26T11:54:28.113145+00:00 tsunami1 kernel: [698635.210141]  [<ffffffff8159af8b>] __do_page_fault+0x16b/0x4c0
2016-04-26T11:54:28.113562+00:00 tsunami1 kernel: [698635.210145]  [<ffffffff815982f8>] page_fault+0x28/0x30
2016-04-26T11:54:28.113565+00:00 tsunami1 kernel: [698635.210158]  [<00007fa9d8a8292b>] 0x7fa9d8a8292a
2016-04-26T11:54:28.120811+00:00 tsunami1 kernel: [698635.226243] Out of memory: Kill process 17144 (glusterfsd) score 694 or sacrifice child
2016-04-26T11:54:28.120811+00:00 tsunami1 kernel: [698635.226251] Killed process 17144 (glusterfsd) total-vm:8956384kB, anon-rss:6670900kB, file-rss:0kB

It makes this version completely useless in production. Bricks servers have 8 GB of RAM (but will be upgraded to 16 GB).

gluster volume info <VOLUME> returns:
Volume Name: home
Type: Distributed-Replicate
Volume ID: 501741ed-4146-4022-af0b-41f5b1297766
Status: Started
Number of Bricks: 14 x 2 = 28
Transport-type: tcp
Bricks:
Brick1: tsunami1:/data/glusterfs/home/brick1
Brick2: tsunami2:/data/glusterfs/home/brick1
Brick3: tsunami1:/data/glusterfs/home/brick2
Brick4: tsunami2:/data/glusterfs/home/brick2
Brick5: tsunami1:/data/glusterfs/home/brick3
Brick6: tsunami2:/data/glusterfs/home/brick3
Brick7: tsunami1:/data/glusterfs/home/brick4
Brick8: tsunami2:/data/glusterfs/home/brick4
Brick9: tsunami3:/data/glusterfs/home/brick1
Brick10: tsunami4:/data/glusterfs/home/brick1
Brick11: tsunami3:/data/glusterfs/home/brick2
Brick12: tsunami4:/data/glusterfs/home/brick2
Brick13: tsunami3:/data/glusterfs/home/brick3
Brick14: tsunami4:/data/glusterfs/home/brick3
Brick15: tsunami3:/data/glusterfs/home/brick4
Brick16: tsunami4:/data/glusterfs/home/brick4
Brick17: tsunami5:/data/glusterfs/home/brick1
Brick18: tsunami6:/data/glusterfs/home/brick1
Brick19: tsunami5:/data/glusterfs/home/brick2
Brick20: tsunami6:/data/glusterfs/home/brick2
Brick21: tsunami5:/data/glusterfs/home/brick3
Brick22: tsunami6:/data/glusterfs/home/brick3
Brick23: tsunami5:/data/glusterfs/home/brick4
Brick24: tsunami6:/data/glusterfs/home/brick4
Brick25: tsunami7:/data/glusterfs/home/brick1
Brick26: tsunami8:/data/glusterfs/home/brick1
Brick27: tsunami7:/data/glusterfs/home/brick2
Brick28: tsunami8:/data/glusterfs/home/brick2
Options Reconfigured:
nfs.export-dir: /gerb-reproc/Archive
nfs.volume-access: read-only
cluster.ensure-durability: on
features.quota: on
performance.cache-size: 512MB
performance.io-thread-count: 32
performance.flush-behind: off
performance.write-behind-window-size: 4MB
performance.write-behind: off
nfs.disable: off
cluster.read-hash-mode: 2
diagnostics.brick-log-level: CRITICAL
cluster.lookup-unhashed: on
server.allow-insecure: on
auth.allow: localhost, <COUPLE OF IP ADDRESSES>
cluster.readdir-optimize: on
performance.readdir-ahead: on
nfs.export-volumes: off

Are you aware if this issue ?


Thanks,


A.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160426/6b63856c/attachment.html>