[Gluster-users] possible memory leak in client/fuse mount

Thu Nov 19 16:58:57 UTC 2020

Dear Gluster Users,

I've a glusterfs process which consumes about all memory of the machine
(~58GB);

# ps -faxu|grep 17317
root     17317  3.1 88.9 59695516 58479708 ?   Ssl  Oct31 839:36
/usr/sbin/glusterfs --process-name fuse --volfile-server=10.201.0.1
--volfile-server=10.201.0.8:10.201.0.5:10.201.0.6:10.201.0.7:10.201.0.9
--volfile-id=/docker2 /mnt/docker2

The gluster version on this machine is 7.8, but i'm currently running a
mixed cluster of 6.10 and 7.8, while awaiting to proceed to upgrade for the
issue mentioned earlier with the self-heal daemon.

The affected volume info looks like;

# gluster v info docker2

Volume Name: docker2
Type: Distributed-Replicate
Volume ID: 4e0670a0-3d00-4360-98bd-3da844cedae5
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x (2 + 1) = 9
Transport-type: tcp
Bricks:
Brick1: 10.201.0.5:/data0/gfs/bricks/brick1/docker2
Brick2: 10.201.0.9:/data0/gfs/bricks/brick1/docker2
Brick3: 10.201.0.3:/data0/gfs/bricks/bricka/docker2 (arbiter)
Brick4: 10.201.0.6:/data0/gfs/bricks/brick1/docker2
Brick5: 10.201.0.7:/data0/gfs/bricks/brick1/docker2
Brick6: 10.201.0.4:/data0/gfs/bricks/bricka/docker2 (arbiter)
Brick7: 10.201.0.1:/data0/gfs/bricks/brick1/docker2
Brick8: 10.201.0.8:/data0/gfs/bricks/brick1/docker2
Brick9: 10.201.0.2:/data0/gfs/bricks/bricka/docker2 (arbiter)
Options Reconfigured:
performance.cache-size: 128MB
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: on

The issue seems to be triggered by a program called zammad, which has an
init process, which runs in a loop. on cycle it re-compiles the
ruby-on-rails application.

I've attached 2 statedumps, but as i only recently noticed the high memory
usage, i believe both statedumps already show an escalated state of the
glusterfs process. If it's needed to also have them from the beginning let
me know. The dumps are taken about an hour apart.
Also i've included the glusterd.log. I couldn't include mnt-docker2.log
since it's too large, since it's littered with: " I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht"
However i've inspected the log and it contains no Error message's all are
of the Info kind;
which look like these;
[2020-11-19 03:29:05.406766] I [glusterfsd-mgmt.c:2282:mgmt_getspec_cbk]
0-glusterfs: No change in volfile,continuing
[2020-11-19 03:29:21.271886] I [socket.c:865:__socket_shutdown]
0-docker2-client-8: intentional socket shutdown(5)
[2020-11-19 03:29:24.479738] I [socket.c:865:__socket_shutdown]
0-docker2-client-2: intentional socket shutdown(5)
[2020-11-19 03:30:12.318146] I [socket.c:865:__socket_shutdown]
0-docker2-client-5: intentional socket shutdown(5)
[2020-11-19 03:31:27.381720] I [socket.c:865:__socket_shutdown]
0-docker2-client-8: intentional socket shutdown(5)
[2020-11-19 03:31:30.579630] I [socket.c:865:__socket_shutdown]
0-docker2-client-2: intentional socket shutdown(5)
[2020-11-19 03:32:18.427364] I [socket.c:865:__socket_shutdown]
0-docker2-client-5: intentional socket shutdown(5)

The rename messages look like these;
[2020-11-19 03:29:05.402663] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/95/75f93c20e375c5.tmp.eVcE5D
(fe083b7e-b0d5-485c-8666-e1f7cdac33e2)
(hash=docker2-replicate-2/cache=docker2-replicate-2) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/95/75f93c20e375c5
((null)) (hash=docker2-replicate-2/cache=<nul>)
[2020-11-19 03:29:05.410972] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/0d/86dd25f3d238ff.tmp.AdDTLu
(b1edadad-1d48-4bf4-be85-ffbe9d69d338)
(hash=docker2-replicate-1/cache=docker2-replicate-1) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/0d/86dd25f3d238ff
((null)) (hash=docker2-replicate-2/cache=<nul>)
[2020-11-19 03:29:05.420064] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/f2/6e44f76b508fd3.tmp.QKmxul
(31f80fcb-977c-433b-9259-5fdfcad1171c)
(hash=docker2-replicate-0/cache=docker2-replicate-0) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/f2/6e44f76b508fd3
((null)) (hash=docker2-replicate-0/cache=<nul>)
[2020-11-19 03:29:05.427537] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/b0/1d7303d9dfe009.tmp.qLUMec
(e2fdf971-731f-4765-80e8-3165433488ea)
(hash=docker2-replicate-2/cache=docker2-replicate-2) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/b0/1d7303d9dfe009
((null)) (hash=docker2-replicate-1/cache=<nul>)
[2020-11-19 03:29:05.440576] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/bd/952a089e164b36.tmp.4qvl22
(3e0bc6d1-13ac-47c6-b221-1256b4b506ef)
(hash=docker2-replicate-2/cache=docker2-replicate-2) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/bd/952a089e164b36
((null)) (hash=docker2-replicate-1/cache=<nul>)
[2020-11-19 03:29:05.452407] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/a3/b587dd08f35e2e.tmp.iIweTT
(9685b5f3-4b14-4050-9b00-1163856239b5)
(hash=docker2-replicate-1/cache=docker2-replicate-1) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/a3/b587dd08f35e2e
((null)) (hash=docker2-replicate-0/cache=<nul>)
[2020-11-19 03:29:05.460720] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/48/89cfb1b971c025.tmp.0W7jMK
(d0a8d0a4-c783-45db-bb4a-68e24044d830)
(hash=docker2-replicate-0/cache=docker2-replicate-0) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/48/89cfb1b971c025
((null)) (hash=docker2-replicate-1/cache=<nul>)
[2020-11-19 03:29:05.468800] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/d9/759d55e8da66eb.tmp.2yXtHB
(e5b61ef5-a3c2-4a2c-aa47-c377a6c090d7)
(hash=docker2-replicate-0/cache=docker2-replicate-0) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/d9/759d55e8da66eb
((null)) (hash=docker2-replicate-0/cache=<nul>)
[2020-11-19 03:29:05.476745] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/1c/f3a658342e36b7.tmp.gSkiEs
(17181a40-f9b2-438f-9dfc-7bb159c516e6)
(hash=docker2-replicate-2/cache=docker2-replicate-2) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/1c/f3a658342e36b7
((null)) (hash=docker2-replicate-0/cache=<nul>)
[2020-11-19 03:29:05.486729] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/f1/6bef7cb6446c7a.tmp.sVT0Dj
(cb6b1d52-b1c0-420c-86b7-2ceb8e8e73db)
(hash=docker2-replicate-0/cache=docker2-replicate-0) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/f1/6bef7cb6446c7a
((null)) (hash=docker2-replicate-1/cache=<nul>)
[2020-11-19 03:29:05.495115] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/45/73ba226559961b.tmp.QdPTFa
(d8450d9e-62a7-4fd5-9dd2-e072e318d9a0)
(hash=docker2-replicate-0/cache=docker2-replicate-0) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/45/73ba226559961b
((null)) (hash=docker2-replicate-1/cache=<nul>)
[2020-11-19 03:29:05.503424] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/13/29c0df35961ca0.tmp.s1xUJ1
(ffc57a77-8b91-4264-8e2d-a9966f0f37ef)
(hash=docker2-replicate-1/cache=docker2-replicate-1) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/13/29c0df35961ca0
((null)) (hash=docker2-replicate-2/cache=<nul>)
[2020-11-19 03:29:05.513532] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/be/8d6a07b6a0d6ad.tmp.A5DzQS
(5a595a65-372d-4377-b547-2c4e23f7be3a)
(hash=docker2-replicate-1/cache=docker2-replicate-1) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/be/8d6a07b6a0d6ad
((null)) (hash=docker2-replicate-0/cache=<nul>)
[2020-11-19 03:29:05.526885] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/ec/4208216d993cbe.tmp.IMXg0J
(2fa99fcd-64f8-4934-aeda-b356816f1132)
(hash=docker2-replicate-2/cache=docker2-replicate-2) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/ec/4208216d993cbe
((null)) (hash=docker2-replicate-2/cache=<nul>)
[2020-11-19 03:29:05.537637] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/57/1527c482cf2d6b.tmp.Y2L0cB
(db24d7bf-4a06-4356-a52e-1ab9537d1c3a)
(hash=docker2-replicate-0/cache=docker2-replicate-0) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/57/1527c482cf2d6b
((null)) (hash=docker2-replicate-1/cache=<nul>)
[2020-11-19 03:29:05.547878] I [MSGID: 109066]
[dht-rename.c:1951:dht_rename] 0-docker2-dht: renaming
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/88/1b60ead8d4c4e5.tmp.u47rss
(b12f041b-5bbd-4e3d-b700-8f673830393f)
(hash=docker2-replicate-1/cache=docker2-replicate-1) =>
/corporate/zammad/tmp/init/cache/bootsnap-compile-cache/88/1b60ead8d4c4e5
((null)) (hash=docker2-replicate-1/cache=<nul>)

if i can provide any more information please let me know.

Thanks Olaf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20201119/d7413c6b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gluster_memleak.7z
Type: application/octet-stream
Size: 1087016 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20201119/d7413c6b/attachment-0001.obj>