<div dir="ltr">Hi,<div><br></div><div>Can you please specify which process has leak? Have you took the statedump of the same process which has leak? </div><div><br></div><div>Thanks,</div><div>Sanju</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Feb 2, 2019 at 3:15 PM Pedro Costa <pedro@pmc.digital> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-GB">
<div class="gmail-m_-5464086349057227150WordSection1">
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">I have a 3x replicated cluster running 4.1.7 on ubuntu 16.04.5, all 3 replicas are also clients hosting a Node.js/Nginx web server.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">The current configuration is as such:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Volume Name: gvol1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Type: Replicate<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Volume ID: XXXXXX<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Status: Started<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Snapshot Count: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Number of Bricks: 1 x 3 = 3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Transport-type: tcp<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Bricks:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Brick1: vm000000:/srv/brick1/gvol1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Brick2: vm000001:/srv/brick1/gvol1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Brick3: vm000002:/srv/brick1/gvol1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Options Reconfigured:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.self-heal-readdir-size: 2KB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.self-heal-window-size: 2<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.background-self-heal-count: 20<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">network.ping-timeout: 5<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">disperse.eager-lock: off<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.parallel-readdir: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.readdir-ahead: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.rda-cache-limit: 128MB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.cache-refresh-timeout: 10<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.nl-cache-timeout: 600<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.nl-cache: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.nufa: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.enable-least-priority: off<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">server.outstanding-rpc-limit: 128<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.strict-o-direct: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.shd-max-threads: 12<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">client.event-threads: 4<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.lookup-optimize: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">network.inode-lru-limit: 90000<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.md-cache-timeout: 600<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.cache-invalidation: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.cache-samba-metadata: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.stat-prefetch: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">features.cache-invalidation-timeout: 600<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">features.cache-invalidation: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">storage.fips-mode-rchecksum: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">transport.address-family: inet<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">nfs.disable: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.client-io-threads: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">features.utime: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">storage.ctime: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">server.event-threads: 4<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.cache-size: 256MB<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.read-ahead: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.readdir-optimize: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.strict-readdir: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">performance.io-thread-count: 8<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">server.allow-insecure: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.read-hash-mode: 0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.lookup-unhashed: auto<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">cluster.choose-local: on<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">I believe there’s a memory leak somewhere, it just keeps going up until it hangs one or more nodes taking the whole cluster down sometimes.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">I have taken 2 statedumps on one of the nodes, one where the memory is too high and another just after a reboot with the app running and the volume fully healed.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><a href="https://pmcdigital.sharepoint.com/:u:/g/EYDsNqTf1UdEuE6B0ZNVPfIBf_I-AbaqHotB1lJOnxLlTg?e=boYP09" target="_blank">https://pmcdigital.sharepoint.com/:u:/g/EYDsNqTf1UdEuE6B0ZNVPfIBf_I-AbaqHotB1lJOnxLlTg?e=boYP09</a>
(high memory)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><a href="https://pmcdigital.sharepoint.com/:u:/g/EWZBsnET2xBHl6OxO52RCfIBvQ0uIDQ1GKJZ1GrnviyMhg?e=wI3yaY" target="_blank">https://pmcdigital.sharepoint.com/:u:/g/EWZBsnET2xBHl6OxO52RCfIBvQ0uIDQ1GKJZ1GrnviyMhg?e=wI3yaY</a>
(after reboot)<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Any help would be greatly appreciated,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Lato">Kindest Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><b><span style="font-family:Lato;color:rgb(177,24,30)"><u></u> <u></u></span></b></p>
<p class="MsoNormal"><b><span lang="PT" style="font-size:10pt;font-family:Lato;color:rgb(177,24,30)">Pedro Maia Costa
<br>
</span></b><i><span lang="PT" style="font-size:10pt;font-family:Lato">Senior Developer,
<b>pmc.</b>digital<u></u><u></u></span></i></p>
</div>
</div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Thanks,<br></div>Sanju<br></div></div>