<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Lato;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:Lato;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">I have a 3x replicated cluster running 4.1.7 on ubuntu 16.04.5, all 3 replicas are also clients hosting a Node.js/Nginx web server.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">The current configuration is as such:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Volume Name: gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Type: Replicate<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Volume ID: XXXXXX<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Status: Started<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Snapshot Count: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Number of Bricks: 1 x 3 = 3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Transport-type: tcp<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Bricks:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Brick1: vm000000:/srv/brick1/gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Brick2: vm000001:/srv/brick1/gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Brick3: vm000002:/srv/brick1/gvol1<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Options Reconfigured:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.self-heal-readdir-size: 2KB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.self-heal-window-size: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.background-self-heal-count: 20<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">network.ping-timeout: 5<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">disperse.eager-lock: off<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.parallel-readdir: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.readdir-ahead: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.rda-cache-limit: 128MB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.cache-refresh-timeout: 10<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.nl-cache-timeout: 600<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.nl-cache: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.nufa: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.enable-least-priority: off<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">server.outstanding-rpc-limit: 128<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.strict-o-direct: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.shd-max-threads: 12<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">client.event-threads: 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.lookup-optimize: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">network.inode-lru-limit: 90000<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.md-cache-timeout: 600<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.cache-invalidation: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.cache-samba-metadata: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.stat-prefetch: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">features.cache-invalidation-timeout: 600<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">features.cache-invalidation: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">storage.fips-mode-rchecksum: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">transport.address-family: inet<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">nfs.disable: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.client-io-threads: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">features.utime: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">storage.ctime: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">server.event-threads: 4<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.cache-size: 256MB<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.read-ahead: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.readdir-optimize: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.strict-readdir: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">performance.io-thread-count: 8<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">server.allow-insecure: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.read-hash-mode: 0<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.lookup-unhashed: auto<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">cluster.choose-local: on<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">I believe there’s a memory leak somewhere, it just keeps going up until it hangs one or more nodes taking the whole cluster down sometimes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">I have taken 2 statedumps on one of the nodes, one where the memory is too high and another just after a reboot with the app running and the volume fully healed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><a href="https://pmcdigital.sharepoint.com/:u:/g/EYDsNqTf1UdEuE6B0ZNVPfIBf_I-AbaqHotB1lJOnxLlTg?e=boYP09">https://pmcdigital.sharepoint.com/:u:/g/EYDsNqTf1UdEuE6B0ZNVPfIBf_I-AbaqHotB1lJOnxLlTg?e=boYP09</a>
(high memory)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><a href="https://pmcdigital.sharepoint.com/:u:/g/EWZBsnET2xBHl6OxO52RCfIBvQ0uIDQ1GKJZ1GrnviyMhg?e=wI3yaY">https://pmcdigital.sharepoint.com/:u:/g/EWZBsnET2xBHl6OxO52RCfIBvQ0uIDQ1GKJZ1GrnviyMhg?e=wI3yaY</a>
(after reboot)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato">Any help would be greatly appreciated,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato;mso-fareast-language:EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Lato;mso-fareast-language:EN-GB">Kindest Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-family:Lato;color:#B1181E;mso-fareast-language:EN-GB"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span lang="PT" style="font-size:10.0pt;font-family:Lato;color:#B1181E;mso-fareast-language:EN-GB">Pedro Maia Costa
<br>
</span></b><i><span lang="PT" style="font-size:10.0pt;font-family:Lato;mso-fareast-language:EN-GB">Senior Developer,
<b>pmc.</b>digital<o:p></o:p></span></i></p>
</div>
</body>
</html>