<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>Dear community,</p>
<p><br>
</p>
<p>actually, I'm running a 3 Node GlusterFS. Simple Wordpress pages needs 4 -10 seconds to load. Since a month we have also problems with memory leaks. All 3 nodes got 24 GB RAM (before 12 GB RAM) but GlusterFS use all the RAM. If all the RAM is used the virtual
maschine loose there mountpoint. After remount everything starts again and that 2-3 times daily.</p>
<p><br>
</p>
<p># Gluster Version: <span>8.0</span></p>
<p><br>
</p>
<p>#Affected process: This is a snapshot from top where the process starts with low memory usage and run so long RAM is available.<br>
</p>
<p><br>
</p>
<p></p>
<div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br>
869835 root 20 0 20,9g 20,3g 4340 S 2,3 86,5 152:10.62 /usr/sbin/glusterfs --process-name fuse --volfile-server=vm01 --volfile-server=vm02 --volfile-id=/gluster /var/www
<br>
</div>
<p></p>
<p><br>
</p>
<p># gluster volume info</p>
<p></p>
<div>Volume Name: gluster<br>
Type: Replicate<br>
Volume ID: c6d3beb1-b841-45e8-aa64-bb2be1e36e39<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x 3 = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: vm01:/srv/glusterfs<br>
Brick2: vm02:/srv/glusterfs<br>
Brick3: vm03:/srv/glusterfs<br>
Options Reconfigured:<br>
performance.io-cache: on<br>
performance.write-behind: on<br>
performance.flush-behind: on<br>
auth.allow: 10.10.10.*<br>
performance.readdir-ahead: on<br>
performance.quick-read: off<br>
performance.cache-size: 1GB<br>
performance.cache-refresh-timeout: 10<br>
performance.read-ahead: off<br>
performance.write-behind-window-size: 4MB<br>
network.ping-timeout: 2<br>
performance.io-thread-count: 32<br>
performance.cache-max-file-size: 2MB<br>
performance.md-cache-timeout: 60<br>
features.cache-invalidation: on<br>
features.cache-invalidation-timeout: 600<br>
performance.stat-prefetch: on<br>
network.inode-lru-limit: 90000</div>
<p></p>
<p><br>
</p>
<p># Logs</p>
<p>I can't find any critical messages on all gluster logs, but in syslog I found the oom-kill. After that, the mountpoint is history.</p>
<p><br>
</p>
<p></p>
<div> oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/srv-web.mount,task=glusterfs,pid=961,uid=0<br>
[68263.478730] Out of memory: Killed process 961 (glusterfs) total-vm:21832212kB, anon-rss:21271576kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:41792kB oom_score_adj:0<br>
[68264.243608] oom_reaper: reaped process 961 (glusterfs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB<br>
<br>
</div>
And after the remount it starts again to use more and more memory.
<p></p>
<p><br>
</p>
<p>Alternatively I can also activate SWAP but this slow down the load time extremely if GlusterFS starts to use SWAP after all RAM is used.</p>
<p><br>
</p>
<p>If you need more information let me know it and i will send this too.<br>
</p>
<p><br>
</p>
<p>Best regards</p>
<p>Benjamin<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
</div>
</body>
</html>