<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="text-align:left; direction:ltr;">
<div>All,</div>
<div><br>
</div>
<div>We're seeing some issues with the default provided logrotate configuration in regards to the bitd.log files.</div>
<div>Logrotate has a postrotate-script to run "killall -HUP glusterfs", to make the processes release the filehandles and create a new logfile, and using "delaycompress".</div>
<div><br>
</div>
<div>Recently we noticed that the 'df' reported usage on our /var/log didn't match "actual" usage reported with 'du'.</div>
<div>Checking 'lsof' we found that basically all "bitd.log.1" files are listed as open but "deleted", when lograte did the compression.</div>
<div><br>
</div>
<div>This only applies to the bitrot-daemon logs, none of the other logs.</div>
<div><br>
</div>
<div>In addition to this we are also seeing that the bitd.log file is significantly larger on the "second" replica-node in the cluster (the "first" node is the one used in fstab on the clients).</div>
<div><br>
</div>
<div>Please note, we are currently running a two-node replica set, we have a plan to introduce an arbiter-node, but need to complete some internal testing, as one of the volumes currently contain over 20 million files, and we are unsure how the introduction
of the arbiter will impact the volume.</div>
<div><br>
</div>
<div>We are running glusterfs-4.1.5-1.el7.x86_64</div>
<div><br>
</div>
<div><br>
</div>
<div>'lsof' output from "first" node</div>
<div>glusterfs 12698 root 5w REG 253,11 611193834 50333986 /var/log/glusterfs/bitd.log.1 (deleted)</div>
<div>glusterfs 12698 root 8w REG 253,11 611193834 50333986 /var/log/glusterfs/bitd.log.1 (deleted)</div>
<div>glusterfs 12698 root 12w REG 253,11 611193834 50333986 /var/log/glusterfs/bitd.log.1 (deleted)</div>
<div><br>
</div>
<div>'lsof' output from "second" node</div>
<div>glusterfs 12742 root 5w REG 253,11 12959954668 50351288 /var/log/glusterfs/bitd.log.1 (deleted)</div>
<div>glusterfs 12742 root 8w REG 253,11 12959954668 50351288 /var/log/glusterfs/bitd.log.1 (deleted)</div>
<div>glusterfs 12742 root 11w REG 253,11 12959954668 50351288 /var/log/glusterfs/bitd.log.1 (deleted)</div>
<div><br>
</div>
<div>Relevant part of logrotate-config</div>
<div>/var/log/glusterfs/*.log {</div>
<div> sharedscripts</div>
<div> weekly</div>
<div> rotate 52</div>
<div> missingok</div>
<div> compress</div>
<div> delaycompress</div>
<div> notifempty</div>
<div> postrotate</div>
<div> /usr/bin/killall -HUP glusterfs > /dev/null 2>&1 || true</div>
<div> /usr/bin/killall -HUP glusterd > /dev/null 2>&1 || true</div>
<div> endscript</div>
<div>}</div>
<div><span><br>
<br>
Best regards<br>
-- <br>
<b>THORGEIR MARTHINUSSEN</b><br>
<div>Systems Consultant</div>
</span></div>
</body>
</html>