[Gluster-users] Weird issue with logrotate of bitd.log on GlusterFS 4.1

[Thorgeir Marthinussen]

All,

We're seeing some issues with the default provided logrotate configuration in regards to the bitd.log files.
Logrotate has a postrotate-script to run "killall -HUP glusterfs", to make the processes release the filehandles and create a new logfile, and using "delaycompress".

Recently we noticed that the 'df' reported usage on our /var/log didn't match "actual" usage reported with 'du'.
Checking 'lsof' we found that basically all "bitd.log.1" files are listed as open but "deleted", when lograte did the compression.

This only applies to the bitrot-daemon logs, none of the other logs.

In addition to this we are also seeing that the bitd.log file is significantly larger on the "second" replica-node in the cluster (the "first" node is the one used in fstab on the clients).

Please note, we are currently running a two-node replica set, we have a plan to introduce an arbiter-node, but need to complete some internal testing, as one of the volumes currently contain over 20 million files, and we are unsure how the introduction of the arbiter will impact the volume.

We are running glusterfs-4.1.5-1.el7.x86_64


'lsof' output from "first" node
glusterfs 12698     root    5w   REG 253,11 611193834 50333986 /var/log/glusterfs/bitd.log.1 (deleted)
glusterfs 12698     root    8w   REG 253,11 611193834 50333986 /var/log/glusterfs/bitd.log.1 (deleted)
glusterfs 12698     root   12w   REG 253,11 611193834 50333986 /var/log/glusterfs/bitd.log.1 (deleted)

'lsof' output from "second" node
glusterfs  12742     root    5w   REG 253,11 12959954668 50351288 /var/log/glusterfs/bitd.log.1 (deleted)
glusterfs  12742     root    8w   REG 253,11 12959954668 50351288 /var/log/glusterfs/bitd.log.1 (deleted)
glusterfs  12742     root   11w   REG 253,11 12959954668 50351288 /var/log/glusterfs/bitd.log.1 (deleted)

Relevant part of logrotate-config
/var/log/glusterfs/*.log {
  sharedscripts
  weekly
  rotate 52
  missingok
  compress
  delaycompress
  notifempty
  postrotate
  /usr/bin/killall -HUP glusterfs > /dev/null 2>&1 || true
  /usr/bin/killall -HUP glusterd > /dev/null 2>&1 || true
  endscript
}


Best regards
--
THORGEIR MARTHINUSSEN
Systems Consultant
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190327/aaa0e848/attachment.html>