[Gluster-users] How do I rotate **all** the Gluster logs?

[Greg Scott]

Thanks Vijay - 

> The appropriate mechanism to do log rotation would be to use logrotate command with the 
> config file available at /etc/logrotate.d/glusterfs. This mechanism rotates log files of all glusterfs processes.
> The log rotate <volname> command will be deprecated in a  future release.

I wonder how to do this?  If I am following this correctly - there are a bunch of layers to make this happen:

First, a facility named anacron runs periodically to handle things such as log file maintenance.  Looking at a file named /etc/anacrontab that drives the whole thing:

[root at chicago-fw1 etc]# more /etc/anacrontab
# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days   delay in minutes   job-identifier   command
1       5       cron.daily              nice run-parts /etc/cron.daily
7       25      cron.weekly             nice run-parts /etc/cron.weekly
@monthly 45     cron.monthly            nice run-parts /etc/cron.monthly
[root at chicago-fw1 etc]#

So cron.daily probably schedules things to run every day, probably at 5 minutes past midnight.   Cron.weekly and cron.monthly are empty.   Looking at /etc/cron.daily:

[root at chicago-fw1 etc]# ls /etc/cron.daily
00webalizer  certwatch  logrotate  man-db.cron  mlocate  prelink
[root at chicago-fw1 etc]#

And the script we care about must be logrotate.  Looking at that:

[root at chicago-fw1 etc]# more /etc/cron.daily/logrotate
#!/bin/sh

/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
    /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0
[root at chicago-fw1 etc]#

A file named /etc/logrotate.conf drives it.  Looking at that file:

[root at chicago-fw1 etc]# more /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# use date as a suffix of the rotated file
dateext

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp and btmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
        minsize 1M
    rotate 1
}

/var/log/btmp {
    missingok
    monthly
    create 0600 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.
[root at chicago-fw1 etc]#

The logrotate .conf file has an include for everything in /etc/logrotate.d, so what's in there?

[root at chicago-fw1 etc]# ls /etc/logrotate.d
chrony      glusterd        httpd        libvirtd       libvirtd.uml  samba  syslog          yum
cups        glusterfsd      iptraf-ng    libvirtd.lxc   ppp           squid  vsftpd
dracut_log  glusterfs-fuse  iscsiuiolog  libvirtd.qemu  psacct        sssd   wpa_supplicant
[root at chicago-fw1 etc]#

And the piece that rotates the Gluster logs:

[root at chicago-fw1 etc]# more /etc/logrotate.d/glusterd
/var/log/glusterfs/*glusterd.vol.log {
    missingok
    postrotate
        /bin/kill -HUP `cat /var/run/glusterd.pid 2>/dev/null` 2>/dev/null || true
    endscript
}
[root at chicago-fw1 etc]#

So the task seems to be editing this file by hand to include the log file for my Gluster volume.  Or better, what about just making a file named <volume> to rotate the log file associated with <volume> named /var/log/glusterfs/<volume>.

My volume name is firewall-scripts.  Here is a first shot at it.  Create a file named /etc/logrotate.d/firewall-scripts that looks like this:

[root at chicago-fw1 logrotate.d]# more /etc/logrotate.d/firewall-scripts
/var/log/glusterfs/firewall-scripts.log

There is no way it will be this simple.    It's **never** this simple.  

And it's not.

[root at chicago-fw1 logrotate.d]# /usr/sbin/logrotate /etc/logrotate.conf
error: firewall-scripts:1 missing '{' after log files definition
error: found error in file firewall-scripts, skipping
[root at chicago-fw1 logrotate.d]#

It looks like you need a section that restarts the process associated with that log file at the end.  Here is where I get a little bit shakey - what process should I  be looking for?  Or should I just edit /etc/logrotate.d/glusterd?

Here I go again stumbling blindly down Trial and Error Avenue.  I could really use some guidance here.  

- Greg