[Gluster-users] Auto Update did not restart services

[Niels de Vos]

On Thu, Jun 26, 2014 at 07:39:22PM -0400, Gene Liverman wrote:
> My systems are set to automatically install updates. Since I installed
> Gluster from the repos this means I went from 3.5 to 3.5.1 automatically.
>  The problem is this: the installer did not stop or restart the service so
> when I checked the status via /sbin/service on RHEL6 it said it was dead. A
> simple service restart fixed this but, until I found the problem, it was
> off line.  Is this a bug or do I need to disable the repo and manually
> check for updates to Gluster or what? Thanks!

This has been reported by an other user yesterday too. We've looked at 
the logs and found the root cause:

> The post installation script for the glusterfs-server handles the 
> restarting of glusterd incorrect. This caused an outage when the 
> glusterfs-server package was automatically updated.
> 
> After checking the logs together with Patrick, we came to the 
> conclusion that the running glusterd should have received a signal and 
> would be exiting. However, the script does not wait for the running 
> glusterd to exit, and starts a new glusterd process immediately after 
> sending the SIGTERM. In case the 1st glusterd process has not exited 
> yet, the new glusterd process can not listen on port 24007 and exits. 
> The 1st glusterd will exit eventually too, leaving the service 
> unavailable.
> 
> Snippet from the .spec:
> 
>  735 %post server
>  ...
>  769 pidof -c -o %PPID -x glusterd &> /dev/null
>  770 if [ $? -eq 0 ]; then
>  ...
>  773     killall glusterd &> /dev/null
>  774     glusterd --xlator-option *.upgrade=on -N
>  775 else
>  776     glusterd --xlator-option *.upgrade=on -N
>  777 fi
>  ...

(also available in https://bugzilla.redhat.com/1113543#c1)

This seems to be a bug that affects any update and I do not think it is 
unique to 3.5.1. A fix will be included in 3.5.2, the 'workaround' is to 
(re)start the glusterd service after updating the packages.

I'm sorry for any unexpected outages that this caused! At least you've 
now tested your Gluster monitoring setup ;-)

HTH,
Niels