[Gluster-infra] Reboot policy for the infra

Michael Scherer mscherer at redhat.com
Tue Aug 21 16:25:20 UTC 2018


so that's kernel reboot time again, this time courtesy of Intel
(again). I do not consider the issue to be "OMG the sky is falling",
but enough to take time to streamline our process to reboot.

Currently, we do not have a policy or anything, and I think the
negociation time around that is cumbersome:
- we need to reach people, which take time and add latency (would be
bad if that was a urgent issue, and likely add undeed stress while

- we need to keep track of what was supposed to be done, which is also

While that's not a problem if I had only gluster to deal with, my team
of 3 do have to deal with a few more projects than 1, and orchestrating
choice for a dozen of group is time consuming (just think last time you
had to go to a restaurant after a conference to see how hard it is to
reach agreements).

So I would propose that we simplify that with the following policy:

- Jenkins builder would be reboot by jenkins on a regular basis. 
I do not know how we can do that, but given that we have enough node to
sustain builds, it shouldn't impact developpers in a big way. The only
exception is the freebsd builder, since we only have 1 functionnal at
the moment. But once the 2nd is working, it should be treated like the

- service in HA (firewall, reverse proxy, internal squid/DNS) would be
reboot during the day without notice. Due to working HA, that's non
user impacting. In fact, that's already what I do.

- service not in HA should be pushed for HA (gerrit might get there one
day, no way for jenkins :/, need to see for postgres and so
fstat/softserve, and maybe try to get something for

- service critical and not in HA should be announced in advance.
Critical mean the service listed here: https://gluster-infra-docs.readt

- service non visible to end user (backup servers, ansible deployment
etc) can be reboot at will

Then the only question is what about stuff not in the previous
category, like softserve, fstat.

Also, all dependencies are as critical as the most critical service
that depend on them. So hypervisors hosting gerrit/jenkins are critical
(until we find a way to avoid outage), the ones for builders are not.

Thoughts, ideas ?

Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-infra/attachments/20180821/dbc2aa99/attachment.sig>

More information about the Gluster-infra mailing list