<div dir="ltr">Thanks for your quick work on this, get some rest! <div>We can look at supercolony next week when you're back in action. <br><div>- amye </div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jan 6, 2018 at 11:48 AM, Michael Scherer <span dir="ltr"><<a href="mailto:mscherer@redhat.com" target="_blank">mscherer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">Le samedi 06 janvier 2018 à 11:44 +0100, Michael Scherer a écrit :<br>
> Le vendredi 05 janvier 2018 à 14:24 +0100, Michael Scherer a écrit :<br>
> > Hi,<br>
> ><br>
> > unless you are living in a place without any internet (a igloo in<br>
> > Antartica, the middle of the Gobi desert, a bunker in Switzerland<br>
> > or<br>
> > simply the Paris underground train), you may have seen the news<br>
> > that<br>
> > this week is again a security nightmare (also called "just a normal<br>
> > Wednesday" among practitioners ), and that we have important kernel<br>
> > patch to push, that do requiers a reboot. <br>
> ><br>
> > See <a href="https://spectreattack.com/" rel="noreferrer" target="_blank">https://spectreattack.com/</a> <br>
> ><br>
> > While I suspect our infra will not be targeted and there is more<br>
> > venue<br>
> > to attack on local computers and browsers who are the one running<br>
> > proprietary random code in form of JS on a regular basis, we still<br>
> > have<br>
> > to upgrade everything to be sure.<br>
> ><br>
> > Therefor, I am gonna have to reboot all the infra (yes, the 83<br>
> > servers), minus the few servers I already did reboot (because in<br>
> > HA,<br>
> > or<br>
> > not customer facing) tomorrow.<br>
> ><br>
> > I will block jenkins, and wait for the jobs to be finished before<br>
> > rebooting the various servers. I will send a email tomorrow once<br>
> > the<br>
> > reboot start (e.g., when/if I wake up), and another one things are<br>
> > good<br>
> > (or if stuff broke in a horrible fashion too, as it happened<br>
> > today).<br>
> ><br>
> > If there is some precaution or anything to take, people have around<br>
> > 24h<br>
> > to voice their concerns. <br>
><br>
> Reboot is starting. I already did various backend servers, the<br>
> document<br>
> I used for tracking the work is on <br>
> <a href="https://bimestriel.framapad.org/p/gluster_infra_reboot" rel="noreferrer" target="_blank">https://bimestriel.framapad.<wbr>org/p/gluster_infra_reboot</a><br>
<br>
</div></div>So almost all Linux servers got rebooted, most without issues, but<br>
during the day, I started to have the first symptom of a cold<br>
(headaches, shivering, etc), so I had to ping Nigel to finish the last<br>
server (who wasn't without issue)<br>
<br>
<br>
For people who do not want gruesome details on the reboots, you can<br>
stop here.<br>
<br>
<br>
We did got some trouble with:<br>
<br>
- a few servers on Rackspace (mostly infra) with cloud-init reseting<br>
the configuration to dhcp, and the dhcp not working. I am finally<br>
changing that and was in the course of fixing it for good before going<br>
back to bed.<br>
<br>
- gerrit didn't start automatically at boot. I know we had a fix for<br>
that, but not sure on why it didn't work, or if we didn't deployed yet.<br>
<br>
- supercolony seems to be unable to boot the latest kernel. It went so<br>
bad that the emergency console wasn't working. A erroneous message said<br>
"disabled for your account", so I did open a rackspace ticket and<br>
waited. This occurred as I started to not feel well, so I didn't really<br>
searched more, or I would have:<br>
- seen that the console was working for others servers (thus<br>
erroneous messages)<br>
- would have tried harder to boot another kernel <br>
- search a bit more on internal list that said "there is some issue<br>
somewhere around RHEL 6". Didn't investigate more, but that's also what<br>
happened.<br>
<br>
In the end, Nigel took over the problem solving and pinged harder<br>
Rackspace, whose support suggested to boot another kernel, which he did<br>
(but better than I did).<br>
<br>
And thus supercolony is back, but not upgraded.<br>
<br>
The last one still puzzle me, because the current configuration is:<br>
"default=2", so that should start the 3rd kernel in the list.<br>
<br>
Grub doc say "The first entry (here, counting starts with number zero,<br>
not one!) will be the default choice", it was "0" when i first tried to<br>
boot another kernel (switched to 1).<br>
<br>
So since we have:<br>
<br>
[root@supercolony ~]# grep title /boot/grub/menu.lst <br>
title Red Hat Enterprise Linux Server (2.6.32-696.18.7.el6.x86_64)<br>
title Red Hat Enterprise Linux Server (2.6.32-696.16.1.el6.x86_64)<br>
title Red Hat Enterprise Linux Server (2.6.32-642.15.1.el6.x86_64)<br>
<br>
default=1 should have used 2.6.32-696.16.1, but it didn't boot.<br>
<br>
Nigel changed it for "default=2", so that should have used 2.6.32-<br>
642.15.1, but plot twist...<br>
<br>
# uname -a<br>
Linux <a href="http://supercolony.gluster.org" rel="noreferrer" target="_blank">supercolony.gluster.org</a> 2.6.32-696.16.1.el6.x86_64 #1 SMP Sun Oct<br>
8 09:45:56 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux<br>
<br>
So there is something fishy for grub, but as I redact that from my bed,<br>
maybe the problem is on my side. I am sure it will be clearer once I<br>
hit "send".<br>
<br>
So, to recap, we have one or two servers to upgrade (cf the pad), the<br>
*bsd are not patched yet (I quickly checked their lists, but I do not<br>
expect it soon), but since the more urgent issues were on the<br>
hypervisor side, we are ok for that.<br>
<br>
The grub on supercolony need to be investigated, and supercolony should<br>
be upgraded as well.<br>
<br>
I also need to take some rest.<br>
<br>
Many thanks for Nigel for taking over when my body failed me.<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
--<br>
Michael Scherer<br>
Sysadmin, Community Infrastructure and Platform, OSAS<br>
<br>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-infra mailing list<br>
<a href="mailto:Gluster-infra@gluster.org">Gluster-infra@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-infra" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-infra</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Amye Scavarda | <a href="mailto:amye@redhat.com" target="_blank">amye@redhat.com</a> | Gluster Community Lead</div></div>
</div>