[Gluster-infra] [Gluster-devel] Infra outage today

[Michael Scherer]

Le mardi 14 mars 2017 à 18:01 +0530, Nigel Babu a écrit :
> All servers are now shutdown in preparation for the move. We will have services
> restored (hopefully) by the end of EDT working day today. Michael or I will
> post updates when we have one.

So the servers have moved, and they are plugged and up since 19h30 CET
(so 10h30 UTC). I was in the train back home, and I was informed that:
"there is some disk with red blinking light". I spare you the suspens,
that's supermicro way of saying "this is a spare drive", while anybody
would think "this disk is broken".

So no issue on the move.

Now on the issues after the move:

- formicary ethernet connection didn't came back on boot. I will have to
investigate, but the server is for now not in production, so it is not
urgent

- jenkins didn't start, or rather, httpd on the jenkins server didn't
start most likely due to this:
[Tue Mar 14 10:45:52 2017] [alert] (EAI 3)Temporary failure in name
resolution: mod_unique_id: unable to find IPv4 address of
"build.gluster.org"

I will dig in more details, but restarting httpd was enough to fix this.

- gerrit wasn't set to start on boot. Like last time, we are working on
it with proper playbook for ansible. 

However, after starting manually, I was unable to see anything on
dev.gluster.org. I found out this was still pointing to the old ip that
was a alias of the old server and that we never migrated. So I added
back the old ip, changed the DNS (so it should be ok next time). The Ip
was also used for another VM but not used yet.

And also, the real service name is "review.gluster.org", who was working
fine (but of course, I forgot).

I will submit a patch to verify everything work after getting dinner and
checking the others server I am supposed to fix. In the mean time, feel
free to verify all is working and ping me on irc if that's not the case.






-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-infra/attachments/20170314/518b307d/attachment.sig>