[Gluster-infra] Migrating Gerrit and Jenkins out of iWeb

[Michael Scherer]

Le mardi 16 février 2016 à 17:39 -0800, Amye Scavarda a écrit :
> On Tue, Feb 16, 2016 at 12:20 PM, Michael Scherer <mscherer at redhat.com> wrote:
> > Le mardi 16 février 2016 à 23:05 +0530, Kaushal M a écrit :
> >> On Feb 12, 2016 10:50 PM, "Kaushal M" <kshlmster at gmail.com> wrote:
> >> >
> >> > On Fri, Feb 12, 2016 at 10:47 PM, Michael Scherer <mscherer at redhat.com>
> >> wrote:
> >> > > Le vendredi 12 février 2016 à 12:16 +0100, Michael Scherer a écrit :
> >> > >> Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit :
> >> > >> > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer <mscherer at redhat.com>
> >> wrote:
> >> > >> > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit :
> >> > >> > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer <
> >> mscherer at redhat.com> wrote:
> >> > >> > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit :
> >> > >> > >> >> I think starting around 0900 UTC on Friday of next week (12th
> >> Feb)
> >> > >> > >> >> should be possible. We should be done with 3.7.8 before that
> >> and can
> >> > >> > >> >> afford a bit of downtime then. In case any assistance is
> >> needed post the
> >> > >> > >> >> migration, we can have folks around the clock to help.
> >> > >> > >> >>
> >> > >> > >> >> If migration fails for an unforeseen reason, would we be able
> >> to
> >> > >> > >> >> rollback and maintain status quo?
> >> > >> > >> >
> >> > >> > >> > Yes. Worst case, I think people would just redo a few reviews
> >> or push
> >> > >> > >> > again patch.
> >> > >> > >> >
> >> > >> > >> > Also, since gerrit is critical, I wonder what is the support of
> >> gerrit
> >> > >> > >> > for slaves and replication.
> >> > >> > >>
> >> > >> > >> Michael are you okay with the time? If you are, I think we should
> >> > >> > >> announce the migration.
> >> > >> > >
> >> > >> > > I am ok.
> >> > >> >
> >> > >> > So as discussed in the community meeting, we will be announcing the
> >> > >> > migration and downtime. Vijay, you said you required the exact
> >> > >> > schedule? This is what I expect
> >> > >> >
> >> > >> > Friday 12th Feb 2016
> >> > >> >
> >> > >> > 0900 UTC : build.gluster.org and review.gluster.org are taken down
> >> and
> >> > >> > migration begins.
> >> > >> > <1-2 hrs?>: Michael copies over data onto the RH community infra and
> >> > >> > sets up the VMs.
> >> > >> > <? hrs>: The DNS records are updated, and some time is for it to
> >> propogate.
> >> > >> > <3hr?>: Verify everything is working well (I can help with this).
> >> We'd
> >> > >> > possibly need to run a regression job, so this will take longest I
> >> > >> > think.
> >> > >> >
> >> > >> > 1700UTC <or earlier>: We announce the finish of the migration and
> >> open
> >> > >> > the services back up. If migration failed, we bring the existing
> >> > >> > servers back on, and continue on.
> >> > >>
> >> > >> So, for people wanting to know, the migration has started.
> >> > >>
> >> > >> 1) gerrit
> >> > >> ----------
> >> > >> gerrit seems to be ok, it has a new IP 66.187.224.201
> >> > >>
> >> > >> people who used to have access there need to contact me so I can
> >> > >> explain/create required account on the new virt host.
> >> > >>
> >> > >> the old VM is shutdown for now. I will make a backup copy of the disk.
> >> > >>
> >> > >> 2) jenkins
> >> > >> ----------
> >> > >>
> >> > >> so the preparation of the migration didn't worked as well, so we have
> >> to
> >> > >> copy the VM when offline, and then prepare it later ( ie, import in
> >> > >> libvirt, adjust network, etc). So jenkins master is offline (sorry manu
> >> > >> for your tcpdump monitoring), we are making a copy of the disk that is
> >> > >> gonna take between 3 and 14h (most likely < 4h, but rsync estimation is
> >> > >> moving a lot), and then restart like before.
> >> > >
> >> > > So turn out I was wrong, and the copy is taking a much longer time.
> >> > > Despites me trying to investigate why, it didn't copied everything.
> >> > >
> >> > > So the new plan is to stop the VM, make a local copy of the disk, start
> >> > > the VM, and then copy that one over the new server, prepare for the new
> >> > > host (firewall, public ip, etc), and sync from the old VM to the new
> >> > > host.
> >> > >
> >> > > All is ok with this plan ?
> >> >
> >> > Sounds good to me. No need to wait for the copy to happen and the
> >> > downtime should be minimal like with gerrit.
> >> >
> >>
> >> Can we schedule this minimal downtime sometime tomorrow? Or if possible
> >> tonight itself. Since we don't require a long downtime we can do this
> >> anytime we like.
> >
> > No. I am at ansiblefest/contributor meeting tomorrow and the day after,
> > and then I took a PTO on friday, until end of February.
> 
> So it sounds like we're going to have to wait until March for this, if
> I'm reading this correctly.
> Accurate?

Yep.

Cf my calendar for the date I am out
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://www.gluster.org/pipermail/gluster-infra/attachments/20160217/ea2c91bd/attachment.sig>