[Gluster-infra] Migrating Gerrit and Jenkins out of iWeb

[Kaushal M]

On Fri, Feb 12, 2016 at 10:47 PM, Michael Scherer <mscherer at redhat.com> wrote:
> Le vendredi 12 février 2016 à 12:16 +0100, Michael Scherer a écrit :
>> Le mercredi 10 février 2016 à 19:18 +0530, Kaushal M a écrit :
>> > On Wed, Feb 10, 2016 at 1:53 PM, Michael Scherer <mscherer at redhat.com> wrote:
>> > > Le mercredi 10 février 2016 à 11:44 +0530, Kaushal M a écrit :
>> > >> On Sun, Feb 7, 2016 at 2:34 PM, Michael Scherer <mscherer at redhat.com> wrote:
>> > >> > Le samedi 06 février 2016 à 18:17 -0500, Vijay Bellur a écrit :
>> > >> >> I think starting around 0900 UTC on Friday of next week (12th Feb)
>> > >> >> should be possible. We should be done with 3.7.8 before that and can
>> > >> >> afford a bit of downtime then. In case any assistance is needed post the
>> > >> >> migration, we can have folks around the clock to help.
>> > >> >>
>> > >> >> If migration fails for an unforeseen reason, would we be able to
>> > >> >> rollback and maintain status quo?
>> > >> >
>> > >> > Yes. Worst case, I think people would just redo a few reviews or push
>> > >> > again patch.
>> > >> >
>> > >> > Also, since gerrit is critical, I wonder what is the support of gerrit
>> > >> > for slaves and replication.
>> > >>
>> > >> Michael are you okay with the time? If you are, I think we should
>> > >> announce the migration.
>> > >
>> > > I am ok.
>> >
>> > So as discussed in the community meeting, we will be announcing the
>> > migration and downtime. Vijay, you said you required the exact
>> > schedule? This is what I expect
>> >
>> > Friday 12th Feb 2016
>> >
>> > 0900 UTC : build.gluster.org and review.gluster.org are taken down and
>> > migration begins.
>> > <1-2 hrs?>: Michael copies over data onto the RH community infra and
>> > sets up the VMs.
>> > <? hrs>: The DNS records are updated, and some time is for it to propogate.
>> > <3hr?>: Verify everything is working well (I can help with this). We'd
>> > possibly need to run a regression job, so this will take longest I
>> > think.
>> >
>> > 1700UTC <or earlier>: We announce the finish of the migration and open
>> > the services back up. If migration failed, we bring the existing
>> > servers back on, and continue on.
>>
>> So, for people wanting to know, the migration has started.
>>
>> 1) gerrit
>> ----------
>> gerrit seems to be ok, it has a new IP 66.187.224.201
>>
>> people who used to have access there need to contact me so I can
>> explain/create required account on the new virt host.
>>
>> the old VM is shutdown for now. I will make a backup copy of the disk.
>>
>> 2) jenkins
>> ----------
>>
>> so the preparation of the migration didn't worked as well, so we have to
>> copy the VM when offline, and then prepare it later ( ie, import in
>> libvirt, adjust network, etc). So jenkins master is offline (sorry manu
>> for your tcpdump monitoring), we are making a copy of the disk that is
>> gonna take between 3 and 14h (most likely < 4h, but rsync estimation is
>> moving a lot), and then restart like before.
>
> So turn out I was wrong, and the copy is taking a much longer time.
> Despites me trying to investigate why, it didn't copied everything.
>
> So the new plan is to stop the VM, make a local copy of the disk, start
> the VM, and then copy that one over the new server, prepare for the new
> host (firewall, public ip, etc), and sync from the old VM to the new
> host.
>
> All is ok with this plan ?

Sounds good to me. No need to wait for the copy to happen and the
downtime should be minimal like with gerrit.

> --
> Michael Scherer
> Sysadmin, Community Infrastructure and Platform, OSAS
>
>