[Gluster-devel] [Gluster-infra] Freebsd builder upgrade to 10.4, maybe 11

[Michael Scherer]

Le mercredi 12 septembre 2018 à 12:35 +0200, Michael Scherer a écrit :
> Le mardi 11 septembre 2018 à 19:54 +0530, Nigel Babu a écrit :
> > On Tue, Sep 11, 2018 at 7:06 PM Michael Scherer <misc at redhat.com>
> > wrote:
> > 
> > > And... rescue mode is not working. So the server is down until
> > > Rackspace fix it.
> > > 
> > > Can someone disable the freebsd smoke test, as I think our 2nd
> > > builder
> > > is not yet building fine ?
> > > 
> > 
> > 
> > Disabled. Please do not merge any JJB review requests until this is
> > fixed.
> 
> So, just to keep people updated on this adventure:
> 
> - after 3 or 4h, the rescue mode managed to appear. Turn out that it
> take time to copy the rescue image to the cloud, and I guess no one
> recently did that for freebsd. Rackspace say "40 minutes", which is
> still a lot.
> 
> This morning, I was greeted by a welcoming prompt saying "zfs, can't
> mount the root" or something, because the rescue mode of rackspace
> was
> a bit broken. Trying to fix it, I did reboot out of rescue mode.
> 
> So I went back to my initial plan: "boot in rescue mode". 
> 
> Thanks to surhuman reflexes I acquired dodging nerfs guns in the
> office, I manage to hit the "s" key at the right time. While it seems
> easy, the remote console of rackspace is a bit slow to show the boot
> loader, the latency from France over the atlantic is a noticable, and
> the interface disconnect itself every minutes of idling, and every
> time
> the video mode is changed (like when it go from POST to bootloader
> menu). So that was a race between the machine and me, and I won it,
> with a "#_ " prompt waiting for me.
> 
> Of course, things would be too simple if that was just that, so / was
> readonly. And that's freebsd, so "mount -o rw,remount /" didn't work.
> It didn't work for 2 reasons:
> 
> - zfs do not work like this ("zfs set readonly=off zroot", for people
> asking how to do later)
> 
> - the keyboard was kinda broken. Not broke like "that's a us layout
> and
> misc is using a french layout" broken, cause that, I know how to deal
> with it. More broken with "only a-z keys are working, and the rest
> are
> randomly placed somewhere else". It turn out that without '.' and
> '/',
> thing can be complicated in the shell. But I am full of ressources,
> and
> judicious use of <tab> + <backspace> did let me get what I needed.
> 
> So I did manage to change the root password, reboot again. 
> 
> Password didn't work the first time (did I mention latency), I retry,
> it worked. 
> 
> And so, network is fine. But sshd didn't start. Why ? because it
> check
> the config, and the config did show a warning on "duplicate line".
> 
> The upgrade did change sshd config file, adding a ton of comment, and
> at the end, a duplicated line:
> 
>   Subsystem sftp /usr/libexec/sftp-server
> 
> And upon removal, things were working.
> 
> However, I did reboot to test and ... still broken. So back to me
> racing against the bootloader I guess. (cause the temp password I set
> is not working again, wonder if there was some fallback due to zfs or
> something).

So for a reason I do not understand, the reboot seems to have reverted
the server to the initial broken state. I would blame either the cloud,
or zfs. Anyway, now the server is working, I am upgrading userspace and
cleaning a bit.

But... I of course hit this bug:
https://github.com/freebsd/pkg/issues/902

But the -f option do seems to work fine. 

So now, after a reboot to verify, a few upgrade, the system is up to
date. I will reenable as a voting member, if there is a issue, please
tell us by opening a bug (with link to the failure).
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180912/ab984e0d/attachment.sig>