[Gluster-devel] Nightly Pipeline Plans

[Nigel Babu]

On Thu, Dec 15, 2016 at 01:13:05PM +0100, Niels de Vos wrote:
> On Thu, Dec 15, 2016 at 05:20:26PM +0530, Nigel Babu wrote:
> > Hello folks,
> >
> > This is a request for feedback for this plan to build a pipeline that will go
> > from a commit to builds that are tested and signed. While the
> > nightly.gluster.org site will be up by the time 3.10 is released, we will not
> > yet be automating releases with this pipeline by then.
>
> Could you explain what distributions, versions and architectures you can
> cover with this? I suspect that the different distributions that provide
> packages for the glusterfs packages have a broader range than what we
> can test in our environment?

I intend to start with with RPM builds primarily targeting CentOS 7, but
eventually, I intend to have DEBs built for Ubuntu and Debian. I'm well aware
of the limits of what we can test. :D

> Also, builds for distributions normally need to be done in the build
> system of those distributions. They will not accept builds from external
> environments... I am not sure if/how this can become a replacement for
> packages that are part of the distributions.

This is not meant to be a replacement of packages in distributions. This is
meant to be done *addition to* the work we do in pushing the packages upstream.
If you'd like to be running tests against other projects that consume Gluster -
I'd welcome the scripts to put on the CI system. Do you have any current
scripts available?

> To me this looks like a lot of duplication of the plans with the builds
> done through the CentOS CI and have the different CentOS SIGs consume
> those builds for their testing. Has there been any thought about running
> tests against other projects that use Gluster?
>
> We've already discussed that the CentOS Build System can be used for
> doing nightly builds by bots, and have these progress from testing to
> release repositories. At least for the RPMs that are used by CentOS (and
> possibly RHEL) users, I would much prefer this approach.

Our approach to packaging has been constrained by time and energy. I've talked
to Kaleb about this at the packaging BOF at GDS. This is the start of an
approach for us to figure out what our users want and what will give us the
best return for our time. There are problems that this approach will solve that
using CBS will not solve. A simple example is nightly builds of the Facebook
branch, as we will need to have them around for us and our users to test.

> Thanks,
> Niels
>
>
> > # Why
> > * Create a trusted way to build and distribute Gluster.
> > * Get our distribution packaging to be less of a mess.
> > * Automate the hell out of our packaging system so we're not caught out of the
> >   blue with packaging bugs.
> > * Fix security bugs discovered via automation so that distributions do not red
> >   flag us.
> > * In long-term, this pipeline will replace our current release process.
> >
> > # How
> > * Do a nightly build.
> > * Test the nightly build. To begin with, use Glusto tests (add Coverty,
> >   rpmlint, and other automated tests in the future).
> > * If the test passes, build packages from that commit, sign it, and make them
> >   available on nightly.gluster.org as a repo.
> >
> > # Potential Issues
> > * We want to build fresh packages from the known good commit rather than
> >   copying the packages for security reasons.
> > * Password-less gpg and security. The solution needs to work for .deb and .rpm
> >
> > # Detailed Steps
> > * Trigger the nightly build from build.gluster.org
> > * Once that passes, trigger a glusto build.
> > * Once that passes, trigger a new RPM build off separate build nodes.
> > * The signing machine will pick up the new packages, sign them, and push them
> >   to nightly.gluster.org
> > * The signing machine needs to be disconnected from the internet entirely.
> >
> > # Quantum of Work
> > * New Jenkins jobs to trigger the process
> > * A key signing machine isolated from the internet.
> > * The code to take built packages, sign them, and push them out.
> > * A machine to deliver the packages running chacra[1].
> >
> > [1]: https://github.com/ceph/chacra
> >
> > --
> > nigelb
>
>
>
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
>



--
nigelb
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20161219/2ba7c096/attachment.sig>