[Gluster-devel] Reconfigured Jenkins (smoke) jobs to not overwrite/reset regression-test results

[Niels de Vos]

On Sat, May 30, 2015 at 09:08:55PM +0530, Atin Mukherjee wrote:
> On 30 May 2015 20:38, "Niels de Vos" <ndevos at redhat.com> wrote:
> >
> > On Sat, May 30, 2015 at 01:19:02PM +0200, Niels de Vos wrote:
> > > Hi all,
> > >
> > > many of the Jenkins regression-tests have finished before all the smoke
> > > jobs were done (we have fewer NetBSD slaves and the smoke tests were
> > > waiting long in the queue). This caused the Verified +1 from the
> > > regression-tests to be overwritten with Verified=0 once the smoke
> > > results were available.
> > >
> > > The configuration change I have made in the several Jenkins jobs that
> > > are doing smoke testing, is like this:
> > >
> > >   - open a Jenkins job configuration
> > >   - scroll down to "Gerrit trigger"
> > >   - click the [Advanced...] button
> > >   - scroll down to "Gerrit Reporting Values"
> > >   - enable "Skip Vote" for "Successful"
> > >   - click the [Save] button on the bottom of the page
> > >
> > > This means that any successful result (Verified=0) will not get passed
> > > on to Gerrit. If the regression test finished before, it will not be
> > > overwritten anymore. This not-overwriting will be marked in a comment in
> > > Gerrit with "SUCCESS (skipped)" next to the job. If a smoke test fails,
> > > the failure will be reported and Verified=-1 gets set.
> > >
> > > Well, that is the/my theory at least. Let me know if you notice any
> > > issues.
> >
> > Unfortunately, this does not seem to work :-/ When all smoke tests
> > succeed and get marked as "skipped", the voting is still setting
> > Verified=0 and with that overwriting a possible Verified=+/-1 from the
> > regressions tests.
> >
> > So, the other solutions would be:
> >
> > a. chaining of Jenkins jobs (1. smoked -> 2. regressions)
> > b. post results of the smoke test as different users
> >
> > The problem with "b" is that there is no requirement to wait for all
> > responses from Jenkins. Once there is one Verified=+1, patches can get
> > merged. Maintainers are supposed to check the passing of all required
> > regression tests, but they do get missed on occasion.
> Makes sense to me, if we can achieve 'a' then definitely its a better
> option over 'b'.

Hmm, the standard Jenkins job dependencies are not sufficient for us. We
need to start multiple different smoke jobs, wait for them, and only
when they succeeded, start the regression tests. Fortunately, Jenkins
has a very rich eco-system with 1000+ plugins, and there is one that
seems to suit our needs:

    https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin

I'll try that out in a test environment and will see if it indeed works
as it is described.

Cheers,
Niels