[Gluster-devel] Reconfigured Jenkins (smoke) jobs to not overwrite/reset regression-test results

[Niels de Vos]

On Sat, May 30, 2015 at 01:19:02PM +0200, Niels de Vos wrote:
> Hi all,
> 
> many of the Jenkins regression-tests have finished before all the smoke
> jobs were done (we have fewer NetBSD slaves and the smoke tests were
> waiting long in the queue). This caused the Verified +1 from the
> regression-tests to be overwritten with Verified=0 once the smoke
> results were available.
> 
> The configuration change I have made in the several Jenkins jobs that
> are doing smoke testing, is like this:
> 
>   - open a Jenkins job configuration
>   - scroll down to "Gerrit trigger"
>   - click the [Advanced...] button
>   - scroll down to "Gerrit Reporting Values"
>   - enable "Skip Vote" for "Successful"
>   - click the [Save] button on the bottom of the page
> 
> This means that any successful result (Verified=0) will not get passed
> on to Gerrit. If the regression test finished before, it will not be
> overwritten anymore. This not-overwriting will be marked in a comment in
> Gerrit with "SUCCESS (skipped)" next to the job. If a smoke test fails,
> the failure will be reported and Verified=-1 gets set.
> 
> Well, that is the/my theory at least. Let me know if you notice any
> issues.

Unfortunately, this does not seem to work :-/ When all smoke tests
succeed and get marked as "skipped", the voting is still setting
Verified=0 and with that overwriting a possible Verified=+/-1 from the
regressions tests.

So, the other solutions would be:

a. chaining of Jenkins jobs (1. smoked -> 2. regressions)
b. post results of the smoke test as different users

The problem with "b" is that there is no requirement to wait for all
responses from Jenkins. Once there is one Verified=+1, patches can get
merged. Maintainers are supposed to check the passing of all required
regression tests, but they do get missed on occasion.

Ideas and suggestions welcome.

Thanks,
Niels