[Gluster-devel] WARNING: patches marked as Verified might not be

[Jeff Darcy]

As mentioned in a previous email, recent changes to speed up regression
testing have uncovered a problem with how we track the verified status
of patches.  Specifically, the fault sequence is as follows:

 1) Regression fails, "Gluster Build System" adds V-1

 2) NetBSD regression passes, "NetBSD Build System" adds V+1

 3) Smoke passes, "Gluster Build System" erases the V-1

The order of the first two might be reversed.  What's new is that
regression never used to finish before smoke, and now it usually does
(for the failure case).  So, reviewers/committers, please be sure to
check the last *regression* result before assuming that green check mark
is valid.  I'm periodically going through the list of recently reviewed
patches and manually fixing up the status, but I can't catch all of them
and TBH don't know how much longer I'll keep trying.

Gerrit/Jenkins maintainers: we have three options for a long term
solution.

 a) Run smoke and regression as separate Gerrit users, require
    concurrence for a patch to be considered V+1.  This is what we
    already do for NetBSD.  It's simple, and seems to work well.

 b) Stop triggering regression directly from the Gerrit event, trigger it
    from a successful smoke completion instead.  This is a bit more
    complicated, but has the additional benefit that regression *won't
    even run* if smoke fails (saving resources).

 c) Write a script to seek out and destroy improperly marked patches
    (basically automate what I've been doing the last coupld of days).
    It'll work, but it still leaves a window when patches are improperly
    marked.  We should only consider it if we run into significant
    problems with the other two approaches.

Anyone else have any preference for (a) vs. (b)?  I can't implement (a)
myself.  I could implement (b), but I don't want to go messing with the
Jenkins configuration unless/until we have a consensus that it's the
right thing to do.