[Gluster-devel] Regression-test pipeline report

[Jeff Darcy]

Today was a bad day.  Mostly it was my fault, because there were two
problems with the patch I submitted to make tests fail more quickly.

   It was incorrectly reporting failed tests as successful.

   It was causing regression to complete before smoke most of the time,
   which exposed a bug in how we handle the Verified flag in regression
   and smoke.  Specifically, smoke would *clear* the flag already set by
   regression, allowing the netbsd-regression result to dominate the
   overall result.

Either problem would cause a patch to show up as V+1 when it shouldn't
have.  As penance, I spent much of the day (normally a day off BTW)
manually checking results and resetting these flags.  If your patch got
a result of "FAILURE" (not "FAILED") or "FAILED (script error)" that's
what happened.  To be clear: the result is exactly what it would have
been before, only arrived at more quickly even with the human delay.
I'd be interested in feedback on how that's working out for people.

After all of this, it became apparent that this was one of our worst
regression-testing days so far.  It's easier to look on Gerrit than on
Jenkins because of all the false positives, and right now I see only
*eight* of the most recent fifty patches passed regression.  Two of
those don't count, because they're my own changes to the test scripts
(more about those in a moment) and three more have no result at all yet.
The most common cause of failure was volume-snapshot-clone.t, with uss.t
coming in a fairly distant second . . . which brings me to those test
script changes.  There are actually two on master, with a backport of
one for 3.7:

   http://review.gluster.org/10259
   http://review.gluster.org/10255 (backport to 3.7)
   http://review.gluster.org/10277

The last one might be controversial, because it explicitly skips the
two tests mentioned above.  *Skipping regression tests is a shameful
thing*!  It frustrates me no end that things have gotten this bad, and
we absolutely should fix those tests before we release anything, but
meanwhile they're getting in the way of doing any *other* work.  We need
to get the tests for that other work flowing, and *simultaneously* work
on fixing the failing regression tests from these two on down.  Yes, I
know I'm shouting.  Sorry about that.  I just want to get us out of this
abyss before I hit the road for Bangalore and Barcelona in the next few
weeks.

Lastly, I have a lead on some of the core dumps that have occurred
during regression tests.  See the following bug for details.

   https://bugzilla.redhat.com/show_bug.cgi?id=1212660