[Gluster-devel] regressions on release-3.7 ?

[Jeff Darcy]

> IMO, if we start skipping test cases we may not be able to
> uncover bugs. Thoughts?

The purpose of a regression test is to catch unanticipated
problems related to new changes.  Once a problem is already
known, a series of tradeoffs must be evaluated.

 * What new information is gained by continuing to run the
   test?

 * What are the odds that disabling the test will prevent
   finding a previously unknown and real problem in other
   code?

 * What are the costs of continuing to run the test,
   especially in terms of endless rebase/retest cycles
   slowing other development (including bug fixes)?

In general, the cost of an individual test is so close to
zero that the benefits don't need to be large.  However,
for some of our most frequently failing tests, the cost
has grown quite large.  Worse still, the presence of so
many such tests has almost led to a complete collapse of
our test system, as each blocks fixes for the others.

*We still need those tests to pass*, but here are our
options.

 * Tests for X, Y, and Z don't pass, blocking completion
   of those features.

 * Tests for X, Y, and Z don't pass, blocking completion
   of those features *and* impeding progress on others.

If you find value in running tests that usually fail, by
all means continue to do so.  I'm sure we could set up a
Jenkins job for that - but it shouldn't IMO be the one
that's used for pre-commit verification.