[Gluster-devel] How to cope with spurious regression failures

[Emmanuel Dreyfus]

On Tue, Jan 19, 2016 at 07:08:03PM +0530, Raghavendra Talur wrote:
> a. Allowing re-running to tests to make them pass leads to complacency with
> how tests are written.
> b. A test is bad if it is not deterministic and running a bad test has *no*
> value. We are wasting time even if the test runs for a few seconds.

I agree with your vision for the long term, but my proposal address the
short term situation. But we could use the retry approahc to fuel your
blacklist approach:

We could immagine a system where the retry feature would cast votes on 
individual tests: each time we fail once and succeed on retry, cast 
a +1 unreliable for the test.

After a few days, we will have a wall of shame for unreliable tests, 
which could either be fixed or go to the blacklist.

I do not know what software to use to collect and display the results, 
though. Should we have a gerrit change for each test?

-- 
Emmanuel Dreyfus
manu at netbsd.org