[Gluster-devel] How to cope with spurious regression failures

[Atin Mukherjee]

On 01/19/2016 10:45 AM, Emmanuel Dreyfus wrote:
> Hi
> 
> Spurious regression failures make developers frustrated. One submits a
> change and gets completely unrelated failures. The only way out is to
> retrigger regression until it passes, a boring and time-wasting task.
> Sometimes after 4 or 5 failed runs, the submitter realize there is a
> real issue and look at it, which is a waste of time and resources.
> 
> The fact that we run regression on multiple platforms makes the
> situation worse. If you have 10% of chances to hit a spurious failure on
> Linux and a 20% chances to hit a spurious failure on NetBSD (random
> number chosen), that means you get roughtly a failure for four
> submissions (random prediction, as I used random input numbers, but you
> get the idea)
> 
> Two solutions are proposed:
> 
> 1) do not run unreliable tests, as proposed by Raghavendra Talur:
> http://review.gluster.org/13173
> 
> I have nothing against the idea, but I voted down the change because it
> fails to address the need for different test blacklists on different
> platforms: we do not have the same unreliable tests on Linux and NetBSD.
> 
> 2) add a regression option to retry a failed test once, and to validate
> the regression if second attempt passes, as I proposed:
> http://review.gluster.org/13245
> 
> The idea is basicaly to automatically do what every submitter has been
> doing: retry without a thought when regression fails. The benefit of
> this approach is also that it gives us a better view of what test failed
> because of the change, and what test failed because it was unreliable.
> 
> The retry feature is optionnal and triggered by using the -r flag to
> run-tests.sh. I intend to use it on NetBSD regression to reduce the
> number of failures that annoy people. It could be used on Linux
> regression too, though I do not plan to touch that on my own.
+1 to option 2
> 
> Please people tell us what approach you prefer. 
>