[Gluster-devel] Automating spurious failure status

[Pranith Kumar Karampuri]

hi,
       I was told that Justin and I were given permission to mark a 
patch as verified+1 when the tests that failed are spurious failures. I 
think this process can be automated as well. I already have a script to 
parse the Console log to identify the tests that failed (I send mails 
using this, yet to automate the mailing part). What we need to do now is 
the following:
1) Find the list of tests that are modified/added as part of the commit.
2) Parse the list of tests that failed the full regression (I already 
have this script).

Run 'prove' on these files separately say 5/10 times. If a particular 
test fails all the time. It is a real failure with more probability. 
Otherwise it is a spurious failure.
If a file that is added as a new test fails even a single time, lets 
accept the patch after fixing the failures.
Otherwise we can give +1 on it, instead of Justin/I manually doing it.
Also send a mail to gluster-devel about the failures for each test.

Let me know if you guys have any suggestions before I start implementing 
this approach.

Pranith