[Gluster-devel] Growing concern about regular testcase failures

[Harshavardhana]

As our regression tests have grown bigger and bigger - i have been
observing a pattern of repeated builds which are requested every
patch.

Here are the list of scenarios that i have observed

- Testcases pass on local laptop, while fail on Jenkins
- Testcases fail on local laptop, while pass on jenkins
- Testcases fail on both laptop and jenkins but on a second run they pass
- Testcases fail randomly for a totally unrelated patch example

Example:

http://review.gluster.com/#/c/7317/

Gluster Build System Mar 28 9:20 AM
---------------------------------------------------------
Patch Set 6: Verified-1

http://build.gluster.org/job/regression/3885/consoleFull : FAILED
---------------------------------------------------------
Gluster Build System Mar 28 8:55 PM
---------------------------------------------------------
Patch Set 6: Verified+1

http://build.gluster.org/job/regression/3893/consoleFull : SUCCESS
---------------------------------------------------------

There has been no change the patch which was submitted.

Another example - http://review.gluster.com/#/c/7278/
---------------------------------------------------------
Gluster Build System Mar 31 10:22 AM
Patch Set 4: Verified-1
http://build.gluster.org/job/regression/3900/consoleFull : FAILED
Kaleb KEITHLEY Mar 31 11:02 AM
Patch Set 4:
build.gluster.org:/d/cores_from_mar_31/core.27188 is from one of these tests:
...
 [10:51:52] ./tests/bugs/bug-1034085.t ........................ ok       11 s
No volumes present
setfattr: /mnt/glusterfs/0/: Operation not permitted setfattr:
/mnt/glusterfs/0/: Operation not permitted
[10:52:05] ./tests/bugs/bug-1034716.t ........................ ok 13 s
[10:52:24] ./tests/bugs/bug-1035576.t ........................ ok 19 s
No volumes present
[10:52:54] ./tests/bugs/bug-1037501.t ........................ ok 30 s
No volumes present
4+0 records in 4+0 records out 4194304 bytes (4.2 MB) copied,
0.0248987 s, 168 MB/s 6+0 records in 6+0 records out 6291456 bytes
(6.3 MB) copied, 0.0393089 s, 160 MB/s
[10:53:16] ./tests/bugs/bug-1038598.t ........................ ok 22 s
Gluster Build System Mar 31 11:54 AM
Patch Set 4: Verified+1
http://build.gluster.org/job/regression/3901/consoleFull : SUCCESS
---------------------------------------------------------

My general growing concern is we might be taking in patches which are
not failing
where they should have while a growing number of valid patches might just fail
and vice-versa.

This will grow larger and larger as the individual test cases are not
consistent in the way they behave in different outcomes - we are
looking a major issue here. Since we can't be certain of the
"regression" validity that a bug is fixed or the same bug is
re-introduced by a new patch.

Cheers
-- 
Religious confuse piety with mere ritual, the virtuous confuse
regulation with outcomes