[Gluster-devel] Growing concern about regular testcase failures

[Justin Clift]

On 01/04/2014, at 10:46 PM, Harshavardhana wrote:
> As our regression tests have grown bigger and bigger - i have been
> observing a pattern of repeated builds which are requested every
> patch.
> 
> Here are the list of scenarios that i have observed
> 
> - Testcases pass on local laptop, while fail on Jenkins
> - Testcases fail on local laptop, while pass on jenkins
> - Testcases fail on both laptop and jenkins but on a second run they pass
> - Testcases fail randomly for a totally unrelated patch example


The regression tests really bother me too. :(

I've been running the regression tests in several different
environments, and some of the tests are very sensitive.

By "environment", I'm meaning VMs in my local desktop (in
VMware Fusion), and VM's of various cpu/mem/disk configuration
in Rackspace.  Using CentOS 6.5 in all cases, just to keep
consistency until I work things out.  Fedora 19/20/etc can
pass, but need the -Wall flag removed from build.sh, else
the compiler warnings kill the build.

These are the completely consistent failures I'm still getting
on git master, after having solved most of the others over the
last few days:

********************************************************

Test Summary Report
-------------------
./tests/basic/quota.t                           (Wstat: 0 Tests: 72 Failed: 5)
  Failed tests:  59-62, 65
./tests/basic/rpm.t                             (Wstat: 0 Tests: 4 Failed: 1)
  Failed test:  4
  Parse errors: Bad plan.  You planned 5 tests but ran 4.
./tests/bugs/bug-767095.t                       (Wstat: 0 Tests: 18 Failed: 1)
  Failed test:  13
  Parse errors: Bad plan.  You planned 19 tests but ran 18.
./tests/bugs/bug-861542.t                       (Wstat: 0 Tests: 13 Failed: 1)
  Failed test:  10
./tests/bugs/bug-963678.t                       (Wstat: 0 Tests: 16 Failed: 8)
  Failed tests:  5-11, 13
  Parse errors: Bad plan.  You planned 18 tests but ran 16.

********************************************************

Test 65 of quota.t is a known bug (3.5 blocker).  Varun is
aware of it:

  https://bugzilla.redhat.com/show_bug.cgi?id=1077159

The rpm.t one is likely something to do with the environment,
(eg no tty), as when I run it manually it's fine.  Haven't
gotten up to investigating it properly yet, but will soon.

The most difficult thing so far has been the lack of logging
for stdout and stderr.  Everything useful seems to be redirected
to /dev/null instead of $TESTNUM.log (or $TESTNUM.stdout &
$TESTNUM.stderr).  With the exception of rpm.t that is, which
has an optional DEBUG=1 flag.

Also non-great is not being able to effectively debug bash
scripts.  eg set breakpoint, step, step, check variable, aha!
problem found, (etc).  But, that's not as critical as the
lack of logging.

I'm not seeing as much random/inconsistent failure with the
tests once the VM they're is sized ok (enough memory mainly),
but it still does happen.

Don't suppose you're really good with bash scripting, and
know it well enough to fix the logging problem?  I had a
go the other day, and it was just too complex for me (ended
up writing a new basic framework in Python).

Regards and best wishes,

Justin Clift

--
Open Source and Standards @ Red Hat

twitter.com/realjustinclift