[Gluster-devel] Growing concern about regular testcase failures

[Harshavardhana]

> I've been running the regression tests in several different
> environments, and some of the tests are very sensitive.
>

Regression tests should be more stricter in their behavior, otherwise
we have no way of knowing what is a stable patch. In future when we
have 1000's of test cases - it could so happen that we will end up
with a merged unstable patch.

>
>
> Test 65 of quota.t is a known bug (3.5 blocker).  Varun is
> aware of it:
>
>   https://bugzilla.redhat.com/show_bug.cgi?id=1077159
>

This is one bug which hits at random for pretty much any unrelated
patch, if test case failure is the issue are we even sure quota is
working as expected?

> The rpm.t one is likely something to do with the environment,
> (eg no tty), as when I run it manually it's fine.  Haven't
> gotten up to investigating it properly yet, but will soon.
>

I agree since these are using pretty much a lot of RHEL/Fedora specific tools.

> The most difficult thing so far has been the lack of logging
> for stdout and stderr.  Everything useful seems to be redirected
> to /dev/null instead of $TESTNUM.log (or $TESTNUM.stdout &
> $TESTNUM.stderr).  With the exception of rpm.t that is, which
> has an optional DEBUG=1 flag.
>
This can be indeed modified by redirection, let me see what i can do.

> Also non-great is not being able to effectively debug bash
> scripts.  eg set breakpoint, step, step, check variable, aha!
> problem found, (etc).  But, that's not as critical as the
> lack of logging.
>
This would be hard to do, but we can implement a way to using
'abrt-cli' to print a backtrace back to the "original patch" as
comment. Since i don't have access to these systems, i wouldn't know
how to do.

-- 
Religious confuse piety with mere ritual, the virtuous confuse
regulation with outcomes