[Gluster-devel] Growing concern about regular testcase failures

[Justin Clift]

On 02/04/2014, at 7:52 PM, Harshavardhana wrote:
>> I've been running the regression tests in several different
>> environments, and some of the tests are very sensitive.
> 
> Regression tests should be more stricter in their behavior, otherwise
> we have no way of knowing what is a stable patch. In future when we
> have 1000's of test cases - it could so happen that we will end up
> with a merged unstable patch.

Btw, some follow up here:

  http://www.gluster.org/pipermail/gluster-infra/2014-April/000117.html

Weirdly, I didn't see the rest of your email below (or at least I
don't remember it.  Maybe was too tired :<).


>> Test 65 of quota.t is a known bug (3.5 blocker).  Varun is
>> aware of it:
>> 
>>  https://bugzilla.redhat.com/show_bug.cgi?id=1077159
> 
> This is one bug which hits at random for pretty much any unrelated
> patch, if test case failure is the issue are we even sure quota is
> working as expected?

From memory, it was just an incorrect assumption in the test, and
it needed to wait for the storage to settle (there was a signal or
something it needed to wait for). 


>> The rpm.t one is likely something to do with the environment,
>> (eg no tty), as when I run it manually it's fine.  Haven't
>> gotten up to investigating it properly yet, but will soon.
> 
> I agree since these are using pretty much a lot of RHEL/Fedora specific tools.

This one was due to a default setting (requiretty) in /etc/sudoers.

Changing that let it work. :)


>> The most difficult thing so far has been the lack of logging
>> for stdout and stderr.  Everything useful seems to be redirected
>> to /dev/null instead of $TESTNUM.log (or $TESTNUM.stdout &
>> $TESTNUM.stderr).  With the exception of rpm.t that is, which
>> has an optional DEBUG=1 flag.
>> 
> This can be indeed modified by redirection, let me see what i can do.

Did you have time to look into this?


>> Also non-great is not being able to effectively debug bash
>> scripts.  eg set breakpoint, step, step, check variable, aha!
>> problem found, (etc).  But, that's not as critical as the
>> lack of logging.
>> 
> This would be hard to do, but we can implement a way to using
> 'abrt-cli' to print a backtrace back to the "original patch" as
> comment. Since i don't have access to these systems, i wouldn't know
> how to do.

I can create an account for you on Rackspace (as a member of our
Gluster Community organisation account), and also show you how
to kick off the regression testing stuff remotely.  The Python
code for it is here:

  https://forge.gluster.org/glusterfs-rackspace-regression-tester/glusterfs-rackspace-regression-tester/trees/master

It's grown messy and needs refactoring at some point, but it works. :)

+ Justin

--
Open Source and Standards @ Red Hat

twitter.com/realjustinclift