[Gluster-devel] Automated regression tests now part of development work flow

[Anand Avati]

Hello all,

The recent ongoing changes to the development work flow are now complete.
The highlight of the rework is the automated regression tests.

So far we have only had fixed smoke tests that were run against every patch
pre-commit (dbench, POSIX compliance). Here after, every change submitted
to Gerrit must be accompanied with test case scripts which (attempts to)
prove the correctness of the code change - as part of the new regression
test in the work flow.

These test cases will be much more concentrated and focused on the changes
brought in by the patch itself. The test cases, once committed will be part
of every future pre-commit test. This will make sure no commit in the
future will accidentally break or change the current commit either directly
or with side effects. If a future change must change the behavior of the
current patch, then the future patch must include corresponding changes to
the test cases as well. This policy will keep code and test cases in sync.

The wiring and framework for supporting this are already available in
glusterfs.git and Jenkins. Currently the infrastructure supports test cases
limited to single node. Most of the test cases can actually be implemented
in single node by having multiple instances of bricks and client mounts.
Support of multi node test cases will be added soon.

The automated smoke test's voting power in Gerrit is now downgraded to +0
Verified for PASS and -1 Verified for FAILURE. Regression test's voting
powers are now +1 Verified and -1 Verified. This means just passing the
smoke test can no more be a qualifier for verification. For those patches
for which a regression test is required but currently not automatable into
a test script, must be tested and voted manually for the Verification vote.

As the above described changes are already in effect, most of the currently
submitted patches under review will have to be resubmitted with test cases.
This is necessary to make the changes start being effective immediately. If
you have currently

Note that this opens up a new avenue for contributors in the community. You
can now contribute test cases. These contributions need not include source
code changes but just extend our regression test set. These test cases can
either be for coverage of old code and functionality, or for coverage of
your use cases which could be applicable to others as well.

The workflow document is also updated at
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow.
Specifically:
Sec 1.4
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Commit_policy
Sec 1.9
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Regression_tests_and_test_cases

More details on how to write test cases with examples and how the framework
works can be found at -

1. "tests: pre-commit regression tests"
https://github.com/gluster/glusterfs/commit/bb41c8ab88f1a3d8c54b635674d0a72133623496
2. tests/README in glusterfs.git
3. Example new change - http://review.gluster.org/4114

Please offer feedback on how the framework can be improved (inclusion of
more tools?) to capture test cases better.

Happy Hacking!
Avati
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20121020/e907a2c7/attachment-0001.html>