[Gluster-devel] [Gluster-users] Automated regression tests now part of development work flow

[James]

On Sat, 2012-10-20 at 14:23 -0700, Anand Avati wrote:
> Hello all,
> 
> The recent ongoing changes to the development work flow are now complete.
> The highlight of the rework is the automated regression tests.
This is awesome!

> 
> So far we have only had fixed smoke tests that were run against every patch
> pre-commit (dbench, POSIX compliance). Here after, every change submitted
> to Gerrit must be accompanied with test case scripts which (attempts to)
> prove the correctness of the code change - as part of the new regression
> test in the work flow.
> 
> These test cases will be much more concentrated and focused on the changes
> brought in by the patch itself. The test cases, once committed will be part
> of every future pre-commit test. This will make sure no commit in the
> future will accidentally break or change the current commit either directly
> or with side effects. If a future change must change the behavior of the
> current patch, then the future patch must include corresponding changes to
> the test cases as well. This policy will keep code and test cases in sync.
I don't suppose someone knows offhand of a way to automatically "watch"
for proposals to certain test cases? This could be a useful way for a
programmer to get a chance to speak up about an upcoming incompatible
change. Anyways, low priority, if someone has already hacked this
together with git, or if you'd be interested in this sort of thing, let
me know.

> 
> The wiring and framework for supporting this are already available in
> glusterfs.git and Jenkins. Currently the infrastructure supports test cases
> limited to single node. Most of the test cases can actually be implemented
> in single node by having multiple instances of bricks and client mounts.
> Support of multi node test cases will be added soon.
> 
> The automated smoke test's voting power in Gerrit is now downgraded to +0
> Verified for PASS and -1 Verified for FAILURE. Regression test's voting
> powers are now +1 Verified and -1 Verified. This means just passing the
> smoke test can no more be a qualifier for verification. For those patches
> for which a regression test is required but currently not automatable into
> a test script, must be tested and voted manually for the Verification vote.
> 
> As the above described changes are already in effect, most of the currently
> submitted patches under review will have to be resubmitted with test cases.
> This is necessary to make the changes start being effective immediately. If
> you have currently
> 
> Note that this opens up a new avenue for contributors in the community. You
> can now contribute test cases. These contributions need not include source
> code changes but just extend our regression test set. These test cases can
> either be for coverage of old code and functionality, or for coverage of
> your use cases which could be applicable to others as well.
I will put it onto my TODO list to write some test cases that "watch"
the functionality used by my puppet-gluster module.

> 
> The workflow document is also updated at
> http://www.gluster.org/community/documentation/index.php/Development_Work_Flow.
> Specifically:
> Sec 1.4
> http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Commit_policy
> Sec 1.9
> http://www.gluster.org/community/documentation/index.php/Development_Work_Flow#Regression_tests_and_test_cases
> 
> More details on how to write test cases with examples and how the framework
> works can be found at -
> 
> 1. "tests: pre-commit regression tests"
> https://github.com/gluster/glusterfs/commit/bb41c8ab88f1a3d8c54b635674d0a72133623496
> 2. tests/README in glusterfs.git
> 3. Example new change - http://review.gluster.org/4114
> 
> Please offer feedback on how the framework can be improved (inclusion of
> more tools?) to capture test cases better.
> 
> Happy Hacking!
> Avati
Thanks!!
James

> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20130225/385f0f4d/attachment-0001.sig>