[Gluster-Maintainers] proposal on few changes in regression voting to keep upstream 'master' fast moving

[Vijay Bellur]

On Mon, May 22, 2017 at 5:23 AM, Amar Tumballi <atumball at redhat.com> wrote:

> All,
>
> NOTE: Currently sent to only maintainers list, planning to send it to
> 'gluster-devel@' by Wednesday (24th) if there are no serious comments.
>
> Below is mainly a proposal, and I would like to hear people's thought on
> these.
>
>
>    - Over the years, we have added many test cases to our regression test
>    suite, and currently the testing time stands at ~5hrs for 1 patch. But the
>    current way of '*.t' based regression can't be called as a 'complete' test
>    for the filesystem.
>
>
>    - Correct me if I am wrong, but even when we have to make a release,
>    other than the same set of tests, we don't have much to validate the build.
>
>
> Now considering above points and taking the proposal of 'Good Build' from
> Nigel
> <http://lists.gluster.org/pipermail/gluster-devel/2017-March/052245.html>
> [1],  I am thinking of making below changes to how we look at testing and
> stability.
>
> 'What to test on nightly build':
>
>    - Build verification
>    - Run all the regression as it runs now.
>       - Run CentOS regression
>       - Run NetBSD regression
>       - Run coverity
>    - Run gcov/lcov (for coverage)
>    - Run more tests with currently optional options made as default (like
>    brick-multiplexing etc).
>    - Open up the infra to community contribution, so anyone can write
>    test cases to make sure GlusterFS passes their usecases, everynight.
>       - Should be possible to run a python script, ruby script, or a bash
>       script, need not be in a 'prove' like setup.
>    - <Add more here>
>
> 'master' branch:
>
>    - make the overall regression lightweight.
>       - Run what netbsd tests run now (ie, basic and features in tests).
>       - Don't run netbsd builds, instead add a compilation test on centos
>       32bit machine to keep reminding ourself how many warnings we get.
>    - Make sure 'master' branch is well tested in 'Nightly'.
>    - Let the approach of maintainers and over all project is to promote
>    new changes, instead of being very sceptical about new patches, ideas.
>    - Provide option to run the whole nightly build suit with a given
>    patchset to maintainers, so when in doubt, they can ask for the build to
>    complete before merging. Mostly applies to new feature or some changes
>    which change the way things behave fundamentally.
>
> 'release-x.y' branch:
>
>    - During the release-plan come out with target number of 'coverity'
>    issues, and line coverage % to meet. Also consider number of 'glusto-tests'
>    to pass.
>    - Agree to branch out early (at least 45days, compared to current
>    30days), so we can iron-out the issues caused by the making the 'master'
>    branch process lean.
>
>
>    - Change the voting logic, add more tests now (Example: fall back to
>    current regression suit).
>    - On the first build, run agreed performance tests and compare the
>    numbers with previous versions.
>    - Run NetBSD regression now.
>       - Btw, noticed the latest NetBSD package is for 3.8.9 (done in Jan).
>       - Work with Emmanuel  <manu at netbsd.org> for better options here.
>    - Run nightly on the branch.
>    - Block the release till we meet the initially agreed metrics during
>    the release-plan. (like coverity/glusto-tests, line coverage etc).
>    - On the final build, run agree performance tests and publish the
>    numbers. Make sure it is not regressed.
>
>
> We all agreeing to this is critical, as I saw that (refer mail on 90days
> old patches)
> <http://lists.gluster.org/pipermail/gluster-devel/2017-May/052844.html>[2],
> there were more than 600 patches which were old, and many of these are good
> patches which would make the project better.
>
> Please take time to review the points here and comment if any. Planning to
> raise a ticket to Infra team to validate these changes by June 1st. Lets
> move towards these changes by June 15th if there are not any serious
> concerns.
>
>
>
+1 to bring about more agility in our processes.

-Vijay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/maintainers/attachments/20170523/3c279602/attachment.html>