[Gluster-devel] Performance Translators' Stability and Usefulness - Regression test outline

[Geoff Kassel]

Hi Anand,
   Thanks for the explanation. 

   I'm sorry, you'll have to endure my criticism for just a little while 
longer, though.

   I've seen your stable/unstable branch tactic before with 1.3 and 1.4/2.x 
and it didn't work then. Any stability gained in 1.3 just seemed to be lost 
again in 1.4/2.x.

   I highly recommend that you don't branch the code again. What would be best 
(although I don't know how commercially viable it is) would be to just freeze 
new features for a short while, and get your existing developers to write the 
unit and integration tests for the code that they know. 

   Or you can follow my plan outlined previously for incremental test case 
development. Make every developer perform this important part of the QA 
process on their new code, as it's written. (With appropriate oversight and 
reviews by the QA team, of course, to prevent under-testing.)

   If you want to earn some trust back in the community, I feel you should 
consider putting whatever you have of the regression tests - incomplete or 
otherwise - into the public repository now.

   This has the bonus that the community can help you complete these tests. 
(With your QA team's approval on each change, of course.) There seems to be a 
bit of enthusiasm on the list at the moment for this - you should exploit 
that. Get us to do some work for you for nothing :)

   You might want to consider getting your QA team to do a code audit. Fix 
what you discover, then audit the resulting patches. (And perhaps prepare to 
do one at some regular interval.)

   I've listed some useful tools in previous messages for automatic analysis 
if a full code audit is likely to prove too expensive to implement.

   However, if OpenBSD's track record is anything to go by, very little beats 
experienced eyeballs on code in terms of issues found before they become 
client-affecting bugs.

Geoff.

On Wed, 8 Jul 2009, you wrote:
> >   I hope my extremely long winded rant here :) has explained adequately
> > what I feel GlusterFS needs to have in a regression testing system.
> >
> > Geoff.
>
> Geoff, and others, we hear you. Loud and clear. We acknowledge that
> the project could have fared better if the QA processes were already
> in place. Regarding the wiki page, which apparently has caused
> confusion about misrepresenting the presence of QA processes, I would
> like to clarify a few points. Even though the reasons involve some
> details about the company (Z Research Inc.) I would still disclose it
> just to clarify to the community that there were no evil intentions to
> fool anyone.
>
> First of all, in Sept 08, we did indeed want to kick in everything
> what the document speaks about. We did make our QA recruitment, and
> that person did start off with this very initiative. And as very first
> line the page speaks, "processes to be followed at", they were in the
> planning phase. That person (also the author of the first version of
> the page) had to quit soon after that for personal reasons. And the
> page was left stale -- yes, we acknowledge this was a mistake. Since
> then we have had some restructuring within (and still recruiting) and
> now, we do have a more dedicated QA team. And in the last month, as
> Shehjar has mentioned earlier in this thread, a unit test and
> regression test framework has indeed been under development, and we
> intend to have the framework and test cases published in a couple of
> months for public review. I was hoping we would not reach a stage to
> discuss things at such degree of detail to clarify the situation.
>
> But it is more important to make clear the actual reasons and events
> to maintain a good relationship with the community. The summary of
> what I intend to convey is that, what has happened (as described
> above), is NOT a justification from the devs, but only to convey that
> nothing was done on purpose as an eyewash, and did not have any
> malicious intent of fooling the community whatsoever -- as alleged. We
> acknowledge that the documentation is out of sync with the codebase,
> including the QA page, but that is only due to shortage of resources
> (than any kind of intentions.) The team here is spread thin and
> everybody plays multiple roles (there is the whole commercial support
> happening under the hood which drains out a lot of our time). We, more
> than anyone acknowledge that the project would be nowhere close to
> where it is, if not for the wonderful support, inputs, debugging and
> help from you guys.
>
> I would like to say that works for bringing about a regression suite
> is actually under progress. We ourselves would have wanted it to be
> out earlier than it is scheduled, but we are limited by the situation
> we are in. There are reasons for the delay (our current QA lead is to
> return from a long leave before the development on the framework
> continues full scale.) And the fact that GlusterFS being this complex
> system does not help in terms for getting help from non full-time
> contributors, or having new recruits be productive from an early
> stage.
>
> There is definitely a concentration within the team towards stability.
> For 2.1 we are bringing about a comprehensive internal resource
> accounting to detect and corner leaks and features to dump internal
> structures of the process for debugging -- all of this apart from the
> testing suite under development. Moving forward we promise to keep the
> release-X.X branches 'stable' and keep the master only "as stable as
> possible". So, people who not want to be on the bleeding edge, please
> switch to the release-2.0 branch for now.
>
> Once again, thank you all for being patient and I hope to wind up this
> thread with no sour feelings around! :)
>
> Avati