[Gluster-devel] Performance Translators' Stability and Usefulness

[Geoff Kassel]

Hi Mickey,
   What you seem to have missed is that we've tried giving constructive 
feedback, and have offered feasible solutions. In my case, over the span of 
years. (Check the archives.) It's been ignored.

   These are not outlandish requests either - requests like the use of 
regression testing, an industry-standard QA method. A request just like 
yours, in other words. Because stability *is* a feature (an aspect of 
resilience, which is how GlusterFS is marketed), and QA is how you achieve 
it.

   I've also gone one better than just advice - I've given up significant 
portions of my limited spare time to audit and patch a not-insignificant 
portion of the GlusterFS code, in order to deal with the stability issues I 
and others were encountering. My patches were ignored, on the grounds that it 
contained otherwise unobtrusive comments which were quite necessary to the 
audit.

   Gordan and I both have problems that the GlusterFS devs have ignored, that 
we have been forced to try to resolve for ourselves. Such issues are 
difficult to resolve without a good idea of what is normal behaviour for the 
low-level code.

   We can't get it from code comments - there aren't any. (Try defending that 
on good programming practice grounds.) The code documentation on the wiki 
lags behind what's current by months. Testing frameworks, on the other hand, 
would supply what we need perfectly, because they need to stay current in 
order to work at all.

   Gordan and I have asked politely for access to the testing frameworks that 
the GlusterFS devs are meant to have, and be using. (That they have, in fact, 
claimed to be using since at least September of last year.)

   However, we have just found out that there are no such unit tests, 
regression tests, or indeed any such testing framework as mentioned in the QA 
process document in existence. But they are, in Shehjar of the GlusterFS 
team's words - 'already working on it'. That's a far cry from how it's 
represented in the QA process wiki page from last year.

   I, at least, feel as if I've been mislead - and I think I've been fairly 
restrained about how I've gone about indicating my dismay at this. 

   (Have I not - despite this dismay - ended most of my emails on this topic 
with just the constructive solutions that you have suggested we should be 
offering?)

   So what exactly is the correct procedure in this case, when you've tried 
for years to act constructively to help a clearly struggling project along, 
but are constantly rebuffed?

   I know what the answer from the history of open source projects in this 
sort of situation is typically - fork the project. But I don't have the spare 
time to look after a project of this size - and it's certainly not in the 
interests of my business to take this on, either.

   So all I can do is kick up a fuss when I find I've been lied to 
(deliberately or otherwise) and ignored.

   Especially when the dev team had the gall to turn around after two 
extremely critical bugs in a row, and say 'everything's fine.'

Geoff.

On Mon, 6 Jul 2009, Mickey Mazarick wrote:
> I have to start with an admonishment for both Gordon and Geoff. Come on
> guys! A dev thread is not the place to rant at developers, but to look
> for solutions. Please keep the complaining to a minimum and look
> constructively for solutions to the problems. I too have had similar
> pains doing gluster upgrades; but understand that when developing a
> product like gluster there's  a stability vs feature balance that a
> developer must decide on.
>
> That being said do you developers or anyone else in the community have a
> good set of regression tests that you use?
>
> We've written our own but we're just using bash to create/delete a bunch
> of files but I don't feel it's very adequate. Something from the gluster
> dev team might be more robust especially if it were a binary that
> performed many reads, writes, updates, and deletes to a random set of
> files. It would be great to have some kind of low level tester we could
> run for a few days before going live with new patches.
>
> Thanks!
> -Mickey Mazarick
>
> Geoff Kassel wrote:
> > Hi Filipe,
> >    I think if the GlusterFS devs actually implement the QA process
> > they've (allegedly) had since September last year, that would go a long
> > way to improving stability and reliability across the whole application.
> >
> >    If the QA process as described on the Wiki was actually implemented,
> > we would most likely not see two critical data corruption bugs in two
> > consecutive major releases, and functionality like DHT and AFR that has
> > been around for years might have a chance of working as described.
> >
> >    As much as I would like to see features like active self-heal and
> > non-stop I/O, perhaps it's time for a GlusterFS feature freeze.
> >
> >    During this freeze they could do a proper human-eyeballs-on-code style
> > code review and get their QA processes and testing frameworks developed
> > properly. In doing so, hopefully provide us with the kind of reliability
> > and stability that we should see in a piece of software with the 2.x tag.
> >
> >    Heck, maybe they could even submit their project to Coverity for
> > analysis, like I've suggested before. It *is* free for open source
> > projects, and the results come highly regarded.
> >
> > Geoff.
> >
> > On Mon, 6 Jul 2009, Filipe Maia wrote:
> >> I strongly agree with both Gordan and Geoff when it comes to stability.
> >> I've been trying to get a simple unify setup (now dht) working to
> >> replace my existing nfs file servers and i'm yet to achieve the
> >> necessary stability.
> >> This kind of feature has long been in GlusterFS and should be stable
> >> enough by now, but according to my experience that is not the case.
> >> I would urge the GlusterFS developers to focus on stability above all
> >> else because that is an absolutely essential quality for a software to
> >> be usable, specially one as critical as a file system.
> >>
> >> Filipe
> >>
> >>
> >> _______________________________________________
> >> Gluster-devel mailing list
> >> Gluster-devel at nongnu.org
> >> http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel