[Gluster-devel] IMPORTANT - Adding further volume types to our smoke tests

[Xavier Hernandez]

Hi Jeff,

On 11/12/2014 11:21 PM, Jeff Darcy wrote:
> I'm ambivalent.  On the one hand, I think this is an important
> step in the right direction.  Sometimes we need to be able to
> run *all* of our existing tests with some feature enabled, not
> just a few feature-specific tests.  SSL is an example of this,
> and transport (or other forms of) multi-threading will be as
> well.

I agree, however this seems a lot more complex since the number of 
combinations grows exponentially. Maybe it should be discussed to create 
another script able to do so (or merge it with regressions).

The intended purpose of this modification is to verify that all volume 
types fulfill posix standards. Since most of the causes that can break 
posix compliance are located in the xlators that implement the different 
volume types, I though that smoke test (that currently executes the 
posix compliance test) was the right place.

>
> On the other hand, I'm not sure smoke is the place to do this.
> Smoke is supposed to be a *quick* test to catch *basic* errors
> (e.g. source fails to build) before we devote hours to a full
> regression test.  How much does this change throughput on the
> smoke-test queue?  Should we be doing this in regression
> instead, or in a third testing tier between the two we have?

To try to minimize the effects of testing more configurations generating 
longer runs, I modified the script to test all volume types in parallel.

A simple tests seems to indicate that the test time does not increase 
much. However this would need to be tested on the jenkins' slaves to be 
sure.

On my test machine, a simple smoke test with a distributed-replicated 
volume took 155 seconds. A run with four volume types (distributed with 
one brick, distributed with three bricks, replicated and 
distributed-replicated) took 227 seconds. This represents an increase of 
46%, while we have multiplied by 4 the number of tests.

I tested this on a quite slow machine, so it could be much better in 
jenkins' slaves.

>
> My gut feel is that we need to think more about how to run
> a matrix of M tests across N configurations, instead of just
> putting feature/regression tests and configuration tests into
> one big bucket.  Or maybe that's a longer-term thing.

I think that trying to parallelize regressions could reduce running time 
enormously, however this would need an important change in the testing 
infrastructure.

Xavi