[Gluster-devel] IMPORTANT - Adding further volume types to our smoke tests
Xavier Hernandez
xhernandez at datalab.es
Thu Nov 13 09:38:35 UTC 2014
Hi Jeff,
On 11/12/2014 11:21 PM, Jeff Darcy wrote:
> I'm ambivalent. On the one hand, I think this is an important
> step in the right direction. Sometimes we need to be able to
> run *all* of our existing tests with some feature enabled, not
> just a few feature-specific tests. SSL is an example of this,
> and transport (or other forms of) multi-threading will be as
> well.
I agree, however this seems a lot more complex since the number of
combinations grows exponentially. Maybe it should be discussed to create
another script able to do so (or merge it with regressions).
The intended purpose of this modification is to verify that all volume
types fulfill posix standards. Since most of the causes that can break
posix compliance are located in the xlators that implement the different
volume types, I though that smoke test (that currently executes the
posix compliance test) was the right place.
>
> On the other hand, I'm not sure smoke is the place to do this.
> Smoke is supposed to be a *quick* test to catch *basic* errors
> (e.g. source fails to build) before we devote hours to a full
> regression test. How much does this change throughput on the
> smoke-test queue? Should we be doing this in regression
> instead, or in a third testing tier between the two we have?
To try to minimize the effects of testing more configurations generating
longer runs, I modified the script to test all volume types in parallel.
A simple tests seems to indicate that the test time does not increase
much. However this would need to be tested on the jenkins' slaves to be
sure.
On my test machine, a simple smoke test with a distributed-replicated
volume took 155 seconds. A run with four volume types (distributed with
one brick, distributed with three bricks, replicated and
distributed-replicated) took 227 seconds. This represents an increase of
46%, while we have multiplied by 4 the number of tests.
I tested this on a quite slow machine, so it could be much better in
jenkins' slaves.
>
> My gut feel is that we need to think more about how to run
> a matrix of M tests across N configurations, instead of just
> putting feature/regression tests and configuration tests into
> one big bucket. Or maybe that's a longer-term thing.
I think that trying to parallelize regressions could reduce running time
enormously, however this would need an important change in the testing
infrastructure.
Xavi
More information about the Gluster-devel
mailing list