[Gluster-devel] Regression tests time

[Xavi Hernandez]

Hi Amar,

On 28 Jan 2018 06:50, "Amar Tumballi" <atumball at redhat.com> wrote:

Thanks for this experiment, Xavi!!

I see two proposals here in the thread.

1. Remove unnecessary sleep commands.
2. Try to bring explicit checks, so our tests are more consistent.

I am personally in favor of 1. Lets do this.

About 2, as its already discussed, we may get into issues due to many
outside glusterfs project setups causing much harder problems to debug. Not
sure if we should depend on our 'eventing' framework in such test cases ?
Would that help?


That would be a good way to detect when something can be done. I've not
worked in these lines yet. But this is not the only way. For example, in
the kill_brick command there was a sleep after that to give time glusterd
to be aware of the change. Instead of the sleep, we can directly request
glusterd the state of the brick. If it's down, we are done without needing
to wait unnecessarily. If for some reason it takes more than one second, we
won't fail spuriously because we are directly checking the state. For
extreme cases where something really fails, we can define a bigger timeout,
for example 5 seconds. This way we cover all cases but in the most common
case it will only take some tens or hundreds of milliseconds.

Reducing timeouts have made more evident some races that currently exist in
the code. Till now I've identified a bug in AFR and a couple of races in
RPC code than were causing spurious failures. I still have to identify
another race (probably in RPC also) that is generating unexpected
disconnections (or incorrect reconnections).

Xavi


Regards,
Amar

On Thu, Jan 25, 2018 at 8:07 PM, Xavi Hernandez <jahernan at redhat.com> wrote:

> On Thu, Jan 25, 2018 at 3:03 PM, Jeff Darcy <jeff at pl.atyp.us> wrote:
>
>>
>>
>>
>> On Wed, Jan 24, 2018, at 9:37 AM, Xavi Hernandez wrote:
>>
>> That happens when we use arbitrary delays. If we use an explicit check,
>> it will work on all systems.
>>
>>
>> You're arguing against a position not taken. I'm not expressing
>> opposition to explicit checks. I'm just saying they don't come for free. If
>> you don't believe me, try adding explicit checks in some of the harder
>> cases where we're waiting for something that's subject to OS scheduling
>> delays, or for large numbers of operations to complete. Geo-replication or
>> multiplexing tests should provide some good examples. Adding explicit
>> conditions is the right thing to do in the abstract, but as a practical
>> matter the returns must justify the cost.
>>
>> BTW, some of our longest-running tests are in EC. Do we need all of
>> those, and do they all need to run as long, or could some be
>> eliminated/shortened?
>>
>
> Some tests were already removed some time ago. Anyway, with the changes
> introduced, it takes between 10 and 15 minutes to execute all ec related
> tests from basic/ec and bugs/ec (an average of 16 to 25 seconds per test).
> Before the changes, the same tests were taking between 30 and 60 minutes.
>
> AFR tests have also improved from almost 60 minutes to around 30.
>
>
>> I agree that parallelizing tests is the way to go, but if we reduce the
>> total time to 50%, the parallelized tests will also take 50% less of the
>> time.
>>
>>
>> Taking 50% less time but failing spuriously 1% of the time, or all of the
>> time in some environments, is not a good thing. If you want to add explicit
>> checks that's great, but you also mentioned shortening timeouts and that's
>> much more risky.
>>
>
> If we have a single test that takes 45 minutes (as we currently have in
> some executions: bugs/nfs/bug-1053579.t), parallelization won't help much.
> We need to make this test to run faster.
>
> Some tests that were failing after the changes have revealed errors in the
> test itself or even in the code, so I think it's a good thing. Currently
> I'm investigating what seems a race in the rpc layer during connections
> that causes some tests to fail. This is a real problem that high delays or
> slow machines were hiding. It seems to cause some gluster requests to fail
> spuriously after reconnecting to a brick or glusterd. I'm not 100% sure
> about this yet, but initial analysis seems to indicate that.
>
> Xavi
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel
>



-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180128/ff2b8d4f/attachment.html>