[Gluster-devel] Regression tests and improvement ideas

Thu Jun 18 06:42:06 UTC 2015

On Thu, Jun 18, 2015 at 10:26 AM, Kotresh Hiremath Ravishankar
<khiremat at redhat.com> wrote:
> Hi All,
>
> Another thing to be considered is every patch automatically triggers regressions.
> It is very unlikely that, the very Patch Set 1 submitted would be a merge candidate.
> There would be some or the other review comments to be addressed. Considering that,
> I think it would be a good idea to trigger regression run on getting the first +1 or
> +2 by the reviewers. In that way, we would be saving lot unnecessary regression cycles.
>
> So on patch submission,
>
> 1. Let the smoke tests run
> 2. Review
> 3. Once +1 or +2 is got on the patch, initiate regression.
>
> Any thoughts?

This is exactly what I've proposed in the other mail thread. Do share
your thoughts on that thread.

>
> Thanks and Regards,
> Kotresh H R
>
> ----- Original Message -----
>> From: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
>> To: "Raghavendra Talur" <rtalur at redhat.com>
>> Cc: "Vishwanath Bhat" <vbhat at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>
>> Sent: Wednesday, June 17, 2015 10:32:11 PM
>> Subject: Re: [Gluster-devel] Regression tests and improvement ideas
>>
>>
>>
>>
>>
>> Sent from one plus one
>> On Jun 17, 2015 10:28 PM, "Raghavendra Talur" < rtalur at redhat.com > wrote:
>> >
>> >
>> > On Jun 17, 2015 17:18, Atin Mukherjee < amukherj at redhat.com > wrote:
>> > >
>> > >
>> > >
>> > > On 06/17/2015 04:26 PM, Raghavendra Talur wrote:
>> > > > Hi,
>> > > >
>> > > >
>> > > > MSV Bhat and I had presented in Gluster Design Summit some ideas about
>> > > > improving our testing infrastructure.
>> > > >
>> > > > Here is the link to the slides: http://redhat.slides.com/rtalur/distaf#
>> > > >
>> > > > Here are the same suggestions,
>> > > >
>> > > > 1. *A .t file for a bug*
>> > > > When a community user discovers a bug in Gluster, they contact us over
>> > > > irc or email and eventually end up filling a bug in bugzilla.
>> > > > Many times it so happens that we find a bug which we don't know the
>> > > > fix for OR not a bug in our module and also end up filling a bug in
>> > > > bugzilla.
>> > > >
>> > > > If we could rather write a .t test to reproduce the bug and add it to
>> > > > say /tests/bug/yet-to-be-fixed/ folder in gluster repo it would be
>> > > > more helpful. As part of bug-triage we could try doing the same for
>> > > > bugs
>> > > > filed by community users.
>> > > >
>> > > > *What do we get?*
>> > > >
>> > > > a. very easy for a new developer to pick up that bug and fix it.
>> > > > If .t passes then the bug is fixed.
>> > > >
>> > > > b. The regression on daily patch sets would skip this folder; but on a
>> > > > nightly basis we could run a test on this folder to see if any of these
>> > > > tests got fixed while we were fixing some other tests. Yay!
>> > > Attaching a reproducer in the form of .t might be difficult, specially
>> > > for the race conditions. It might pass pre and post fix as well. So it
>> > > *should not* be a must criteria to have .t file.
>> >
>> > Agreed, it is only a good to have thing. For easy fix and/or easy
>> > reproducible bugs.
>> > > >
>> > > >
>> > > >
>> > > > 2. *New gerrit/review work flow*
>> > > >
>> > > > Our gerrit setup currently has a 2 hour average for regression run.
>> > > > Due to long queue of commits the round about time is around 4-6 hours.
>> > > >
>> > > > Kaushal has proposed on how to reduce round about time more in this
>> > > > thread http://www.spinics.net/lists/gluster-devel/msg15798.html .
>> > > >
>> > > >
>> > > > 3. *Make sure tests can be done in docker and run in parallel*
>> > > >
>> > > > To reduce time for one test run from 2 hours we can look at running
>> > > > tests in parallel. I did a prototype and got test time down to 40 mins
>> > > > on a 16 GB RAM and 4 core VM.
>> > > >
>> > > > Current blocked at :
>> > > > Some of the tests fail in docker while they pass in a VM.
>> > > > Note that it is .t failing, Gluster works fine in docker.
>> > > > Need some help on this. More on this in a mail I will be sending later
>> > > > today at gluster-devel.
>> > > >
>> > > >
>> > > > *what do we get?*
>> > > > Running 4 docker containers on our Laptops itself can reduce time
>> > > > taken by test runs down to 90 mins. Running them on powerful machines,
>> > > > it is down to 40 mins as seen in the prototype.
>> > > How about NetBSD, yesterday Niels point out to me that there is no
>> > > docker service for NetBSD.
>> >
>> > There are two take aways here,
>> > 1. Reducing regression time on Jenkins
>> > 2. Reducing regression time on our laptops.
>> >
>> > For 1 we will still have NETBSD bottleneck. Haven't thought of how to avoid
>> > that.
>> >
>> > At least getting 2 is still a win.
>> I am bit ambitious for (1 && 2) ;)
>> > > >
>> > > >
>> > > > 4. *Test definitions for every .t*
>> > > >
>> > > > May be the time has come to upgrade our test infra to have tests with
>> > > > test definitions. Every .t file could have a corresponding .def file
>> > > > which is
>> > > > A JSON/YAML/XML config
>> > > > Defines the requirements of test
>> > > > Type of volume
>> > > > Special knowledge of brick size required?
>> > > > Which repo source folders should trigger this test
>> > > > Running time
>> > > > Test RUN level
>> > > >
>> > > > *what do we get?*
>> > > > a. Run a partial set of tests on a commit based on git log and test
>> > > > definitions and run complete regression as nightly.
>> > > > b. Order test run based on run times. This combined with fail on first
>> > > > test setting we have, we will fail as early as possible.
>> > > > c. Order tests based on functionality level, which means a mount.t
>> > > > basic
>> > > > test should run before a complex DHT test that makes use of FUSE mount.
>> > > > Again, this will help us to fail as early as possible in failure
>> > > > scenarios.
>> > > > d. With knowledge of type of volume required and number of bricks
>> > > > required, we can re-use volumes that are created for subsequent tests.
>> > > > Even the cleanup() function we have takes time. DiSTAF already has a
>> > > > function equivalent to use_existing_else_create_new.
>> > > >
>> > > >
>> > > > 5. *Testing GFAPI*
>> > > > We don't have a good test framework for gfapi as of today.
>> > > >
>> > > > However, with the recent design proposal at
>> > > > https://docs.google.com/document/d/1yuRLRbdccx_0V0UDAxqWbz4g983q5inuINHgM1YO040/edit?usp=sharing
>> > > >
>> > > >
>> > > > and
>> > > >
>> > > > Craig Cabrey from Facebook developing a set of coreutils using
>> > > > GFAPI as mentioned here
>> > > > http://www.spinics.net/lists/gluster-devel/msg15753.html
>> > > >
>> > > > I guess we have it well covered :)
>> > > >
>> > > >
>> > > > Reviews and suggestions welcome!
>> > > >
>> > > > Thanks,
>> > > > Raghavendra Talur
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > _______________________________________________
>> > > > Gluster-devel mailing list
>> > > > Gluster-devel at gluster.org
>> > > > http://www.gluster.org/mailman/listinfo/gluster-devel
>> > >
>> > > --
>> > > ~Atin
>> > _______________________________________________
>> > Gluster-devel mailing list
>> > Gluster-devel at gluster.org
>> > http://www.gluster.org/mailman/listinfo/gluster-devel
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel