[Gluster-devel] Regression tests and improvement ideas

Thu Jun 18 08:01:29 UTC 2015

On Thu, Jun 18, 2015 at 12:12:06PM +0530, Kaushal M wrote:
> On Thu, Jun 18, 2015 at 10:26 AM, Kotresh Hiremath Ravishankar
> <khiremat at redhat.com> wrote:
> > Hi All,
> >
> > Another thing to be considered is every patch automatically triggers regressions.
> > It is very unlikely that, the very Patch Set 1 submitted would be a merge candidate.
> > There would be some or the other review comments to be addressed. Considering that,
> > I think it would be a good idea to trigger regression run on getting the first +1 or
> > +2 by the reviewers. In that way, we would be saving lot unnecessary regression cycles.
> >
> > So on patch submission,
> >
> > 1. Let the smoke tests run
> > 2. Review
> > 3. Once +1 or +2 is got on the patch, initiate regression.
> >
> > Any thoughts?
> 
> This is exactly what I've proposed in the other mail thread. Do share
> your thoughts on that thread.

Well, we do not need Zuul for that. Jenkins could start the regression
tests after a +1 or +2 code-review, thats one of the standard options
which we could enable now already.

In a Jenkins job, you can find the configuration here:
  - open a job configuration
  - scroll to "Gerrit Trigger"
  - "Trigger on" -> [Add]
  - "Comment added" - "Verdict Category" = "Code Review", Value = +1

Screenshot with details attached.

If this is what we all agree should be done, we can enable that now. We
could even only enable it for NetBSD, so that the queue there can drain
a little.

Niels

> 
> >
> > Thanks and Regards,
> > Kotresh H R
> >
> > ----- Original Message -----
> >> From: "Atin Mukherjee" <atin.mukherjee83 at gmail.com>
> >> To: "Raghavendra Talur" <rtalur at redhat.com>
> >> Cc: "Vishwanath Bhat" <vbhat at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>
> >> Sent: Wednesday, June 17, 2015 10:32:11 PM
> >> Subject: Re: [Gluster-devel] Regression tests and improvement ideas
> >>
> >>
> >>
> >>
> >>
> >> Sent from one plus one
> >> On Jun 17, 2015 10:28 PM, "Raghavendra Talur" < rtalur at redhat.com > wrote:
> >> >
> >> >
> >> > On Jun 17, 2015 17:18, Atin Mukherjee < amukherj at redhat.com > wrote:
> >> > >
> >> > >
> >> > >
> >> > > On 06/17/2015 04:26 PM, Raghavendra Talur wrote:
> >> > > > Hi,
> >> > > >
> >> > > >
> >> > > > MSV Bhat and I had presented in Gluster Design Summit some ideas about
> >> > > > improving our testing infrastructure.
> >> > > >
> >> > > > Here is the link to the slides: http://redhat.slides.com/rtalur/distaf#
> >> > > >
> >> > > > Here are the same suggestions,
> >> > > >
> >> > > > 1. *A .t file for a bug*
> >> > > > When a community user discovers a bug in Gluster, they contact us over
> >> > > > irc or email and eventually end up filling a bug in bugzilla.
> >> > > > Many times it so happens that we find a bug which we don't know the
> >> > > > fix for OR not a bug in our module and also end up filling a bug in
> >> > > > bugzilla.
> >> > > >
> >> > > > If we could rather write a .t test to reproduce the bug and add it to
> >> > > > say /tests/bug/yet-to-be-fixed/ folder in gluster repo it would be
> >> > > > more helpful. As part of bug-triage we could try doing the same for
> >> > > > bugs
> >> > > > filed by community users.
> >> > > >
> >> > > > *What do we get?*
> >> > > >
> >> > > > a. very easy for a new developer to pick up that bug and fix it.
> >> > > > If .t passes then the bug is fixed.
> >> > > >
> >> > > > b. The regression on daily patch sets would skip this folder; but on a
> >> > > > nightly basis we could run a test on this folder to see if any of these
> >> > > > tests got fixed while we were fixing some other tests. Yay!
> >> > > Attaching a reproducer in the form of .t might be difficult, specially
> >> > > for the race conditions. It might pass pre and post fix as well. So it
> >> > > *should not* be a must criteria to have .t file.
> >> >
> >> > Agreed, it is only a good to have thing. For easy fix and/or easy
> >> > reproducible bugs.
> >> > > >
> >> > > >
> >> > > >
> >> > > > 2. *New gerrit/review work flow*
> >> > > >
> >> > > > Our gerrit setup currently has a 2 hour average for regression run.
> >> > > > Due to long queue of commits the round about time is around 4-6 hours.
> >> > > >
> >> > > > Kaushal has proposed on how to reduce round about time more in this
> >> > > > thread http://www.spinics.net/lists/gluster-devel/msg15798.html .
> >> > > >
> >> > > >
> >> > > > 3. *Make sure tests can be done in docker and run in parallel*
> >> > > >
> >> > > > To reduce time for one test run from 2 hours we can look at running
> >> > > > tests in parallel. I did a prototype and got test time down to 40 mins
> >> > > > on a 16 GB RAM and 4 core VM.
> >> > > >
> >> > > > Current blocked at :
> >> > > > Some of the tests fail in docker while they pass in a VM.
> >> > > > Note that it is .t failing, Gluster works fine in docker.
> >> > > > Need some help on this. More on this in a mail I will be sending later
> >> > > > today at gluster-devel.
> >> > > >
> >> > > >
> >> > > > *what do we get?*
> >> > > > Running 4 docker containers on our Laptops itself can reduce time
> >> > > > taken by test runs down to 90 mins. Running them on powerful machines,
> >> > > > it is down to 40 mins as seen in the prototype.
> >> > > How about NetBSD, yesterday Niels point out to me that there is no
> >> > > docker service for NetBSD.
> >> >
> >> > There are two take aways here,
> >> > 1. Reducing regression time on Jenkins
> >> > 2. Reducing regression time on our laptops.
> >> >
> >> > For 1 we will still have NETBSD bottleneck. Haven't thought of how to avoid
> >> > that.
> >> >
> >> > At least getting 2 is still a win.
> >> I am bit ambitious for (1 && 2) ;)
> >> > > >
> >> > > >
> >> > > > 4. *Test definitions for every .t*
> >> > > >
> >> > > > May be the time has come to upgrade our test infra to have tests with
> >> > > > test definitions. Every .t file could have a corresponding .def file
> >> > > > which is
> >> > > > A JSON/YAML/XML config
> >> > > > Defines the requirements of test
> >> > > > Type of volume
> >> > > > Special knowledge of brick size required?
> >> > > > Which repo source folders should trigger this test
> >> > > > Running time
> >> > > > Test RUN level
> >> > > >
> >> > > > *what do we get?*
> >> > > > a. Run a partial set of tests on a commit based on git log and test
> >> > > > definitions and run complete regression as nightly.
> >> > > > b. Order test run based on run times. This combined with fail on first
> >> > > > test setting we have, we will fail as early as possible.
> >> > > > c. Order tests based on functionality level, which means a mount.t
> >> > > > basic
> >> > > > test should run before a complex DHT test that makes use of FUSE mount.
> >> > > > Again, this will help us to fail as early as possible in failure
> >> > > > scenarios.
> >> > > > d. With knowledge of type of volume required and number of bricks
> >> > > > required, we can re-use volumes that are created for subsequent tests.
> >> > > > Even the cleanup() function we have takes time. DiSTAF already has a
> >> > > > function equivalent to use_existing_else_create_new.
> >> > > >
> >> > > >
> >> > > > 5. *Testing GFAPI*
> >> > > > We don't have a good test framework for gfapi as of today.
> >> > > >
> >> > > > However, with the recent design proposal at
> >> > > > https://docs.google.com/document/d/1yuRLRbdccx_0V0UDAxqWbz4g983q5inuINHgM1YO040/edit?usp=sharing
> >> > > >
> >> > > >
> >> > > > and
> >> > > >
> >> > > > Craig Cabrey from Facebook developing a set of coreutils using
> >> > > > GFAPI as mentioned here
> >> > > > http://www.spinics.net/lists/gluster-devel/msg15753.html
> >> > > >
> >> > > > I guess we have it well covered :)
> >> > > >
> >> > > >
> >> > > > Reviews and suggestions welcome!
> >> > > >
> >> > > > Thanks,
> >> > > > Raghavendra Talur
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > >
> >> > > > _______________________________________________
> >> > > > Gluster-devel mailing list
> >> > > > Gluster-devel at gluster.org
> >> > > > http://www.gluster.org/mailman/listinfo/gluster-devel
> >> > >
> >> > > --
> >> > > ~Atin
> >> > _______________________________________________
> >> > Gluster-devel mailing list
> >> > Gluster-devel at gluster.org
> >> > http://www.gluster.org/mailman/listinfo/gluster-devel
> >>
> >> _______________________________________________
> >> Gluster-devel mailing list
> >> Gluster-devel at gluster.org
> >> http://www.gluster.org/mailman/listinfo/gluster-devel
> >>
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jenkins-trigger-on-code-review.png
Type: image/png
Size: 36581 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20150618/9fe68c4e/attachment-0001.png>