<div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div dir="ltr">On Wed, Oct 3, 2018 at 11:57 AM Deepshikha Khandelwal <<a href="mailto:dkhandel@redhat.com">dkhandel@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello folks,<br>
<br>
Distributed-regression job[1] is now a part of Gluster's<br>
nightly-master build pipeline. The following are the issues we have<br>
resolved since we started working on this:<br>
<br>
1) Collecting gluster logs from servers.<br>
2) Tests failed due to infra-related issues have been fixed.<br>
3) Time taken to run regression testing reduced to ~50-60 minutes.<br>
<br>
To get time down to 40 minutes needs your help!<br>
<br>
Currently, there is a test that is failing:<br>
<br>
tests/bugs/glusterd/optimized-basic-testcases-in-cluster.t<br>
<br>
This needs fixing first.<br>
<br>
There's a test that takes 14 minutes to complete -<br>
`tests/bugs/index/bug-1559004-EMLINK-handling.t`. A single test taking<br>
14 minutes is not something we can distribute. Can we look at how we<br>
can speed this up[2]? When this test fails, it is re-attempted,<br>
further increasing the time. This happens in the regular<br>
centos7-regression job as well.<br></blockquote><div><br></div><div>I made a change [1] to reduce the amount of time this tests needs. With this change the test completes in about 90 seconds. It would need some reviews from maintainers though.</div><div><br></div><div>Do you want I send a patch with this change alone ?</div><div><br></div><div>Xavi</div><div><br></div><div>[1] <a href="https://review.gluster.org/#/c/glusterfs/+/19254/22/tests/bugs/index/bug-1559004-EMLINK-handling.t">https://review.gluster.org/#/c/glusterfs/+/19254/22/tests/bugs/index/bug-1559004-EMLINK-handling.t</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
If you see any other issues, please file a bug[3].<br>
<br>
[1]: <a href="https://build.gluster.org/job/distributed-regression" rel="noreferrer" target="_blank">https://build.gluster.org/job/distributed-regression</a><br>
[2]: <a href="https://build.gluster.org/job/distributed-regression/264/console" rel="noreferrer" target="_blank">https://build.gluster.org/job/distributed-regression/264/console</a><br>
[3]: <a href="https://bugzilla.redhat.com/enter_bug.cgi?product=glusterfs&component=project-infrastructure" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/enter_bug.cgi?product=glusterfs&component=project-infrastructure</a><br>
<br>
Thanks,<br>
Deepshikha Khandelwal<br>
On Tue, Jun 26, 2018 at 9:02 AM Nigel Babu <<a href="mailto:nigelb@redhat.com" target="_blank">nigelb@redhat.com</a>> wrote:<br>
><br>
><br>
><br>
> On Mon, Jun 25, 2018 at 7:28 PM Amar Tumballi <<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>> wrote:<br>
>><br>
>><br>
>><br>
>>> There are currently a few known issues:<br>
>>> * Not collecting the entire logs (/var/log/glusterfs) from servers.<br>
>><br>
>><br>
>> If I look at the activities involved with regression failures, this can wait.<br>
><br>
><br>
> Well, we can't debug the current failures without having the logs. So this has to be fixed first.<br>
><br>
>><br>
>><br>
>>><br>
>>> * A few tests fail due to infra-related issues like geo-rep tests.<br>
>><br>
>><br>
>> Please open bugs for this, so we can track them, and take it to closure.<br>
><br>
><br>
> These are failing due to infra reasons. Most likely subtle differences in the setup of these nodes vs our normal nodes. We'll only be able to debug them once we get the logs. I know the geo-rep ones are easy to fix. The playbook for setting up geo-rep correctly just didn't make it over to the playbook used for these images.<br>
><br>
>><br>
>><br>
>>><br>
>>> * Takes ~80 minutes with 7 distributed servers (targetting 60 minutes)<br>
>><br>
>><br>
>> Time can change with more tests added, and also please plan to have number of server as 1 to n.<br>
><br>
><br>
> While the n is configurable, however it will be fixed to a single digit number for now. We will need to place *some* limitation somewhere or else we'll end up not being able to control our cloud bills.<br>
><br>
>><br>
>><br>
>>><br>
>>> * We've only tested plain regressions. ASAN and Valgrind are currently untested.<br>
>><br>
>><br>
>> Great to have it running not 'per patch', but as nightly, or weekly to start with.<br>
><br>
><br>
> This is currently not targeted until we phase out current regressions.<br>
><br>
>>><br>
>>><br>
>>> Before bringing it into production, we'll run this job nightly and<br>
>>> watch it for a month to debug the other failures.<br>
>>><br>
>><br>
>> I would say, bring it to production sooner, say 2 weeks, and also plan to have the current regression as is with a special command like 'run regression in-one-machine' in gerrit (or something similar) with voting rights, so we can fall back to this method if something is broken in parallel testing.<br>
>><br>
>> I have seen that regardless of amount of time we put some scripts in testing, the day we move to production, some thing would be broken. So, let that happen earlier than later, so it would help next release branching out. Don't want to be stuck for branching due to infra failures.<br>
><br>
><br>
> Having two regression jobs that can vote is going to cause more confusion than it's worth. There are a couple of intermittent memory issues with the test script that we need to debug and fix before I'm comfortable in making this job a voting job. We've worked around these problems right now. It still pops up now and again. The fact that things break often is not an excuse to prevent avoidable failures. The one month timeline was taken with all these factors into consideration. The 2-week timeline is a no-go at this point.<br>
><br>
> When we are ready to make the switch, we won't be switching 100% of the job. We'll start with a sliding scale so that we can monitor failures and machine creation adequately.<br>
><br>
> --<br>
> nigelb<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div></div></div>