<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 25, 2018 at 7:17 PM, Deepshikha Khandelwal <span dir="ltr"><<a href="mailto:dkhandel@redhat.com" target="_blank">dkhandel@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">Hello folks,<br>
<br>
>From last few months, I've been working on bringing distributed<br>
regression testing to production. Our regression framework takes about<br>
4+ hours to run on a single machine. To reduce the waiting time,<br>
Facebook contributed a distributed test runner.<br>
<br>
The solution supports the following:<br>
<br>
1) Shares worker pool across different testers.<br>
2) Try failure 3 times on 3 different machines before calling it a failure.<br>
3) Supports running ASAN, Valgrind, ASAN without leaks.<br>
4) Store the failed test logs on a centralized server[1].<br>
<br>
distributed-regression[2] is the Jenkins job for this<br>
distributed-regression testing.<br>
<br>
There are currently a few known issues:<br>
* Not collecting the entire logs (/var/log/glusterfs) from servers.<br></div></div></blockquote><div><br></div><div>If I look at the activities involved with regression failures, this can wait.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
* A few tests fail due to infra-related issues like geo-rep tests.<br></div></div></blockquote><div><br></div><div>Please open bugs for this, so we can track them, and take it to closure.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
* Takes ~80 minutes with 7 distributed servers (targetting 60 minutes)<br></div></div></blockquote><div><br></div><div>Time can change with more tests added, and also please plan to have number of server as 1 to n.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
* We've only tested plain regressions. ASAN and Valgrind are currently untested.<br></div></div></blockquote><div><br></div><div>Great to have it running not 'per patch', but as nightly, or weekly to start with. </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
<br>
Before bringing it into production, we'll run this job nightly and<br>
watch it for a month to debug the other failures.<br>
<br></div></div></blockquote><div><br></div><div>I would say, bring it to production sooner, say 2 weeks, and also plan to have the current regression as is with a special command like 'run regression in-one-machine' in gerrit (or something similar) with voting rights, so we can fall back to this method if something is broken in parallel testing.</div><div><br></div><div>I have seen that regardless of amount of time we put some scripts in testing, the day we move to production, some thing would be broken. So, let that happen earlier than later, so it would help next release branching out. Don't want to be stuck for branching due to infra failures.</div><div><br></div><div>Regards,</div><div>Amar</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">
Please let us know if you find any issues.<br>
<br>
[1] <a href="https://ci-logs.gluster.org" rel="noreferrer" target="_blank">https://ci-logs.gluster.org</a><br>
[2] <a href="https://build.gluster.org/job/distributed-regression" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>distributed-regression</a><br>
<br>
Regards,<br>
Deepshikha Khandelwal<br>
</div></div>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
<br>
<br>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div></div>