<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 2 August 2018 at 05:46, Shyam Ranganathan <span dir="ltr"><<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Below is a summary of failures over the last 7 days on the nightly<br>
health check jobs. This is one test per line, sorted in descending order<br>
of occurrence (IOW, most frequent failure is on top).<br>
<br>
The list includes spurious failures as well, IOW passed on a retry. This<br>
is because if we do not weed out the spurious errors, failures may<br>
persist and make it difficult to gauge the health of the branch.<br>
<br>
The number at the end of the test line are Jenkins job numbers where<br>
these failed. The job numbers runs as follows,<br>
- <a href="https://build.gluster.org/job/regression-test-burn-in/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-test-burn-in/</a> ID: 4048 - 4053<br>
- <a href="https://build.gluster.org/job/line-coverage/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>line-coverage/</a> ID: 392 - 407<br>
- <a href="https://build.gluster.org/job/regression-test-with-multiplex/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-test-with-<wbr>multiplex/</a> ID: 811<br>
- 817<br>
<br>
So to get to job 4051 (say), use the link<br>
<a href="https://build.gluster.org/job/regression-test-burn-in/4051" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-test-burn-in/4051</a><br>
<br>
Atin has called out some folks for attention to some tests, consider<br>
this a call out to others, if you see a test against your component,<br>
help around root causing and fixing it is needed.<br>
<br>
tests/bugs/core/bug-1432542-<wbr>mpx-restart-crash.t, 4049, 4051, 4052, 405,<br>
404, 403, 396, 392<br>
<br>
tests/00-geo-rep/georep-basic-<wbr>dr-tarssh.t, 811, 814, 817, 4050, 4053<br>
<br>
tests/bugs/bug-1368312.t, 815, 816, 811, 813, 403<br>
<br>
tests/bugs/distribute/bug-<wbr>1122443.t, 4050, 407, 403, 815, 816<br>
<br>
tests/bugs/glusterd/add-brick-<wbr>and-validate-replicated-<wbr>volume-options.t,<br>
814, 816, 817, 812, 815<br>
<br>
tests/bugs/replicate/bug-<wbr>1586020-mark-dirty-for-entry-<wbr>txn-on-quorum-failure.t,<br>
4049, 812, 814, 405, 392<br>
<br>
tests/bitrot/bug-1373520.t, 811, 816, 817, 813<br>
<br>
tests/bugs/ec/bug-1236065.t, 812, 813, 815<br>
<br>
tests/00-geo-rep/georep-basic-<wbr>dr-rsync.t, 813, 4046<br>
<br>
tests/basic/ec/ec-1468261.t, 817, 812<br>
<br>
tests/bugs/glusterd/quorum-<wbr>validation.t, 4049, 407<br>
<br>
tests/bugs/quota/bug-1293601.<wbr>t, 811, 812<br>
<br>
tests/basic/afr/add-brick-<wbr>self-heal.t, 407<br>
<br>
tests/basic/afr/granular-esh/<wbr>replace-brick.t, 392<br>
<br>
tests/bugs/core/multiplex-<wbr>limit-issue-151.t, 405<br>
<br>
tests/bugs/distribute/bug-<wbr>1042725.t, 405<br></blockquote><div><br></div><div>I think this was caused by a failure to cleanup the mounts from the previous test. It succeeds on retry.</div><div><br></div><div><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);text-decoration-style:initial;text-decoration-color:initial"><font size="1"><span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:10</b> </span>================================================================================
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:10</b> </span>[16:59:12] Running tests in file ./tests/bugs/distribute/bug-1042725.t
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>./tests/bugs/distribute/bug-1042725.t ..
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>1..16
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>Aborting.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>/mnt/nfs/1 could not be deleted, here are the left over items
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>drwxr-xr-x. 2 root root 6 Jul 31 16:59 /d/backends
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>drwxr-xr-x. 2 root root 4096 Jul 31 16:59 /mnt/glusterfs/0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>drwxr-xr-x. 2 root root 4096 Jul 31 16:59 /mnt/glusterfs/1
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>drwxr-xr-x. 2 root root 4096 Jul 31 16:59 /mnt/glusterfs/2
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>drwxr-xr-x. 2 root root 4096 Jul 31 16:59 /mnt/glusterfs/3
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>drwxr-xr-x. 2 root root 4096 Jul 31 16:59 /mnt/nfs/0
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>drwxr-xr-x. 2 root root 4096 Jul 31 16:59 /mnt/nfs/1
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span>Please correct the problem and try again.
<span class="gmail-timestamp" style="box-sizing:border-box"><b style="box-sizing:border-box">16:59:27</b> </span></font></pre><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);text-decoration-style:initial;text-decoration-color:initial"><font size="1"><span class="gmail-timestamp" style="box-sizing:border-box"><br></span></font></pre><pre class="gmail-console-output" style="box-sizing:border-box;white-space:pre-wrap;word-wrap:break-word;margin:0px;color:rgb(51,51,51);text-decoration-style:initial;text-decoration-color:initial"><font face="arial, helvetica, sans-serif"><span class="gmail-timestamp" style="box-sizing:border-box">I don't think there is anything to be done for this one.</span></font></pre> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
tests/bugs/distribute/bug-<wbr>1117851.t, 405<br>
<br>
tests/bugs/glusterd/rebalance-<wbr>operations-in-single-node.t, 405<br>
<br>
tests/bugs/index/bug-1559004-<wbr>EMLINK-handling.t, 405<br>
<br>
tests/bugs/replicate/bug-<wbr>1386188-sbrain-fav-child.t, 4048<br>
<br>
tests/bugs/replicate/bug-<wbr>1433571-undo-pending-only-on-<wbr>up-bricks.t, 813 <br>
<br>
<br>
Thanks,<br>
Shyam<br>
<span class="im HOEnZb"><br>
<br>
On 07/30/2018 03:21 PM, Shyam Ranganathan wrote:<br>
</span><div class="HOEnZb"><div class="h5">> On 07/24/2018 03:12 PM, Shyam Ranganathan wrote:<br>
>> 1) master branch health checks (weekly, till branching)<br>
>> - Expect every Monday a status update on various tests runs<br>
> <br>
> See <a href="https://build.gluster.org/job/nightly-master/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>nightly-master/</a> for a report on<br>
> various nightly and periodic jobs on master.<br>
> <br>
> RED:<br>
> 1. Nightly regression (3/6 failed)<br>
> - Tests that reported failure:<br>
> ./tests/00-geo-rep/georep-<wbr>basic-dr-rsync.t<br>
> ./tests/bugs/core/bug-1432542-<wbr>mpx-restart-crash.t<br>
> ./tests/bugs/replicate/bug-<wbr>1586020-mark-dirty-for-entry-<wbr>txn-on-quorum-failure.t<br>
> ./tests/bugs/distribute/bug-<wbr>1122443.t<br>
> <br>
> - Tests that needed a retry:<br>
> ./tests/00-geo-rep/georep-<wbr>basic-dr-tarssh.t<br>
> ./tests/bugs/glusterd/quorum-<wbr>validation.t<br>
> <br>
> 2. Regression with multiplex (cores and test failures)<br>
> <br>
> 3. line-coverage (cores and test failures)<br>
> - Tests that failed:<br>
> ./tests/bugs/core/bug-1432542-<wbr>mpx-restart-crash.t (patch<br>
> <a href="https://review.gluster.org/20568" rel="noreferrer" target="_blank">https://review.gluster.org/<wbr>20568</a> does not fix the timeout entirely, as<br>
> can be seen in this run,<br>
> <a href="https://build.gluster.org/job/line-coverage/401/consoleFull" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>line-coverage/401/consoleFull</a> )<br>
> <br>
> Calling out to contributors to take a look at various failures, and post<br>
> the same as bugs AND to the lists (so that duplication is avoided) to<br>
> get this to a GREEN status.<br>
> <br>
> GREEN:<br>
> 1. cpp-check<br>
> 2. RPM builds<br>
> <br>
> IGNORE (for now):<br>
> 1. clang scan (@nigel, this job requires clang warnings to be fixed to<br>
> go green, right?)<br>
> <br>
> Shyam<br>
> ______________________________<wbr>_________________<br>
</div></div><span class="im HOEnZb">> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
> <br>
</span><div class="HOEnZb"><div class="h5">______________________________<wbr>_________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org">maintainers@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/maintainers</a><br>
</div></div></blockquote></div><br></div></div>