<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 2 August 2018 at 05:46, Shyam Ranganathan <span dir="ltr"><<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Below is a summary of failures over the last 7 days on the nightly<br>
health check jobs. This is one test per line, sorted in descending order<br>
of occurrence (IOW, most frequent failure is on top).<br>
<br>
The list includes spurious failures as well, IOW passed on a retry. This<br>
is because if we do not weed out the spurious errors, failures may<br>
persist and make it difficult to gauge the health of the branch.<br>
<br>
The number at the end of the test line are Jenkins job numbers where<br>
these failed. The job numbers runs as follows,<br>
- <a href="https://build.gluster.org/job/regression-test-burn-in/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-test-burn-in/</a> ID: 4048 - 4053<br>
- <a href="https://build.gluster.org/job/line-coverage/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>line-coverage/</a> ID: 392 - 407<br>
- <a href="https://build.gluster.org/job/regression-test-with-multiplex/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-test-with-<wbr>multiplex/</a> ID: 811<br>
- 817<br>
<br>
So to get to job 4051 (say), use the link<br>
<a href="https://build.gluster.org/job/regression-test-burn-in/4051" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-test-burn-in/4051</a><br>
<br>
Atin has called out some folks for attention to some tests, consider<br>
this a call out to others, if you see a test against your component,<br>
help around root causing and fixing it is needed.<br>
<br>
tests/bugs/core/bug-1432542-<wbr>mpx-restart-crash.t, 4049, 4051, 4052, 405,<br>
404, 403, 396, 392<br>
<br>
tests/00-geo-rep/georep-basic-<wbr>dr-tarssh.t, 811, 814, 817, 4050, 4053<br>
<br>
tests/bugs/bug-1368312.t, 815, 816, 811, 813, 403<br>
<br>
tests/bugs/distribute/bug-<wbr>1122443.t, 4050, 407, 403, 815, 816<br>
<br>
tests/bugs/glusterd/add-brick-<wbr>and-validate-replicated-<wbr>volume-options.t,<br>
814, 816, 817, 812, 815<br>
<br>
tests/bugs/replicate/bug-<wbr>1586020-mark-dirty-for-entry-<wbr>txn-on-quorum-failure.t,<br>
4049, 812, 814, 405, 392<br>
<br>
tests/bitrot/bug-1373520.t, 811, 816, 817, 813<br>
<br>
tests/bugs/ec/bug-1236065.t, 812, 813, 815<br>
<br>
tests/00-geo-rep/georep-basic-<wbr>dr-rsync.t, 813, 4046<br>
<br>
tests/basic/ec/ec-1468261.t, 817, 812<br>
<br>
tests/bugs/glusterd/quorum-<wbr>validation.t, 4049, 407<br>
<br>
tests/bugs/quota/bug-1293601.<wbr>t, 811, 812<br>
<br>
tests/basic/afr/add-brick-<wbr>self-heal.t, 407<br>
<br>
tests/basic/afr/granular-esh/<wbr>replace-brick.t, 392<br>
<br>
tests/bugs/core/multiplex-<wbr>limit-issue-151.t, 405<br>
<br>
tests/bugs/distribute/bug-<wbr>1042725.t, 405<br>
<br>
tests/bugs/distribute/bug-<wbr>1117851.t, 405<br></blockquote><div><br></div><div><div>From the non-lcov vs lcov runs:</div><div><br></div><div>Non-lcov:</div><div><font face="monospace, monospace" size="1"><br></font></div><div><font face="monospace, monospace" size="1">[nbalacha@myserver glusterfs]$ grep TEST mnt-glusterfs-0.log </font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:30:36.930726]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 72 create_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:31:47.649022]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 75 glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://builder104.cloud.gluster.org">builder104.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:31:47.746734]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 77 move_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:31:47.783606]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 78 move_files /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:31:47.842878]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 85 done cat /mnt/glusterfs/0/status_0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:14.849807]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 86 done cat /mnt/glusterfs/1/status_1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:14.872184]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 88 Y force_umount /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:14.900334]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 89 Y force_umount /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:14.929238]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 90 glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://builder104.cloud.gluster.org">builder104.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:15.027094]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 91 check_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:20.268030]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 93 gluster --mode=script --wignore volume stop patchy ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:22.392247]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 94 Stopped volinfo_field patchy Status ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:22.492175]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 96 gluster --mode=script --wignore volume delete patchy ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:25.475566]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 97 ! gluster --mode=script --wignore volume info patchy ++++++++++</font></div><div><br></div><div><br></div><div>Total time for the tests: <b>169</b> seconds</div><div><br></div><div><br></div><div>Lcov:</div><div><br></div><div><font face="monospace, monospace" size="1">[nbalacha@myserver glusterfs]$ grep TEST mnt-glusterfs-0.log </font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:33:05.737012]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 72 create_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:34:29.133045]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 75 glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://builder100.cloud.gluster.org">builder100.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:34:29.257888]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 77 move_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:34:29.306725]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 78 move_files /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:34:29.372790]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 85 done cat /mnt/glusterfs/0/status_0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:12.934406]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 86 done cat /mnt/glusterfs/1/status_1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:12.970541]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 88 Y force_umount /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:13.009152]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 89 Y force_umount /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:13.046645]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 90 glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://builder100.cloud.gluster.org">builder100.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:13.163337]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 91 check_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:19.175564]:++++++++++ G_LOG:./tests/bugs/distribute/bug-1117851.t: TEST: 93 gluster --mode=script --wignore volume stop patchy ++++++++++</font></div><div><br></div><div>Total time: <b>194</b> seconds</div></div><div> </div><div><br></div><div>We need to increase the timeouts for the line cov tests.</div><div><br></div><div>regards,</div><div>Nithya</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
tests/bugs/glusterd/rebalance-<wbr>operations-in-single-node.t, 405<br>
<br>
tests/bugs/index/bug-1559004-<wbr>EMLINK-handling.t, 405<br>
<br>
tests/bugs/replicate/bug-<wbr>1386188-sbrain-fav-child.t, 4048<br>
<br>
tests/bugs/replicate/bug-<wbr>1433571-undo-pending-only-on-<wbr>up-bricks.t, 813 <br>
<br>
<br>
Thanks,<br>
Shyam<br>
<span class="gmail-im gmail-HOEnZb"><br>
<br>
On 07/30/2018 03:21 PM, Shyam Ranganathan wrote:<br>
</span><div class="gmail-HOEnZb"><div class="gmail-h5">> On 07/24/2018 03:12 PM, Shyam Ranganathan wrote:<br>
>> 1) master branch health checks (weekly, till branching)<br>
>> - Expect every Monday a status update on various tests runs<br>
> <br>
> See <a href="https://build.gluster.org/job/nightly-master/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>nightly-master/</a> for a report on<br>
> various nightly and periodic jobs on master.<br>
> <br>
> RED:<br>
> 1. Nightly regression (3/6 failed)<br>
> - Tests that reported failure:<br>
> ./tests/00-geo-rep/georep-<wbr>basic-dr-rsync.t<br>
> ./tests/bugs/core/bug-1432542-<wbr>mpx-restart-crash.t<br>
> ./tests/bugs/replicate/bug-<wbr>1586020-mark-dirty-for-entry-<wbr>txn-on-quorum-failure.t<br>
> ./tests/bugs/distribute/bug-<wbr>1122443.t<br>
> <br>
> - Tests that needed a retry:<br>
> ./tests/00-geo-rep/georep-<wbr>basic-dr-tarssh.t<br>
> ./tests/bugs/glusterd/quorum-<wbr>validation.t<br>
> <br>
> 2. Regression with multiplex (cores and test failures)<br>
> <br>
> 3. line-coverage (cores and test failures)<br>
> - Tests that failed:<br>
> ./tests/bugs/core/bug-1432542-<wbr>mpx-restart-crash.t (patch<br>
> <a href="https://review.gluster.org/20568" rel="noreferrer" target="_blank">https://review.gluster.org/<wbr>20568</a> does not fix the timeout entirely, as<br>
> can be seen in this run,<br>
> <a href="https://build.gluster.org/job/line-coverage/401/consoleFull" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>line-coverage/401/consoleFull</a> )<br>
> <br>
> Calling out to contributors to take a look at various failures, and post<br>
> the same as bugs AND to the lists (so that duplication is avoided) to<br>
> get this to a GREEN status.<br>
> <br>
> GREEN:<br>
> 1. cpp-check<br>
> 2. RPM builds<br>
> <br>
> IGNORE (for now):<br>
> 1. clang scan (@nigel, this job requires clang warnings to be fixed to<br>
> go green, right?)<br>
> <br>
> Shyam<br>
> ______________________________<wbr>_________________<br>
</div></div><span class="gmail-im gmail-HOEnZb">> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
> <a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
> <br>
</span><div class="gmail-HOEnZb"><div class="gmail-h5">______________________________<wbr>_________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org">maintainers@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/maintainers</a><br>
</div></div></blockquote></div><br></div></div>