<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On 6 August 2018 at 18:03, Nithya Balachandran <span dir="ltr"><<a href="mailto:nbalacha@redhat.com" target="_blank">nbalacha@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="gmail-h5">On 2 August 2018 at 05:46, Shyam Ranganathan <span dir="ltr"><<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Below is a summary of failures over the last 7 days on the nightly<br>
health check jobs. This is one test per line, sorted in descending order<br>
of occurrence (IOW, most frequent failure is on top).<br>
<br>
The list includes spurious failures as well, IOW passed on a retry. This<br>
is because if we do not weed out the spurious errors, failures may<br>
persist and make it difficult to gauge the health of the branch.<br>
<br>
The number at the end of the test line are Jenkins job numbers where<br>
these failed. The job numbers runs as follows,<br>
- <a href="https://build.gluster.org/job/regression-test-burn-in/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-test-burn-in/</a> ID: 4048 - 4053<br>
- <a href="https://build.gluster.org/job/line-coverage/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>line-coverage/</a> ID: 392 - 407<br>
- <a href="https://build.gluster.org/job/regression-test-with-multiplex/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-test-with-multiplex<wbr>/</a> ID: 811<br>
- 817<br>
<br>
So to get to job 4051 (say), use the link<br>
<a href="https://build.gluster.org/job/regression-test-burn-in/4051" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>regression-test-burn-in/4051</a><br>
<br>
Atin has called out some folks for attention to some tests, consider<br>
this a call out to others, if you see a test against your component,<br>
help around root causing and fixing it is needed.<br>
<br>
tests/bugs/core/bug-1432542-mp<wbr>x-restart-crash.t, 4049, 4051, 4052, 405,<br>
404, 403, 396, 392<br>
<br>
tests/00-geo-rep/georep-basic-<wbr>dr-tarssh.t, 811, 814, 817, 4050, 4053<br>
<br>
tests/bugs/bug-1368312.t, 815, 816, 811, 813, 403<br>
<br>
tests/bugs/distribute/bug-1122<wbr>443.t, 4050, 407, 403, 815, 816<br>
<br>
tests/bugs/glusterd/add-brick-<wbr>and-validate-replicated-volume<wbr>-options.t,<br>
814, 816, 817, 812, 815<br>
<br>
tests/bugs/replicate/bug-15860<wbr>20-mark-dirty-for-entry-txn-<wbr>on-quorum-failure.t,<br>
4049, 812, 814, 405, 392<br>
<br>
tests/bitrot/bug-1373520.t, 811, 816, 817, 813<br>
<br>
tests/bugs/ec/bug-1236065.t, 812, 813, 815<br>
<br>
tests/00-geo-rep/georep-basic-<wbr>dr-rsync.t, 813, 4046<br>
<br>
tests/basic/ec/ec-1468261.t, 817, 812<br>
<br>
tests/bugs/glusterd/quorum-val<wbr>idation.t, 4049, 407<br>
<br>
tests/bugs/quota/bug-1293601.t<wbr>, 811, 812<br>
<br>
tests/basic/afr/add-brick-self<wbr>-heal.t, 407<br>
<br>
tests/basic/afr/granular-esh/r<wbr>eplace-brick.t, 392<br>
<br>
tests/bugs/core/multiplex-limi<wbr>t-issue-151.t, 405<br>
<br>
tests/bugs/distribute/bug-1042<wbr>725.t, 405<br>
<br>
tests/bugs/distribute/bug-1117<wbr>851.t, 405<br></blockquote><div><br></div></div></div><div><div>From the non-lcov vs lcov runs:</div><div><br></div><div>Non-lcov:</div><div><font face="monospace, monospace" size="1"><br></font></div><div><font face="monospace, monospace" size="1">[nbalacha@myserver glusterfs]$ grep TEST mnt-glusterfs-0.log </font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:30:36.930726]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 72 create_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:31:47.649022]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 75 glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://builder104.cloud.gluster.org" target="_blank">builder104.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:31:47.746734]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 77 move_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:31:47.783606]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 78 move_files /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:31:47.842878]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 85 done cat /mnt/glusterfs/0/status_0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:14.849807]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 86 done cat /mnt/glusterfs/1/status_1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:14.872184]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 88 Y force_umount /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:14.900334]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 89 Y force_umount /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:14.929238]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 90 glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://builder104.cloud.gluster.org" target="_blank">builder104.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:15.027094]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 91 check_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:20.268030]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 93 gluster --mode=script --wignore volume stop patchy ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:22.392247]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 94 Stopped volinfo_field patchy Status ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:22.492175]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 96 gluster --mode=script --wignore volume delete patchy ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-07-31 16:33:25.475566]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 97 ! gluster --mode=script --wignore volume info patchy ++++++++++</font></div><div><br></div><div><br></div><div>Total time for the tests: <b>169</b> seconds</div><div><br></div><div><br></div><div>Lcov:</div><div><br></div><div><font face="monospace, monospace" size="1">[nbalacha@myserver glusterfs]$ grep TEST mnt-glusterfs-0.log </font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:33:05.737012]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 72 create_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:34:29.133045]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 75 glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://builder100.cloud.gluster.org" target="_blank">builder100.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:34:29.257888]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 77 move_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:34:29.306725]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 78 move_files /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:34:29.372790]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 85 done cat /mnt/glusterfs/0/status_0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:12.934406]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 86 done cat /mnt/glusterfs/1/status_1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:12.970541]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 88 Y force_umount /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:13.009152]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 89 Y force_umount /mnt/glusterfs/1 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:13.046645]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 90 glusterfs --entry-timeout=0 --attribute-timeout=0 -s <a href="http://builder100.cloud.gluster.org" target="_blank">builder100.cloud.gluster.org</a> --volfile-id patchy /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:13.163337]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 91 check_files /mnt/glusterfs/0 ++++++++++</font></div><div><font face="monospace, monospace" size="1">[2018-08-06 08:36:19.175564]:++++++++++ G_LOG:./tests/bugs/distribute/<wbr>bug-1117851.t: TEST: 93 gluster --mode=script --wignore volume stop patchy ++++++++++</font></div><div><br></div><div>Total time: <b>194</b> seconds</div></div><div> </div><div><br></div><div>We need to increase the timeouts for the line cov tests.</div></div></div></div></blockquote><div><br></div><div><br></div><div>Checked a few more regression runs for this test:</div><div><pre class="gmail-console-output" style="box-sizing:border-box;word-wrap:break-word;margin:0px;text-decoration-style:initial;text-decoration-color:initial"><font color="#333333"><span style="white-space:pre-wrap">./tests/bugs/distribute/bug-1117851.t - 179 second
./tests/bugs/distribute/bug-1117851.t - 182 second
./tests/bugs/distribute/bug-1117851.t - 178 second
</span></font><span style="color:rgb(51,51,51);white-space:pre-wrap">
</span></pre><pre class="gmail-console-output" style="box-sizing:border-box;word-wrap:break-word;margin:0px;text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(51,51,51);white-space:pre-wrap"><br></span></pre><pre class="gmail-console-output" style="box-sizing:border-box;word-wrap:break-word;margin:0px;text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(51,51,51);white-space:pre-wrap">Looks like the lcov pushed it over the 200s limit.</span></pre> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>regards,</div><div>Nithya</div><div><div class="gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
tests/bugs/glusterd/rebalance-<wbr>operations-in-single-node.t, 405<br>
<br>
tests/bugs/index/bug-1559004-E<wbr>MLINK-handling.t, 405<br>
<br>
tests/bugs/replicate/bug-13861<wbr>88-sbrain-fav-child.t, 4048<br>
<br>
tests/bugs/replicate/bug-14335<wbr>71-undo-pending-only-on-up-<wbr>bricks.t, 813 <br>
<br>
<br>
Thanks,<br>
Shyam<br>
<span class="gmail-m_3047951223464547004gmail-im gmail-m_3047951223464547004gmail-HOEnZb"><br>
<br>
On 07/30/2018 03:21 PM, Shyam Ranganathan wrote:<br>
</span><div class="gmail-m_3047951223464547004gmail-HOEnZb"><div class="gmail-m_3047951223464547004gmail-h5">> On 07/24/2018 03:12 PM, Shyam Ranganathan wrote:<br>
>> 1) master branch health checks (weekly, till branching)<br>
>> - Expect every Monday a status update on various tests runs<br>
> <br>
> See <a href="https://build.gluster.org/job/nightly-master/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>nightly-master/</a> for a report on<br>
> various nightly and periodic jobs on master.<br>
> <br>
> RED:<br>
> 1. Nightly regression (3/6 failed)<br>
> - Tests that reported failure:<br>
> ./tests/00-geo-rep/georep-basi<wbr>c-dr-rsync.t<br>
> ./tests/bugs/core/bug-1432542-<wbr>mpx-restart-crash.t<br>
> ./tests/bugs/replicate/bug-158<wbr>6020-mark-dirty-for-entry-txn-<wbr>on-quorum-failure.t<br>
> ./tests/bugs/distribute/bug-11<wbr>22443.t<br>
> <br>
> - Tests that needed a retry:<br>
> ./tests/00-geo-rep/georep-basi<wbr>c-dr-tarssh.t<br>
> ./tests/bugs/glusterd/quorum-v<wbr>alidation.t<br>
> <br>
> 2. Regression with multiplex (cores and test failures)<br>
> <br>
> 3. line-coverage (cores and test failures)<br>
> - Tests that failed:<br>
> ./tests/bugs/core/bug-1432542-<wbr>mpx-restart-crash.t (patch<br>
> <a href="https://review.gluster.org/20568" rel="noreferrer" target="_blank">https://review.gluster.org/205<wbr>68</a> does not fix the timeout entirely, as<br>
> can be seen in this run,<br>
> <a href="https://build.gluster.org/job/line-coverage/401/consoleFull" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>line-coverage/401/consoleFull</a> )<br>
> <br>
> Calling out to contributors to take a look at various failures, and post<br>
> the same as bugs AND to the lists (so that duplication is avoided) to<br>
> get this to a GREEN status.<br>
> <br>
> GREEN:<br>
> 1. cpp-check<br>
> 2. RPM builds<br>
> <br>
> IGNORE (for now):<br>
> 1. clang scan (@nigel, this job requires clang warnings to be fixed to<br>
> go green, right?)<br>
> <br>
> Shyam<br>
> ______________________________<wbr>_________________<br>
</div></div><span class="gmail-m_3047951223464547004gmail-im gmail-m_3047951223464547004gmail-HOEnZb">> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> <a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-devel</a><br>
> <br>
</span><div class="gmail-m_3047951223464547004gmail-HOEnZb"><div class="gmail-m_3047951223464547004gmail-h5">______________________________<wbr>_________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org" target="_blank">maintainers@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/maintainers</a><br>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div>