<div dir="ltr"><div><div><div>Hi Shyam/Atin,<br><br></div>I have posted the patch[1] for geo-rep test cases failure:<br> tests/00-geo-rep/georep-basic-dr-rsync.t<br> tests/00-geo-rep/georep-basic-dr-tarssh.t<br> tests/00-geo-rep/00-georep-verify-setup.t<br><br></div>Please include patch [1] while triggering tests. <br>The instrumentation patch [2] which was included can be removed.</div><div><br></div><div>[1] <a href="https://review.gluster.org/#/c/glusterfs/+/20704/">https://review.gluster.org/#/c/glusterfs/+/20704/</a><br></div><div>[2] <a href="https://review.gluster.org/#/c/glusterfs/+/20477/">https://review.gluster.org/#/c/glusterfs/+/20477/</a></div><div><br></div><div>Thanks,<br></div><div>Kotresh HR<br></div><br><div><div><div><br><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Aug 10, 2018 at 3:21 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><span class=""><div dir="ltr">On Thu, Aug 9, 2018 at 4:02 PM Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Thu, Aug 9, 2018 at 6:34 AM Shyam Ranganathan <<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Today's patch set 7 [1], included fixes provided till last evening IST,<br>
and its runs can be seen here [2] (yay! we can link to comments in<br>
gerrit now).<br>
<br>
New failures: (added to the spreadsheet)<br>
./tests/bugs/protocol/bug-<wbr>808400-repl.t (core dumped)<br>
./tests/bugs/quick-read/bug-<wbr>846240.t<br>
<br>
Older tests that had not recurred, but failed today: (moved up in the<br>
spreadsheet)<br>
./tests/bugs/replicate/bug-<wbr>1433571-undo-pending-only-on-<wbr>up-bricks.t<br>
./tests/bugs/index/bug-<wbr>1559004-EMLINK-handling.t<br></blockquote><div><br></div><div>The above test is timing out. I had to increase the timeout while adding the .t so that creation of maximum number of links that will max-out in ext4. Will re-check if it is the same issue and get back.<br></div></div></div></blockquote><div><br></div></span><div>This test is timing out with lcov. I bumped up timeout to 30 minutes @ <a href="https://review.gluster.org/#/c/glusterfs/+/20699" target="_blank">https://review.gluster.org/#/<wbr>c/glusterfs/+/20699</a>, I am not happy that this test takes so long, but without this it is difficult to find regression on ext4 which has limits on number of hardlinks in a directory(It took us almost one year after we introduced regression to find this problem when we did introduce regression last time). If there is a way of running this .t once per day and before each release. I will be happy to make it part of that. Let me know.<br></div><div><div class="h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Other issues;<br>
Test ./tests/basic/ec/ec-5-2.t core dumped again<br>
Few geo-rep failures, Kotresh should have more logs to look at with<br>
these runs<br>
Test ./tests/bugs/glusterd/quorum-<wbr>validation.t dumped core again<br>
<br>
Atin/Amar, we may need to merge some of the patches that have proven to<br>
be holding up and fixing issues today, so that we do not leave<br>
everything to the last. Check and move them along or lmk.<br>
<br>
Shyam<br>
<br>
[1] Patch set 7: <a href="https://review.gluster.org/c/glusterfs/+/20637/7" rel="noreferrer" target="_blank">https://review.gluster.org/c/<wbr>glusterfs/+/20637/7</a><br>
[2] Runs against patch set 7 and its status (incomplete as some runs<br>
have not completed):<br>
<a href="https://review.gluster.org/c/glusterfs/+/20637/7#message-37bc68ce6f2157f2947da6fd03b361ab1b0d1a77" rel="noreferrer" target="_blank">https://review.gluster.org/c/<wbr>glusterfs/+/20637/7#message-<wbr>37bc68ce6f2157f2947da6fd03b361<wbr>ab1b0d1a77</a><br>
(also updated in the spreadsheet)<br>
<br>
On 08/07/2018 07:37 PM, Shyam Ranganathan wrote:<br>
> Deserves a new beginning, threads on the other mail have gone deep enough.<br>
> <br>
> NOTE: (5) below needs your attention, rest is just process and data on<br>
> how to find failures.<br>
> <br>
> 1) We are running the tests using the patch [2].<br>
> <br>
> 2) Run details are extracted into a separate sheet in [3] named "Run<br>
> Failures" use a search to find a failing test and the corresponding run<br>
> that it failed in.<br>
> <br>
> 3) Patches that are fixing issues can be found here [1], if you think<br>
> you have a patch out there, that is not in this list, shout out.<br>
> <br>
> 4) If you own up a test case failure, update the spreadsheet [3] with<br>
> your name against the test, and also update other details as needed (as<br>
> comments, as edit rights to the sheet are restricted).<br>
> <br>
> 5) Current test failures<br>
> We still have the following tests failing and some without any RCA or<br>
> attention, (If something is incorrect, write back).<br>
> <br>
> ./tests/bugs/replicate/bug-<wbr>1290965-detect-bitrotten-<wbr>objects.t (needs<br>
> attention)<br>
> ./tests/00-geo-rep/georep-<wbr>basic-dr-tarssh.t (Kotresh)<br>
> ./tests/bugs/glusterd/add-<wbr>brick-and-validate-replicated-<wbr>volume-options.t<br>
> (Atin)<br>
> ./tests/bugs/ec/bug-1236065.t (Ashish)<br>
> ./tests/00-geo-rep/georep-<wbr>basic-dr-rsync.t (Kotresh)<br>
> ./tests/basic/ec/ec-1468261.t (needs attention)<br>
> ./tests/basic/afr/add-brick-<wbr>self-heal.t (needs attention)<br>
> ./tests/basic/afr/granular-<wbr>esh/replace-brick.t (needs attention)<br>
> ./tests/bugs/core/multiplex-<wbr>limit-issue-151.t (needs attention)<br>
> ./tests/bugs/glusterd/<wbr>validating-server-quorum.t (Atin)<br>
> ./tests/bugs/replicate/bug-<wbr>1363721.t (Ravi)<br>
> <br>
> Here are some newer failures, but mostly one-off failures except cores<br>
> in ec-5-2.t. All of the following need attention as these are new.<br>
> <br>
> ./tests/00-geo-rep/00-georep-<wbr>verify-setup.t<br>
> ./tests/basic/afr/gfid-<wbr>mismatch-resolution-with-fav-<wbr>child-policy.t<br>
> ./tests/basic/stats-dump.t<br>
> ./tests/bugs/bug-1110262.t<br>
> ./tests/bugs/glusterd/mgmt-<wbr>handshake-and-volume-sync-<wbr>post-glusterd-restart.t<br>
> ./tests/basic/ec/ec-data-heal.<wbr>t<br>
> ./tests/bugs/replicate/bug-<wbr>1448804-check-quorum-type-<wbr>values.t<br>
> ./tests/bugs/snapshot/bug-<wbr>1482023-snpashot-issue-with-<wbr>other-processes-accessing-<wbr>mounted-path.t<br>
> ./tests/basic/ec/ec-5-2.t<br>
> <br>
> 6) Tests that are addressed or are not occurring anymore are,<br>
> <br>
> ./tests/bugs/glusterd/<wbr>rebalance-operations-in-<wbr>single-node.t<br>
> ./tests/bugs/index/bug-<wbr>1559004-EMLINK-handling.t<br>
> ./tests/bugs/replicate/bug-<wbr>1386188-sbrain-fav-child.t<br>
> ./tests/bugs/replicate/bug-<wbr>1433571-undo-pending-only-on-<wbr>up-bricks.t<br>
> ./tests/bitrot/bug-1373520.t<br>
> ./tests/bugs/distribute/bug-<wbr>1117851.t<br>
> ./tests/bugs/glusterd/quorum-<wbr>validation.t<br>
> ./tests/bugs/distribute/bug-<wbr>1042725.t<br>
> ./tests/bugs/replicate/bug-<wbr>1586020-mark-dirty-for-entry-<wbr>txn-on-quorum-failure.t<br>
> ./tests/bugs/quota/bug-<wbr>1293601.t<br>
> ./tests/bugs/bug-1368312.t<br>
> ./tests/bugs/distribute/bug-<wbr>1122443.t<br>
> ./tests/bugs/core/bug-1432542-<wbr>mpx-restart-crash.t<br>
> <br>
> Shyam (and Atin)<br>
> <br>
> On 08/05/2018 06:24 PM, Shyam Ranganathan wrote:<br>
>> Health on master as of the last nightly run [4] is still the same.<br>
>><br>
>> Potential patches that rectify the situation (as in [1]) are bunched in<br>
>> a patch [2] that Atin and myself have put through several regressions<br>
>> (mux, normal and line coverage) and these have also not passed.<br>
>><br>
>> Till we rectify the situation we are locking down master branch commit<br>
>> rights to the following people, Amar, Atin, Shyam, Vijay.<br>
>><br>
>> The intention is to stabilize master and not add more patches that my<br>
>> destabilize it.<br>
>><br>
>> Test cases that are tracked as failures and need action are present here<br>
>> [3].<br>
>><br>
>> @Nigel, request you to apply the commit rights change as you see this<br>
>> mail and let the list know regarding the same as well.<br>
>><br>
>> Thanks,<br>
>> Shyam<br>
>><br>
>> [1] Patches that address regression failures:<br>
>> <a href="https://review.gluster.org/#/q/starredby:srangana%2540redhat.com" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>q/starredby:srangana%<wbr>2540redhat.com</a><br>
>><br>
>> [2] Bunched up patch against which regressions were run:<br>
>> <a href="https://review.gluster.org/#/c/20637" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/20637</a><br>
>><br>
>> [3] Failing tests list:<br>
>> <a href="https://docs.google.com/spreadsheets/d/1IF9GhpKah4bto19RQLr0y_Kkw26E_-crKALHSaSjZMQ/edit?usp=sharing" rel="noreferrer" target="_blank">https://docs.google.com/<wbr>spreadsheets/d/<wbr>1IF9GhpKah4bto19RQLr0y_Kkw26E_<wbr>-crKALHSaSjZMQ/edit?usp=<wbr>sharing</a><br>
>><br>
>> [4] Nightly run dashboard: <a href="https://build.gluster.org/job/nightly-master/" rel="noreferrer" target="_blank">https://build.gluster.org/job/<wbr>nightly-master/</a><br>
> ______________________________<wbr>_________________<br>
> Gluster-devel mailing list<br>
> <a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
> <a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
> <br>
______________________________<wbr>_________________<br>
maintainers mailing list<br>
<a href="mailto:maintainers@gluster.org" target="_blank">maintainers@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/maintainers" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/maintainers</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="m_3940024385734226497gmail-m_1035026861379571241gmail_signature"><div dir="ltr">Pranith<br></div></div></div>
</blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br clear="all"><br>-- <br><div dir="ltr" class="m_3940024385734226497gmail_signature"><div dir="ltr">Pranith<br></div></div></font></span></div>
<br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</div>