<div dir="ltr"><div dir="ltr">On Wed, Apr 10, 2019 at 4:01 PM Atin Mukherjee <<a href="mailto:amukherj@redhat.com">amukherj@redhat.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>And now for last 15 days:</div><div><br></div><div><a href="https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10" target="_blank">https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10</a></div><div><br></div><div>./tests/bitrot/bug-1373520.t 18 ==> Fixed through <a href="https://review.gluster.org/#/c/glusterfs/+/22481/" target="_blank">https://review.gluster.org/#/c/glusterfs/+/22481/</a>, I don't see this failing in brick mux post 5th April<br>./tests/bugs/ec/bug-1236065.t 17 ==> happens only in brick mux, needs analysis.<br></div></div></div></div></div></blockquote><div><br></div><div>I've identified the problem here, but not the cause yet. There's a stale inodelk acquired by a process that is already dead, which causes inodelk requests from self-heal and other processes to block.</div><div><br></div><div>The reason why it seemed to block in random places is that all commands are executed with the working directory pointing to a gluster directory which needs healing after the initial tests. Because of the stale inodelk, when any application tries to open a file in the working directory, it's blocked.</div><div><br></div><div>I'll investigate what causes this.</div><div><br></div><div>Xavi</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>./tests/basic/uss.t 15 ==> happens in both brick mux and non brick mux runs, test just simply times out. Needs urgent analysis.<br>./tests/basic/ec/ec-fix-openfd.t 13 ==> Fixed through <a href="https://review.gluster.org/#/c/22508/" target="_blank">https://review.gluster.org/#/c/22508/</a> , patch merged today. <br>./tests/basic/volfile-sanity.t 8 ==> Some race, though this succeeds in second attempt every time.<br><br>There're plenty more with 5 instances of failure from many tests. We need all maintainers/owners to look through these failures and fix them, we certainly don't want to get into a stage where master is unstable and we have to lock down the merges till all these failures are resolved. So please help.</div><div><br></div><div>(Please note fstat stats show up the retries as failures too which in a way is right)<br></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>[1] captures the test failures report since last 30 days and we'd need volunteers/component owners to see why the number of failures are so high against few tests.</div><div><br></div><div>[1] <a href="https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all" target="_blank">https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all</a><br></div></div></div>
</blockquote></div>
</div></div>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a></blockquote></div></div>