<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 24, 2018 at 8:36 PM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Tue, Jul 24, 2018 at 8:35 PM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Tue, Jul 24, 2018 at 6:30 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF"><span>
<p><br>
</p>
<br>
<div class="m_7908131115220070365m_5808951623070163661m_-1358384196294352466moz-cite-prefix">On 07/24/2018 02:56 PM, Raghavendra
Gowdappa wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>All,<br>
<br>
</div>
I was trying to debug regression failures on [1] and observed
that split-brain-resolution.t was failing consistently.<br>
<div>
<div>
<div><br>
=========================<br>
TEST 45 (line 88): 0 get_pending_heal_count patchy<br>
./tests/basic/afr/split-brain-<wbr>resolution.t .. 45/45 RESULT
45: 1<br>
./tests/basic/afr/split-brain-<wbr>resolution.t .. Failed 17/45
subtests <br>
<br>
Test Summary Report<br>
-------------------<br>
./tests/basic/afr/split-brain-<wbr>resolution.t (Wstat: 0
Tests: 45 Failed: 17)<br>
Failed tests: 24-26, 28-36, 41-45<br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>On probing deeper, I observed a curious fact - on most
of the failures stat was not served from md-cache, but
instead was wound down to afr which failed stat with EIO
as the file was in split brain. So, I did another test:</div>
<div>* disabled md-cache</div>
<div>* mount glusterfs with attribute-timeout 0 and
entry-timeout 0</div>
<div><br>
</div>
<div>Now the test fails always. So, I think the test relied
on stat requests being absorbed either by kernel attribute
cache or md-cache. When its not happening stats are
reaching afr and resulting in failures of cmds like
getfattr etc. </div>
</div>
</div>
</div>
</blockquote>
<br></span>
This indeed seems to be the case. Is there any way we can avoid the
stat? When a getfattr is performed on the mount, aren't lookup +
getfattr are the only fops that need to be hit in gluster? <br></div></blockquote><div><br></div></span><div>Its a black box to me how kernel decides whether to do lookup or stat. But I guess, if only stat is needed and its not available in cache it would do a stat.</div></div></div></div></blockquote><div><br></div></span><div>Another thing you can do is mounting with a higher value of attribute-timeout. Let us know whether it works.</div></div></div></div></blockquote><div><br></div><div>I tried higher values of attribute-timeout and its not helping. Are there any other similar split brain related tests? Can I mark these tests bad for time being as the md-cache patch has a deadline?</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class=""><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div text="#000000" bgcolor="#FFFFFF">
-Ravi<br>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>
<div>Thoughts?<br>
</div>
<div><br>
</div>
<div>[1] <a href="https://review.gluster.org/#/c/20549/" target="_blank">https://review.gluster.org/#/c<wbr>/20549/</a><br>
</div>
</div>
</div>
</div>
<br>
<fieldset class="m_7908131115220070365m_5808951623070163661m_-1358384196294352466mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Gluster-devel mailing list
<a class="m_7908131115220070365m_5808951623070163661m_-1358384196294352466moz-txt-link-abbreviated" href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a>
<a class="m_7908131115220070365m_5808951623070163661m_-1358384196294352466moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-devel" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-devel</a></pre>
</blockquote>
<br>
</div>
</blockquote></span></div><br></div></div>
</blockquote></span></div><br></div></div>
</blockquote></div><br></div></div>