<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Wed, Jan 24, 2018 at 3:11 PM, Jeff Darcy <span dir="ltr"><<a href="mailto:jeff@pl.atyp.us" target="_blank">jeff@pl.atyp.us</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>
<div><span class=""><div style="font-family:Arial"><br></div>
<div><br></div>
<div><br></div>
<div>On Tue, Jan 23, 2018, at 12:58 PM, Xavi Hernandez wrote:<br></div>
<blockquote type="cite"><div dir="ltr"><div style="font-family:Arial">I've made some experiments [1] with the time that centos regression takes to complete. After some changes the time taken to run a full regression has dropped between 2.5 and 3.5 hours (depending on the run time of 2 tests, see below).<br></div>
<div><br></div>
<div>Basically the changes are related with delays manually introduced in some places (sleeps in test files or even in the code, or delays in timer events). I've changed some sleeps with better ways to detect some condition, and I've left the delays in other places but with reduced time. Probably the used values are not the best ones in all cases, but it highlights that we should seriously consider how we detect things instead of simply waiting for some amount of time (and hope it's enough). The total test time is more than 2 hours less with these changes, so this means that >2 hours of the whole regression time is spent waiting unnecessarily.<br></div>
</div>
</blockquote><div style="font-family:Arial"><br></div>
</span><div style="font-family:Arial">We should definitely try to detect specific conditions instead of just sleeping for a fixed amount of time. That said, sometimes it would take significant additional effort to add a marker for a condition plus code to check for it. We need to be *really* careful about changing timeouts in these cases. It's easy to come up with something that works on one development system and then causes spurious failures for others.</div></div></blockquote><div><br></div><div>That happens when we use arbitrary delays. If we use an explicit check, it will work on all systems. Additionally, using specific checks makes it possible to define bigger timeouts to handle corner cases because in the normal case we'll continue as soon as the check is satisfied, which will be almost always. But if it really fails, on that particular cases it will take some time to detect it, which is fine because this way we allow enough time for "normal" delays.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:Arial"> One of the biggest problems I had to deal with when I implemented multiplexing was these kinds of timing dependencies in tests, and I had to go through it all again when I came to Facebook. While I applaud the effort to reduce single-test times, I believe that parallelizing tests will long-term be a more effective (and definitely safer) route to reducing overall latency.<br></div></div></blockquote><div><br></div><div>I agree that parallelizing tests is the way to go, but if we reduce the total time to 50%, the parallelized tests will also take 50% less of the time.</div><div><br></div><div>Additionally, reducing the time it takes to do each test, is a good way to detect corner cases. If we always sleep in some cases, we could be missing some failures that can happen if there's no sleep (and users can do the same requests than us but without sleeping).</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br></blockquote></div><br></div></div>