<div dir="ltr">Both the tests are now marked as bad since there's has been more than one instance where these tests have failed even after fixing the infra problem. Request geo-rep team to take a look at and revive the tests back soon.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 23, 2018 at 2:30 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Mon, Jan 22, 2018 at 5:13 PM, Nigel Babu <span dir="ltr"><<a href="mailto:nigelb@redhat.com" target="_blank">nigelb@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Update: All the nodes that had problems with geo-rep are now fixed. Waiting on the patch to be merged before we switch over to Centos 7. If things go well, we'll replace nodes one by one as soon as we have one green on Centos 7.<br></div></blockquote><div><br></div></span><div>I just noticed we failed again on the geo-rep tests @ <a href="https://build.gluster.org/job/centos6-regression/8604/console" target="_blank">https://build.gluster.org/job/<wbr>centos6-regression/8604/<wbr>console</a> . Nigel reconfirmed that we have all the machines cleaned up. What else could be going wrong here?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class=""><div dir="ltr"></div><div class="gmail_extra"><br><div class="gmail_quote"><span class="m_-7341667262883757415gmail-">On Mon, Jan 22, 2018 at 12:21 PM, Nigel Babu <span dir="ltr"><<a href="mailto:nigelb@redhat.com" target="_blank">nigelb@redhat.com</a>></span> wrote:<br></span><div><div class="m_-7341667262883757415gmail-h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello folks,</div><div><br></div><div><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-"><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-block m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-ltr"><span><span>As you may have noticed, we've had a lot of centos6-regression failures lately. The geo-replication failures are the new ones which particularly concern me. These failures have nothing to do with the test. The tests are exposing a problem in our infrastructure that we've carried around for a long time. Our machines are not clean machines that we automated. We setup automation on machines that were already created. At some point, we loaned machines for debugging. </span></span><span class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-veryhardreadability"><span><span>During this time, developers have </span></span></span><span class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-adverb"><span><span>inadvertently</span></span></span><span class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-veryhardreadability"><span><span> done 'make install' on the system to install onto system paths rather than into /build/install</span></span></span><span><span>. This is what is causing the geo-replication tests to fail. I've tried cleaning the machines up several times with little to no success.</span></span></div></div><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-"><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-block m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-ltr"><span><br></span></div></div><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-"><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-block m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-ltr"><span><span>Last week, we decided to take an aggressive path to fix this problem. We planned to replace all our problematic nodes with new Centos 7 nodes. This exposed more problems. We expected a specific type of machine from Rackspace. These are no longer offered. Thus, our automation fails on some steps. </span></span><span class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-hardreadability"><span><span>I've spent this weekend tweaking our automation so that it works on the new Rackspace machines and I'm down to </span></span></span><span class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-qualifier"><span><span>just</span></span></span><span class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-hardreadability"><span><span> one test failure[1]</span></span></span><span><span>. I have a patch up to fix this failure[2]. As soon as that patch </span></span><span class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-passivevoice"><span><span>is merged</span></span></span><span><span>, we can push forward with Centos7 nodes. In 4.0, we're dropping support for Centos 6, so this decision makes more sense to do sooner than later.</span></span></div></div><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-"><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-block m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-ltr"><span><br></span></div></div><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-"><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-block m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-ltr"><span><span>We'll not be lending machines anymore from production. We'll be creating new nodes which are a snapshots of an existing production node. This machine will </span></span><span class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-passivevoice"><span><span>be destroyed</span></span></span><span><span> after use. This helps prevent this particular problem in the future. This also means that our machine capacity at all times is at 100 with very minimal wastage.</span></span></div></div><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-"><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-block m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-ltr"><span><br></span></div></div><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-"><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-block m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-ltr"><span><span>[1]: <a href="https://build.gluster.org/job/cage-test/184/consoleText" target="_blank">https://build.gluster.org/job/<wbr>cage-test/184/consoleText</a></span></span></div></div><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-"><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-block m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-public-DraftStyleDefault-ltr"><span><span>[2]: <a href="https://review.gluster.org/#/c/19262/" target="_blank">https://review.gluster.org/#/c<wbr>/19262/</a></span></span></div></div></div><span class="m_-7341667262883757415gmail-m_-3441308167537280585HOEnZb"><font color="#888888"><div><div><br>-- <br><div class="m_-7341667262883757415gmail-m_-3441308167537280585m_5000121705177514626gmail-m_-1559655015143039186gmail_signature"><div dir="ltr">nigelb<br></div></div>
</div></div></font></span></div>
</blockquote></div></div></div><span class="m_-7341667262883757415gmail-HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_-7341667262883757415gmail-m_-3441308167537280585gmail_signature"><div dir="ltr">nigelb<br></div></div>
</font></span></div>
<br></span>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br></blockquote></div><br></div></div>
</blockquote></div><br></div>