<div dir="ltr">Adding maintainers ML here as the concerns raised are very very important.<br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 18, 2018 at 11:47 AM, Kotresh Hiremath Ravishankar <span dir="ltr"><<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div>Hi Nigel,<br><br></div>I debugged the issue and found the root cause. It is indeed setup issue where two gluster binary instances are present on this machine,<br>one at /usr/local/sbin/gluster and one at /build/install/sbin/gluster. Geo-rep is failing with gluster version being mismatch between master<br></div>and slave. It's finding instance "/usr/local/sbin/gluster" on master and finding "/build/install/sbin/gluster" on slave (when run via ssh).<br><br></div></div></div></div></div></div></div></blockquote><div><br></div><div>Thanks for finding this, Kotresh. Very helpful.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div></div>This is scary! In all these kind of machines, we don't even know whether our patches are being tested properly as it might be<br></div>using wrong gluster binary ("/usr/local/sbin/gluster") where regression tests should use "/build/install/sbin/gluster"<br><br></div>May be it is the result of developers taking machines for debugging and pulling there own instance of gluster source and installing<br></div>in normally without using "/opt/qa/build.sh". We need to address these in some way.</div><div><br></div></div></div></blockquote><div><br></div><div>The above is very concerning, and scary for sure. I guess by moving to chunked regression where we don't get machines but only the logs, we may be at better state as every time machine comes up, it will start from a fresh instance.</div><div> </div><div>Everyone using the regression machine till we have new setup, please make sure you cleanup your stuff.</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div></div><div>I have sent the geo-rep patch[1] which throws out more specific error messages in these scenarios.</div><div>[1] <a href="https://review.gluster.org/19224" target="_blank">https://review.gluster.org/<wbr>19224</a><br></div><div><br></div>Thanks,<br></div>Kotresh HR<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jan 17, 2018 at 8:57 PM, Nigel Babu <span dir="ltr"><<a href="mailto:nigelb@redhat.com" target="_blank">nigelb@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I've granted you access to slave23. You should be able to SSH in as <a href="mailto:jenkins@slave23.cloud.gluster.org" target="_blank">jenkins@slave23.cloud.gluster.<wbr>org</a><br><div class="gmail_extra"><div><div class="m_3922535591542031134h5"><br><div class="gmail_quote">On Wed, Jan 17, 2018 at 8:21 PM, Kotresh Hiremath Ravishankar <span dir="ltr"><<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">If it's happening consistently. Give me the machine. I will root cause it. </div><div class="m_3922535591542031134m_-7425945847402253668HOEnZb"><div class="m_3922535591542031134m_-7425945847402253668h5"><div class="gmail_extra"><br><div class="gmail_quote">On 17 Jan 2018 7:33 pm, "Nigel Babu" <<a href="mailto:nigelb@redhat.com" target="_blank">nigelb@redhat.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Kotresh,<br><br></div>I can reliably reproduce the geo-rep failure on a machine where `ssh <a href="mailto:root@127.0.0.1" target="_blank">root@127.0.0.1</a>` works. What are the next steps I can take to debug this? I can also provide you with access if you'd like.<br clear="all"><div><div><br>-- <br><div class="m_3922535591542031134m_-7425945847402253668m_7557570669631658461m_-6409695330468994463gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">nigelb<br></div></div>
</div></div></div>
</blockquote></div></div>
</div></div></blockquote></div><br><br clear="all"><span class="HOEnZb"><font color="#888888"><br></font></span></div></div><span class="HOEnZb"><font color="#888888"><span class="m_3922535591542031134HOEnZb"><font color="#888888">-- <br><div class="m_3922535591542031134m_-7425945847402253668gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">nigelb<br></div></div>
</font></span></font></span></div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_3922535591542031134gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</font></span></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div></div>