<div dir="ltr"><div><div><div><div><div>I ran a variety of tests today, and what is happening is the RPyC server is taken down and cannot resume connection after restart.<br></div>Glusto maintains a cache of SSH connection, RPyC DeployedServer, and RPyC connection.</div><div>SSH handles it gracefully, but the DeployedServer (the piece that manages the remote RPyC server) is lost along with the RPyC connection that sits on top of it.<br></div><div><br></div><div><div>As the test scripts are written, it is possible to add a g.rpyc_close_deployed_server(host) in the library function either before the reboot (assuming nothing else has to use RPyC before the system comes down) or immediately following the reboot and before the next RPyC call is made. The next RPyC connection made will essentially build a new connection and things will work as expected.<br></div></div><div><br></div>Because the RPyC connection is typically used directly after Glusto sets it up and caches it, it is not straightforward to inject a connection check when direct calls are made as we typically use it.</div><div>conn = g.rpyc_get_connection(...)</div><div>conn.modules.sys.platform <-- this call is directly to the RPyC connection object.<br></div><div><br></div>I am testing addition(s) to Glusto that will help manage the connection in these cases, but closing the DeployedServer, as described above, is the immediate solution.<br></div></div><div><br></div><div>Cheers,</div><div>Jonathan<br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 13, 2018 at 12:07 PM, Jonathan Holloway <span dir="ltr"><<a href="mailto:jhollowa@redhat.com" target="_blank">jhollowa@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div>Hey Nigel,<br><br></div>The RPyC server does not automatically restart nor does it reconnect after the server process is restarted.<br></div>There are a couple of ways to handle it.</div><div>I'll send steps that can be used without a change to Glusto, but right now I'm testing something that can be quickly injected into Glusto to make this seamless.<br></div><div><br></div><div>Cheers,<br></div></div>Jonathan<br><br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jun 13, 2018 at 5:15 AM, Nigel Babu <span dir="ltr"><<a href="mailto:nigelb@redhat.com" target="_blank">nigelb@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Jonathan,</div><div><br></div><div>After a machine reboot, will rpyc reconnect automatically? Or are the communication issues a symptom of a larger problem that you can't restart a client and expect the connection to exist when it comes back online?<br></div></div><div class="gmail_extra"><div><div class="m_-5952037454782774565h5"><br><div class="gmail_quote">On Mon, Jun 11, 2018 at 7:36 PM, Jonathan Holloway <span dir="ltr"><<a href="mailto:jhollowa@redhat.com" target="_blank">jhollowa@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hey Vijay,<br><br></div>In the AFR test run I started on Saturday, it looks like the 039 system had that communication issue we were tracking down on Friday, and it had just been rebooted as part of the test.</div><div>Definitely worth re-running AFR after the fix.<br></div><div><br></div><div>Cheers,</div><div>Jonathan<br></div></div><div class="m_-5952037454782774565m_505255949082149427HOEnZb"><div class="m_-5952037454782774565m_505255949082149427h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 11, 2018 at 6:49 AM, Vijay Bhaskar Reddy Avuthu <span dir="ltr"><<a href="mailto:vavuthu@redhat.com" target="_blank">vavuthu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>I will take a look.<br><br></div>Regards,<br></div>Vijay A<br></div><div class="m_-5952037454782774565m_505255949082149427m_-78204693713898739HOEnZb"><div class="m_-5952037454782774565m_505255949082149427m_-78204693713898739h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 11, 2018 at 5:05 PM, Nigel Babu <span dir="ltr"><<a href="mailto:nigelb@redhat.com" target="_blank">nigelb@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Oh dear. That's a problem. Vijay, I think you wrote the original code? Can you take a look?<br></div><div class="gmail_extra"><div><div class="m_-5952037454782774565m_505255949082149427m_-78204693713898739m_-968949305472805332h5"><br><div class="gmail_quote">On Mon, Jun 11, 2018 at 1:58 PM, Vitalii Koriakov <span dir="ltr"><<a href="mailto:vkoriako@redhat.com" target="_blank">vkoriako@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello all<br>
Noticed such behavior:<br>
<br>
Reboot nodes with the method reboot_nodes_and_wait_to_come_<wbr>online. In case when nodes are not online after timeout - it says that all nodes are online.<br>
So logs are:<br>
<br>
<br>
2018-06-08 18:28:00,210 INFO (are_nodes_online) 172.19.2.122 is offline<br>
2018-06-08 18:28:00,211 INFO (reboot_nodes_and_wait_to_come<wbr>_online) Nodes are offline, Retry after 5 seconds ..... <br>
2018-06-08 18:28:05,216 INFO (reboot_nodes_and_wait_to_come<wbr>_online) All nodes ['172.19.2.86', '172.19.2.126', '172.19.3.113', '172.19.2.122'] are up and running<br>
<br>
So it doesn't check are nodes online after 5 sec and just return that all nodes are online.<br>
<br>
Regards,<br>
Vitalii<br>
______________________________<wbr>_________________<br>
automated-testing mailing list<br>
<a href="mailto:automated-testing@gluster.org" target="_blank">automated-testing@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/automated-testing" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/automated-testing</a><br>
</blockquote></div><br><br clear="all"><br></div></div><span class="m_-5952037454782774565m_505255949082149427m_-78204693713898739m_-968949305472805332HOEnZb"><font color="#888888">-- <br><div class="m_-5952037454782774565m_505255949082149427m_-78204693713898739m_-968949305472805332m_-6622573084053882685gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">nigelb<br></div></div>
</font></span></div>
</blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
automated-testing mailing list<br>
<a href="mailto:automated-testing@gluster.org" target="_blank">automated-testing@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/automated-testing" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/automated-testing</a><br>
<br></blockquote></div><br></div>
</div></div></blockquote></div><br><br clear="all"><br></div></div><span class="m_-5952037454782774565HOEnZb"><font color="#888888">-- <br><div class="m_-5952037454782774565m_505255949082149427gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">nigelb<br></div></div>
</font></span></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>