<div dir="ltr"><div><div>Ok, Nigel helped me in understanding the trace back time is not something we should look and the right way to dig through this problem is by looking at glusto logs. As per the last rebalance instance from the log I see the following:<br><br>volume rebalance: testvol_distributed-dispersed: success: Rebalance on testvol_distributed-dispersed has been started successfully. Use rebalance status command to check status of the rebalance process.<br>ID: 96c645c7-710c-4c4c-a434-4157cbb75753 <br> <br>2017-08-28 15:13:58,952 INFO (test_expanding_volume_when_io_in_progress) Successfully started rebalance on the volume testvol_distributed-dispersed<br>2017-08-28 15:13:58,952 INFO (test_expanding_volume_when_io_in_progress) Checking Rebalance status<br>2017-08-28 15:13:58,953 INFO (run) <a href="mailto:root@172.19.2.86">root@172.19.2.86</a> (cp): gluster volume rebalance testvol_distributed-dispersed status<br>2017-08-28 15:13:58,953 DEBUG (_get_ssh_connection) Retrieved connection from cache: <a href="mailto:root@172.19.2.86">root@172.19.2.86</a><br>2017-08-28 15:13:58,993 INFO (_log_results) RETCODE (<a href="mailto:root@172.19.2.86">root@172.19.2.86</a>): 1 <br>2017-08-28 15:13:58,994 INFO (_log_results) STDERR (<a href="mailto:root@172.19.2.86">root@172.19.2.86</a>)... <br>volume rebalance: testvol_distributed-dispersed: failed: error <br><br></div>Again here I see the rebalance status failure was logged at 15:13:58,994 where as rebalance start was triggered at 15:13:58,952. <br><br></div>@Shwetha - could you help me in understanding how do we log the rebalance status ret code in glusto log?<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 30, 2017 at 4:07 PM, Atin Mukherjee <span dir="ltr"><<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><pre class="m_4930344345152034227gmail-console-output"><span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> # Start Rebalance
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> <a href="http://g.log.info" target="_blank">g.log.info</a>("Starting Rebalance on the volume")
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> ret, _, _ = rebalance_start(self.mnode, self.volname)
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> self.assertEqual(ret, 0, ("Failed to start rebalance on the volume "
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> "%s", self.volname))
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> <a href="http://g.log.info" target="_blank">g.log.info</a>("Successfully started rebalance on the volume %s",
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> self.volname)
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span>
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> # Check Rebalance status
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> <a href="http://g.log.info" target="_blank">g.log.info</a>("Checking Rebalance status")
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> ret, _, _ = rebalance_status(self.mnode, self.volname)
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span> self.assertEqual(ret, 0, ("Failed to get rebalance status for the "
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span>> "volume %s", self.volname))
<span class="m_4930344345152034227gmail-timestamp"><b>14:15:57</b> </span>E AssertionError: ('Failed to get rebalance status for the volume %s', 'testvol_distributed-<wbr>dispersed')</pre> <br></div>The above is the snip extracted from <a href="https://ci.centos.org/view/Gluster/job/gluster_glusto/377/console" target="_blank">https://ci.centos.org/view/<wbr>Gluster/job/gluster_glusto/<wbr>377/console</a><br><br></div>If we had gone for rebalance status checks multiple times, I should have seen multiple entries of rebalance_status failure or at least a difference in time, isn't it?<br><br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 30, 2017 at 3:39 PM, Shwetha Panduranga <span dir="ltr"><<a href="mailto:spandura@redhat.com" target="_blank">spandura@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div>Case:<br><br></div>1) add-brick when IO is in progress , wait for 30 seconds<br><br></div>2) Trigger rebalance<br><br></div>3) Execute: 'rebalance status' ( there is no time delay b/w 2) and 3) )<br><br></div>4) wait_for_rebalance_to_complete ( This get's the xml output of rebalance status and keep checking for rebalance status to be 'complete' for every 10 seconds uptil 5 minutes. 5 minutes wait time can be passed as parameter )<br><br></div>At every step we check the exit status of the command output. If the exit status is non-zero we fail the test case. <br><span class="m_4930344345152034227HOEnZb"><font color="#888888"><br></font></span></div><span class="m_4930344345152034227HOEnZb"><font color="#888888">-Shwetha</font></span></div><div class="m_4930344345152034227HOEnZb"><div class="m_4930344345152034227h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 30, 2017 at 6:06 AM, Sankarshan Mukhopadhyay <span dir="ltr"><<a href="mailto:sankarshan.mukhopadhyay@gmail.com" target="_blank">sankarshan.mukhopadhyay@gmail<wbr>.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Wed, Aug 30, 2017 at 6:03 AM, Atin Mukherjee <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br>
><br>
> On Wed, 30 Aug 2017 at 00:23, Shwetha Panduranga <<a href="mailto:spandura@redhat.com" target="_blank">spandura@redhat.com</a>><br>
> wrote:<br>
>><br>
>> Hi Shyam, we are already doing it. we wait for rebalance status to be<br>
>> complete. We loop. we keep checking if the status is complete for '20'<br>
>> minutes or so.<br>
><br>
><br>
> Are you saying in this test rebalance status was executed multiple times<br>
> till it succeed? If yes then the test shouldn't have failed. Can I get to<br>
> access the complete set of logs?<br>
<br>
</span>Would you not prefer to look at the specific test under discussion as well?<br>
<span>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
</span><a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br>
</blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-devel</a><br></blockquote></div><br></div>
</div></div></blockquote></div><br></div>