[Gluster-devel] Release 3.12: Glusto run status

[Shyam Ranganathan]

Nigel, Shwetha,

The latest Glusto run [a] that was started by Nigel, post fixing the 
prior timeout issue, failed (much later though) again.

I took a look at the logs and my analysis is here [b]

@atin, @kaushal, @ppai can you take a look and see if the analysis is 
correct?

In short glusterd has got an error when checking for rebalance stats 
from one of the nodes as:
"Received commit RJT from uuid: 6f9524e6-9f9e-44aa-b2f4-393404adfd9d"

and the rebalance deamon on the node with that UUID is not really ready 
to serve requests when this was called, hence I am assuming this is 
causing the error. But need a once over by one of you folks.

@Shwetha, can we add a further timeout between rebalance start and 
checking the status, just so that we avoid this timing issue on these nodes.

Thanks,
Shyam

[a] glusto run: https://ci.centos.org/view/Gluster/job/gluster_glusto/377/

[b] analysis of the failure: 
https://paste.fedoraproject.org/paste/mk6ynJ0B9AH6H9ncbyru5w
On 08/25/2017 04:29 PM, Shyam Ranganathan wrote:
> Nigel was kind enough to kick off a glusto run on 3.12 head a couple of 
> days back. The status can be seen here [1].
> 
> The run failed, but managed to get past what Glusto does on master (see 
> [2]). Not that this is a consolation, but just stating the fact.
> 
> The run [1] failed at,
> 17:05:57 
> functional/bvt/test_cvt.py::TestGlusterHealSanity_dispersed_glusterfs::test_self_heal_when_io_in_progress 
> FAILED
> 
> The test case failed due to,
> 17:10:28 E       AssertionError: ('Volume %s : All process are not 
> online', 'testvol_dispersed')
> 
> The test case can be seen here [3], and the reason for failure is that 
> Glusto did not wait long enough for the down brick to come up (it waited 
> for 10 seconds, but the brick came up after 12 seconds or within the 
> same second as the test for it being up. The log snippets pointing to 
> this problem are here [4]. In short there was no real bug or issue that 
> caused the failure as yet.
> 
> Glusto as a gating factor for this release was desirable, but having got 
> this far on 3.12 does help.
> 
> @nigel, we could try post increasing the timeout between bringing the 
> brick up to checking if it is up, and try another run, let me know if 
> that works, and what is needed from me to get this going.
> 
> Shyam
> 
> [1] Glusto 3.12 run: 
> https://ci.centos.org/view/Gluster/job/gluster_glusto/365/
> 
> [2] Glusto on master: 
> https://ci.centos.org/view/Gluster/job/gluster_glusto/360/testReport/functional.bvt.test_cvt/ 
> 
> 
> [3] Failed test case: 
> https://ci.centos.org/view/Gluster/job/gluster_glusto/365/testReport/functional.bvt.test_cvt/TestGlusterHealSanity_dispersed_glusterfs/test_self_heal_when_io_in_progress/ 
> 
> 
> [4] Log analysis pointing to the failed check: 
> https://paste.fedoraproject.org/paste/znTPiFLrc2~vsWuoYRToZA
> 
> "Releases are made better together"
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel