[Cinder.glusterfs.ci] [Third-party-announce] Cisco ml2 CI is disabled

[Anita Kuno]

On 06/12/2015 06:20 PM, Nikolay Fedotov (nfedotov) wrote:
> Hello Anita
> 
> Thank you for pointing me out to the cacti and IRC log.
> 
> I did not change comment text of the "Cisco ml2 CI". If tests failed the CI should reply :
> " Build failed. For help on isolating this failure, please contact cisco-openstack-neutron-ci at cisco.com. To re-run, post a 'recheck cisco-ml2' comment. "

The belief is it is the recheck syntax in the comment text that is the
problem.

Have you logs for your system activity from yesterday?

Is anyone associated with this account active on irc?

Thank you,
Anita

anteaya on irc


> 
> I see there are guesses that looping might be caused by wrong " comment_filter " (regexp for check text) value. The CI uses:
>         - event: comment-added
>           comment_filter: (?i)^(Patch Set [0-9]+:\n\n)?\s*recheck(( (?:bug|lp)[\s#:]*(\d+))|( no bug))\s*$
>         - event: comment-added
>           comment_filter: (?i)^(Patch Set [0-9]+:\n\n)?\s*recheck cisco-ml2\s*$
> 
> My guess is: Jenkins died at " Jun 11, 2015 8:15:45 PM"  due to error='Cannot allocate memory'. And, maybe, it caused zuul to start looping. But I am not sure.
> 
> Could you advice further steps that help me to get the CI account enabled?
> 
> Thanks.
> 
> -----Original Message-----
> From: Anita Kuno [mailto:anteaya at anteaya.info] 
> Sent: Friday, June 12, 2015 11:13 PM
> To: third-party-announce at lists.openstack.org
> Subject: Re: [Third-party-announce] Cisco ml2 CI is disabled
> 
> On 06/12/2015 04:06 PM, Anita Kuno wrote:
>> On 06/12/2015 03:47 PM, Nikolay Fedotov (nfedotov) wrote:
>>> Hello All!
>>>
>>> I was wondering what kind of looping are you talking about? 
>>>
>>> I see the change https://review.openstack.org/#/c/188221/ has 5  patchset and each Cisco CI replied :
>>> - Cisco Tail-f CI - 4 times
>>> - Cisco N1kv CI - 4 times
>>> - Cisco APIC CI - 4 times
>>> - Cisco ml2 CI - 0 times
>>
>> We had to perform database surgery to remove the hundreds of comments 
>> left by Cisco ml2 CI. Trust us it was looping.
>>
>> We also had to restart gerrit to close the connection as our gerrit 
>> input spiked because of it. See the inbound spike from yesterday?
>> http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=
>> 34&rra_id=all
>> that was you.
>>
>> Your comment text changed yesterday sometime. No other patches were 
>> affected by this.
>>
>> Please confirm your comment text changed yesterday.
>>
>> Thank you,
>> Anita.
> 
> Also here is the log of the incident:
> 
> http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-06-11.log.html#t2015-06-11T20:34:08
> 
>>
>>>
>>> So each CI systems tested every (except the 5th) patchset only once.
>>>
>>> Thanks.
>>>
>>> -----Original Message-----
>>> From: Asselin, Ramy [mailto:ramy.asselin at hp.com]
>>> Sent: Thursday, June 11, 2015 11:55 PM
>>> To: Announcements for third party CI operators.
>>> Subject: Re: [Third-party-announce] Cisco ml2 CI is disabled
>>>
>>> Wow, good call. We should document that and find a way to protect that from happening.
>>>
>>> -----Original Message-----
>>> From: Doug Wiegley [mailto:dougw at a10networks.com]
>>> Sent: Thursday, June 11, 2015 1:49 PM
>>> To: Announcements for third party CI operators.
>>> Subject: Re: [Third-party-announce] Cisco ml2 CI is disabled
>>>
>>> Not sure if this is your issue, but this looks similar to a looping issue that I inflicted on myself, by including my recheck text in my CI comment.
>>>
>>>
>>> "Cisco ml2 CI2:22 PMPatch Set 4:
>>> Build failed. For help on isolating this failure, please contact cisco-openstack-neutron-ci at cisco.com. To re-run, post a 'recheck cisco-ml2' comment.
>>>
>>> * ml2-nexus <http://128.107.233.10:8080/job/ml2-nexus/None> EXCEPTION
>>> * python27 <http://128.107.233.10:8080/job/python27/225> SUCCESS in 6m 06s (non-voting) "
>>>
>>> Thanks,
>>> Doug
>>>
>>>
>>>
>>>
>>>
>>> On 6/11/15, 2:40 PM, "Anita Kuno" <anteaya at anteaya.info> wrote:
>>>
>>>> https://review.openstack.org/#/c/188221/
>>>>
>>>> The patch speaks for itself.
>>>>
>>>> Thank you, Russell Bryant for the alert.
>>>>
>>>> Before this system is enabled we will need to hear the results of 
>>>> your investigation as to why this happened as well as you assurance 
>>>> that it won't happen again.
>>>>
>>>> Thank you,
>>>> Anita.
>>>>
>>>> _______________________________________________
>>>> Third-party-announce mailing list
>>>> Third-party-announce at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-anno
>>>> unc
>>>> e
>>> _______________________________________________
>>> Third-party-announce mailing list
>>> Third-party-announce at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-annou
>>> nce
>>>
>>> _______________________________________________
>>> Third-party-announce mailing list
>>> Third-party-announce at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-annou
>>> nce
>>>
>>> _______________________________________________
>>> Third-party-announce mailing list
>>> Third-party-announce at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-annou
>>> nce
>>>
>>
> 
> 
> _______________________________________________
> Third-party-announce mailing list
> Third-party-announce at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> 
> _______________________________________________
> Third-party-announce mailing list
> Third-party-announce at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> 


_______________________________________________
Third-party-announce mailing list
Third-party-announce at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce