[Cinder.glusterfs.ci] [Third-party-announce] OpenStackCI2 at hds.com is disabled

Fri Jul 17 16:32:42 UTC 2015

On 07/14/2015 02:16 PM, Erlon Cruz wrote:
> On Tue, Jul 14, 2015 at 1:06 PM, Anita Kuno <anteaya at anteaya.info> wrote:
> 
>> On 07/14/2015 11:39 AM, Erlon Cruz wrote:
>>> Hi Anita,
>>>
>>> I think I knew what happened, we had a power failure this weekend, and
>> the
>>> VM that was running the Jenkins master for this CI was automatically
>>> booted, if you see on the CI reports all of then where sent on 13/07.
>>
>> Great are you able to take any steps so that if you have a power failure
>> in future that a human needs to evaluate the system prior to it
>> reporting on patches.
>>
>>
> Sure, we have done it!
> 
> 
>>>
>>> Whats is there any criteria for removing CIs?
>>
>> This gerrit account has been disabled. The CI has not be removed. There
>> is a difference.
>>
>> That was what I meant. Disable.
> 
> 
>> The criteria for disabling CIs is meeting infra requirements as outlined
>> here:
>> http://docs.openstack.org/infra/system-config/third_party.html#requirements
>> as well as infra having discretion at disabling a system at any time:
>> "The OpenStack Infrastructure team disables mis-behaving third-party ci
>> accounts at its discretion." from
>>
>> http://docs.openstack.org/infra/system-config/third_party.html#permissions-on-your-third-party-system
>>
>>> I fell this removal was very
>>> unfair.
>>
>> That's fine and I appreciate you sharing your thoughts with me.
>>
>>
>>> I have seen CIs reporting errors for weeks
>>
>> Have you communicated those errors to anyone? The core reviewers of the
>> project? The operators of the CI in question? Shared them at a third
>> party meeting?
>>
>>
> Yes, we talked[1] to cinder guys and informed about the problem when it
> happened, also informed about the patch that was ready to fix the
> problem[2], that was only merged after I asked again.
> 
> 
>> Part of the reason why my actions may seem so heavy handed is that my
>> vision of CI operators working collaboratively with each other to ensure
>> high quality of all systems has not come to pass. Quality as a whole is
>> low and folks tend to ignore CI reports as noise, even the useful ones.
>> As a survival technique I disable systems when I find out about them.
>> This system came to my attention and I diabled it.
>>
> 
> We worked in a collaboration scheme since the problem came up. Since we
> detected the problem, that was not our fault BTW, we are working with
> Cinder team to get it fixed. Collaboration is opposed to the action that
> was taken here. You see a problem and before informing the people that
> could solve the problem or trying to understand the status of the problem
> just disable the CI.
> 
> 
>>
>>
>> http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-07-13.log.html#t2015-07-13T21:36:48
>>
>>
>>
>>> and then, only after
>>> warnings to operators in this mailing list, they are removed.
>>
>> Nothing has been removed. Infra takes care of the gerrit accounts,
>> nothing more.
>>
>>> In our case,
>>> it was less than 24h, we got no warnings and got this account disabled.
>>> Today, we can't test the CI against the patch that entered (yesterday)
>> and
>>> will have to do all the process to get the account again. Is it possible
>> to
>>> re-enable the account so we can continue the process to test the patch
>> that
>>> should fix the problem?
>>
>> Let's get some confirmation a power outage in future will have a
>> different outcome.
>>
> 
> I have done this. If you re-enable it, it will run in silent mode until we
> assure that it is stable enough to post to gerrit.
> 
> 
>>
>> Thanks Erlon,
>> Anita.
>>
>>>
>>> Thanks,
>>> Erlon
>>>
>>> On Tue, Jul 14, 2015 at 12:06 PM, Anita Kuno <anteaya at anteaya.info>
>> wrote:
>>>
>>>> On 07/14/2015 10:18 AM, Erlon Cruz wrote:
>>>>> Hi Anita,
>>>>>
>>>>> There was a change in an external library (lockutils) that was causing
>>>> the
>>>>> patches run in our CI to fail. We have reported the problem[1],
>>>> submitted a
>>>>> fix[2], and shut down the CI while the fix wasn't merged.
>>>>
>>>> Okay well somehow you shut down your system but it was still leaving
>>>> comments on gerrit patches. How did that happen?
>>>>
>>>> Thanks Erlon,
>>>> Anita.
>>>>
>>>>
>>>>> That was about 10
>>>>> days ago, and the patch only landed yesterday. We will re-run the above
>>>>> tests and submit the successful results.
>>>>>
>>>>> [1] https://bugs.launchpad.net/cinder/+bug/1470112
>>>>> [2] https://review.openstack.org/#/c/197686/
>>>>>
>>>>> On Mon, Jul 13, 2015 at 7:06 PM, Anita Kuno <anteaya at anteaya.info>
>>>> wrote:
>>>>>
>>>>>> https://review.openstack.org/#/c/199537/
>>>>>>
>>>>>> https://review.openstack.org/#/c/198853/
>>>>>>
>>>>>> https://review.openstack.org/#/c/193937/
>>>>>>
>>>>>> https://review.openstack.org/#/c/201241/
>>>>>>
>>>>>> This account is posting NOT_REGISTERED on patches.
>>>>>>
>>>>>> This is an indication that zuul (if you are using zuul) is unable to
>>>>>> clearly communicate with Jenkins nodes. This is an internal error and
>>>>>> has nothing to do with the patch.
>>>>>>
>>>>>> Please fix your system and post your incident report with details as a
>>>>>> response.
>>>>>>
>>>>>> Please test your system on ci-sandbox after it is working again.
>>>>>>
>>>>>> Thank you,
>>>>>> Anita.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Third-party-announce mailing list
>>>>>> Third-party-announce at lists.openstack.org
>>>>>>
>>>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
>>>>>> Please attend the third party meetings:
>>>>>> http://eavesdrop.openstack.org/#Third_Party_Meeting
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Third-party-announce mailing list
>>>>> Third-party-announce at lists.openstack.org
>>>>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
>>>>> Please attend the third party meetings:
>>>> http://eavesdrop.openstack.org/#Third_Party_Meeting
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Third-party-announce mailing list
>>>> Third-party-announce at lists.openstack.org
>>>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
>>>> Please attend the third party meetings:
>>>> http://eavesdrop.openstack.org/#Third_Party_Meeting
>>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Third-party-announce mailing list
>>> Third-party-announce at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
>>> Please attend the third party meetings:
>> http://eavesdrop.openstack.org/#Third_Party_Meeting
>>>
>>
>>
>> _______________________________________________
>> Third-party-announce mailing list
>> Third-party-announce at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
>> Please attend the third party meetings:
>> http://eavesdrop.openstack.org/#Third_Party_Meeting
>>
> 
> 
> [1]
> http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-02.log.html
> [2
> http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-08.log.html
> 
> 
> 
> _______________________________________________
> Third-party-announce mailing list
> Third-party-announce at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> Please attend the third party meetings: http://eavesdrop.openstack.org/#Third_Party_Meeting
> 

This account is now re-enabled.

In future please keep a conversation in the same thread, starting a new
thread which is a part of the conversation is confusing when I go to
look for information.

Reference:
lists.openstack.org/pipermail/third-party-announce/2015-July/000245.htmls

Thank you Erlon,
Anita.

_______________________________________________
Third-party-announce mailing list
Third-party-announce at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
Please attend the third party meetings: http://eavesdrop.openstack.org/#Third_Party_Meeting