[Cinder.glusterfs.ci] [Third-party-announce] OpenStackCI2 at hds.com is disabled

Erlon Cruz sombrafam at gmail.com
Tue Jul 14 18:16:23 UTC 2015


On Tue, Jul 14, 2015 at 1:06 PM, Anita Kuno <anteaya at anteaya.info> wrote:

> On 07/14/2015 11:39 AM, Erlon Cruz wrote:
> > Hi Anita,
> >
> > I think I knew what happened, we had a power failure this weekend, and
> the
> > VM that was running the Jenkins master for this CI was automatically
> > booted, if you see on the CI reports all of then where sent on 13/07.
>
> Great are you able to take any steps so that if you have a power failure
> in future that a human needs to evaluate the system prior to it
> reporting on patches.
>
>
Sure, we have done it!


> >
> > Whats is there any criteria for removing CIs?
>
> This gerrit account has been disabled. The CI has not be removed. There
> is a difference.
>
> That was what I meant. Disable.


> The criteria for disabling CIs is meeting infra requirements as outlined
> here:
> http://docs.openstack.org/infra/system-config/third_party.html#requirements
> as well as infra having discretion at disabling a system at any time:
> "The OpenStack Infrastructure team disables mis-behaving third-party ci
> accounts at its discretion." from
>
> http://docs.openstack.org/infra/system-config/third_party.html#permissions-on-your-third-party-system
>
> > I fell this removal was very
> > unfair.
>
> That's fine and I appreciate you sharing your thoughts with me.
>
>
> > I have seen CIs reporting errors for weeks
>
> Have you communicated those errors to anyone? The core reviewers of the
> project? The operators of the CI in question? Shared them at a third
> party meeting?
>
>
Yes, we talked[1] to cinder guys and informed about the problem when it
happened, also informed about the patch that was ready to fix the
problem[2], that was only merged after I asked again.


> Part of the reason why my actions may seem so heavy handed is that my
> vision of CI operators working collaboratively with each other to ensure
> high quality of all systems has not come to pass. Quality as a whole is
> low and folks tend to ignore CI reports as noise, even the useful ones.
> As a survival technique I disable systems when I find out about them.
> This system came to my attention and I diabled it.
>

We worked in a collaboration scheme since the problem came up. Since we
detected the problem, that was not our fault BTW, we are working with
Cinder team to get it fixed. Collaboration is opposed to the action that
was taken here. You see a problem and before informing the people that
could solve the problem or trying to understand the status of the problem
just disable the CI.


>
>
> http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2015-07-13.log.html#t2015-07-13T21:36:48
>
>
>
> > and then, only after
> > warnings to operators in this mailing list, they are removed.
>
> Nothing has been removed. Infra takes care of the gerrit accounts,
> nothing more.
>
> > In our case,
> > it was less than 24h, we got no warnings and got this account disabled.
> > Today, we can't test the CI against the patch that entered (yesterday)
> and
> > will have to do all the process to get the account again. Is it possible
> to
> > re-enable the account so we can continue the process to test the patch
> that
> > should fix the problem?
>
> Let's get some confirmation a power outage in future will have a
> different outcome.
>

I have done this. If you re-enable it, it will run in silent mode until we
assure that it is stable enough to post to gerrit.


>
> Thanks Erlon,
> Anita.
>
> >
> > Thanks,
> > Erlon
> >
> > On Tue, Jul 14, 2015 at 12:06 PM, Anita Kuno <anteaya at anteaya.info>
> wrote:
> >
> >> On 07/14/2015 10:18 AM, Erlon Cruz wrote:
> >>> Hi Anita,
> >>>
> >>> There was a change in an external library (lockutils) that was causing
> >> the
> >>> patches run in our CI to fail. We have reported the problem[1],
> >> submitted a
> >>> fix[2], and shut down the CI while the fix wasn't merged.
> >>
> >> Okay well somehow you shut down your system but it was still leaving
> >> comments on gerrit patches. How did that happen?
> >>
> >> Thanks Erlon,
> >> Anita.
> >>
> >>
> >>> That was about 10
> >>> days ago, and the patch only landed yesterday. We will re-run the above
> >>> tests and submit the successful results.
> >>>
> >>> [1] https://bugs.launchpad.net/cinder/+bug/1470112
> >>> [2] https://review.openstack.org/#/c/197686/
> >>>
> >>> On Mon, Jul 13, 2015 at 7:06 PM, Anita Kuno <anteaya at anteaya.info>
> >> wrote:
> >>>
> >>>> https://review.openstack.org/#/c/199537/
> >>>>
> >>>> https://review.openstack.org/#/c/198853/
> >>>>
> >>>> https://review.openstack.org/#/c/193937/
> >>>>
> >>>> https://review.openstack.org/#/c/201241/
> >>>>
> >>>> This account is posting NOT_REGISTERED on patches.
> >>>>
> >>>> This is an indication that zuul (if you are using zuul) is unable to
> >>>> clearly communicate with Jenkins nodes. This is an internal error and
> >>>> has nothing to do with the patch.
> >>>>
> >>>> Please fix your system and post your incident report with details as a
> >>>> response.
> >>>>
> >>>> Please test your system on ci-sandbox after it is working again.
> >>>>
> >>>> Thank you,
> >>>> Anita.
> >>>>
> >>>> _______________________________________________
> >>>> Third-party-announce mailing list
> >>>> Third-party-announce at lists.openstack.org
> >>>>
> >>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> >>>> Please attend the third party meetings:
> >>>> http://eavesdrop.openstack.org/#Third_Party_Meeting
> >>>>
> >>>
> >>>
> >>>
> >>> _______________________________________________
> >>> Third-party-announce mailing list
> >>> Third-party-announce at lists.openstack.org
> >>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> >>> Please attend the third party meetings:
> >> http://eavesdrop.openstack.org/#Third_Party_Meeting
> >>>
> >>
> >>
> >> _______________________________________________
> >> Third-party-announce mailing list
> >> Third-party-announce at lists.openstack.org
> >>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> >> Please attend the third party meetings:
> >> http://eavesdrop.openstack.org/#Third_Party_Meeting
> >>
> >
> >
> >
> > _______________________________________________
> > Third-party-announce mailing list
> > Third-party-announce at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> > Please attend the third party meetings:
> http://eavesdrop.openstack.org/#Third_Party_Meeting
> >
>
>
> _______________________________________________
> Third-party-announce mailing list
> Third-party-announce at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
> Please attend the third party meetings:
> http://eavesdrop.openstack.org/#Third_Party_Meeting
>


[1]
http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-02.log.html
[2
http://eavesdrop.openstack.org/irclogs/%23openstack-cinder/%23openstack-cinder.2015-07-08.log.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/cinder.glusterfs.ci/attachments/20150714/cff0044a/attachment.html>
-------------- next part --------------
_______________________________________________
Third-party-announce mailing list
Third-party-announce at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
Please attend the third party meetings: http://eavesdrop.openstack.org/#Third_Party_Meeting


More information about the Cinder.glusterfs.ci mailing list