[Cinder.glusterfs.ci] [Third-party-announce] Vedams-HPMSA FCISCSIDriver CI is disabled

[Nikesh Kumar Mahalka]

Hi,
The primary cause of the failures are provided below. We have come up with
the following action plan to resolve these failures.

1. Array virtual disks/volumes were not getting cleaned up after a CI job
run, and were piling up causing some tests to fail in the next CI job run.
We have written scripts for automated array cleanup at the beginning of a
CI job.

2. We were also having intermittent network issues and this was causing
failures while the Jenkins slave node VM was connecting to the git
repository.
===============
20:13:36 [vedams-lenovo-dsvm-tempest-full-iscsi] $ /bin/bash -xe
/tmp/hudson5584133818276563996.sh
20:13:36 + [[ ! -e devstack-gate ]]
20:13:36 + git clone git://git.openstack.org/openstack-infra/devstack-gate
20:13:36 Cloning into 'devstack-gate'...
20:13:37 fatal: unable to connect to git.openstack.org:
20:13:37 git.openstack.org: Name or service not known
20:13:37
20:13:37 Build step 'Execute shell' marked build as failure
===============

We are enhancing the network infrastructure and also planning moving the
servers from our lab to DotHill datacenter with better network
connectivity, so we don't have these failures in the future.

Currently, we are testing with our modifications, and we have stopped
posting CI results to the patch. We will be completing the testing/sandbox
testing within a few days and will start posting the CI results to the
patch.
We are extremely sorry for the inconvenience and hope to have a stable CI
very soon.

Regards
-Nikesh


On Fri, Jul 24, 2015 at 5:13 AM, Mike Perez <thingee at gmail.com> wrote:

> On Thu, Jul 16, 2015 at 7:24 AM, Anita Kuno <anteaya at anteaya.info> wrote:
> > On 07/16/2015 09:06 AM, Nikesh Kumar Mahalka wrote:
> >> Hi Anita,
> >>   We debugged the system logs when the incident occurred and found that
> >> there was an I/O error that occurred on the operating system disk of the
> >> Jenkins Master node.
> >> We have setup a new Jenkins master node that resolves the I/O error
> issue.
> >> I am also providing below a snippet of the system log file that
> indicates
> >> the failure that occurred.
> >> Could you please enable the HPMSA CI account, so that we can complete
> the
> >> sandbox testing and have the HPMSA CI report on the cinder patches after
> >> discussing with cinder developers.
> >>
> >> To prevent this issue from re-occurring, we will pro-actively monitor
> the
> >> operating system logs for any warnings from the disk subsystem. Also, we
> >> will pro-actively run disk checks to make sure the disk is healthy.
> >
> > Thank you for clearly stating your commitment to taking responsibility
> > for your system and paying attention to how it is affecting the
> > community. Thank you.
> >
> > This CI gerrit account is now re-enabled.
>
> I would like to please re-request the CI's to be disabled again. There
> is just way too many false failures in all three in the last 60 runs:
>
> Vedams-HPMSA FCISCSIDriver CI
> Vedams DotHillDriver CI
> Vedams-LenovoStorage FCISCSI CI
>
> http://paste.openstack.org/show/404811/
> http://paste.openstack.org/show/404812/
> http://paste.openstack.org/show/404813/
>
> --
> Mike Perez
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/cinder.glusterfs.ci/attachments/20150724/49d9cde9/attachment.html>
-------------- next part --------------
_______________________________________________
Third-party-announce mailing list
Third-party-announce at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/third-party-announce
Please attend the third party meetings: http://eavesdrop.openstack.org/#Third_Party_Meeting