[Gluster-infra] Jenkins and Gerrit issues today

[Nigel Babu]

Hello,

## Highlights
* If you pushed a patch today or did "recheck centos", please do a recheck.
Those jobs were not triggered.
* Please actually verify that the jobs for your patches have started. You
can do that by visiting https://build.gluster.org/job/smoke/ (for smoke) or
https://build.gluster.org/job/centos6-regression/ (for regression) and
searching for your review. Verify that the patchset is correct.

## The Details

This morning I installed critical security updates for Jenkins that needed
a restart of Jenkins. After this restart, it appears that the Gerrit plugin
failed to load because of an XML error in the config file. As far as I
know, this error has always existed, but the newer version of the plugin
became more strict in xml parsing. I noticed this only about an hour so ago
and I've fixed it. Please let me know if there are further problems. Due to
this any jobs that should have been triggered since about 8:30 am this
morning were not triggered. Please manually do a recheck for your patches.

Additionally, Ravi and Nithya pointed me to a problem where Gerrit wasn't
responding. We've noticed this quite often because we've configured Gerrit
to not drop idle connections. This forces us to restart Gerrit when there
are too many long-running idle connections. I've put a timeout of 10 mins
for idle connections. This issue should be sorted.

However, Jenkins does an SSH connection with Gerrit by running `ssh
jenkins at review.gluster.org stream-events`. I'm not sure if this Gerrit
config change will cause a conflict with Jenkins, but we'll see in the next
few hours. None of the documentation explicitly points to a problem.

-- 
nigelb
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-infra/attachments/20170714/418e3ce1/attachment.html>