[Gluster-infra] Postmortem for RPM build failures

[Nigel Babu]

Hello,

Some of you may have noticed that we've been particularly plagued with RPM
build failures for the past couple of days. The failure looks like a Java error
from Jenkins[1]. I first noticed the error on Tuesday, 31st January. I narrowed
down the issue with a problem with permissions. Jenkins cleans the workspace at
the start of every run and Jenkins was unable to delete the RPM folder for some
reason. My best guess is that something changed in the rpmbuild or mock package
in the last couple of days, which lead to this. We have not deployed any change
from the infra side which should have caused this.

I immediately ran an Ansible job that deleted all the RPM workspaces so that
fresh ones would work and I added this line at the end of every Jenkins job:

    sudo chown -R jenkins:jenkins ${WORKSPACE}/RPMS

This seemed to work for the moment and I'd moved onto other things. Yesterday,
we saw quite a resurgence of the errors. Atin filed a bug this morning and I've
figured out what I did wrong. The permission change runs only when the job is
successful. Not exactly my best idea :)

I've now fixed it up by making it a post-job script[2] that runs the same command
irrespective of whether the job failed or worked. That should fix up any
problems in the future. Apologies for the inconvenience. As always, please file
a bug when you notice any unexpected Jenkins behavior.

[1]: https://build.gluster.org/job/strfmt_errors/2789/console
[2]: http://git.gluster.org/cgit/build-jobs.git/commit/?id=3b77dbdac288bf21f802be41019e2cd4d4dc3e3c

--
nigelb
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-infra/attachments/20170202/96a04e19/attachment.sig>