[Gluster-infra] Clang-format change postmortem (12th Sept, 2018)
atumball at redhat.com
Wed Sep 12 14:25:47 UTC 2018
of events (in IST)
1725 - Nigel merges Amar’s patch with the .clang-format file
1727 - Nigel lands the .clang-format changes to master as gluster-ant.
Smoke jobs pass at this point)
1746 - Amar notices that some files are missing in the clang patch.
1752 - Nigel lands a new patch with the missing files (all .c files)
1811 - Amar notices compilation issues after landing the .c changes because
it modifes files with the pattern -tmpl.c. Amar starts working on a fix.
1839 - Nigel notices that the Jenkins job for clang-format doesn’t fail
when it’s supposed to fail and goes to fix.
1855 - Clang-format Jenkins job is now fixed.
1906 - Amar’s fixes are merged with manual votes for Smoke and Centos
Regression from the Infra team. At this point, the builds were passing, but
we had voting issues
- We staged the changes on Github on Sept 10th. Given the size of
changes, we we missed that the command used to make the changes only caught
.h files and not .c files. The following is the command in question.
find . -path ./contrib -prune -o -name '*.c' -or -name '*.h' -print |
xargs clang-format -i
- With the changes that we landed, we did run into build bugs 1
<https://review.gluster.org/#/c/21128/> and fixed them. However, we did
not verify that all the files were in fact modified or sync up on the find
- We had a general framework of agreement on the steps to do but we
looked at it as a code change rather than an infrastructure change. There
wasn’t a well defined go/no-go checklist.
- In the middle of this, we had a freebsd-builder enabled that made the
smoke job for the final fix not vote. This needed a manual vote from the
<https://hackmd.io/_zKtoPXUT7mDAyV6OxzZRQ?both#What-Went-Well>What Went Well
- We did reasonably good planning to find potential issues and in fact,
did find some potential issues.
- Nigel and Amar were on hand and available to fix any potential issues
that popped up
- The changes landed at the end of a working day for India the day
before a public holiday. While there was impact, it was much less than a
similar change performed at working hours.
- Template files need to be caught by the clang-format job correctly so
that they are not checked for formatting. Or the name of the file needs to
be changed so that they don’t end with .c or .h.
- In the future, high impact changes need a good process which has at
least an acceptance criteria, go/no-go checklist, and a rollback procedure.
- This work is currently incomplete and the bug3
<https://bugzilla.redhat.com/show_bug.cgi?id=1564149#c39> tracks the
remaining action items.
Thanks Nigel for the postmortem report.
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Gluster-infra