<div><br></div><div><br><div class="gmail_quote"><div dir="ltr">On Wed, 12 Sep 2018 at 19:56, Amar Tumballi <<a href="mailto:atumball@redhat.com">atumball@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><h2 class="m_-495277073226778567gmail-part" id="m_-495277073226778567gmail-People-Involved" style="box-sizing:border-box;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-weight:600;line-height:1.25;color:rgb(51,51,51);margin-top:24px;margin-bottom:16px;font-size:1.5em;padding-bottom:0.3em;border-bottom:1px solid rgb(238,238,238);letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial">People Involved</h2><ul class="m_-495277073226778567gmail-part" style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;padding-left:2em;color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16px;letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial"><li class="m_-495277073226778567gmail-" style="box-sizing:border-box">Nigel</li><li class="m_-495277073226778567gmail-" style="box-sizing:border-box;margin-top:0.25em">Amar</li></ul><h2 class="m_-495277073226778567gmail-part" id="m_-495277073226778567gmail-Timeline-of-events-in-IST" style="box-sizing:border-box;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-weight:600;line-height:1.25;color:rgb(51,51,51);margin-top:24px;margin-bottom:16px;font-size:1.5em;padding-bottom:0.3em;border-bottom:1px solid rgb(238,238,238);letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial"><a class="m_-495277073226778567gmail-anchor m_-495277073226778567gmail-hidden-xs" href="https://hackmd.io/_zKtoPXUT7mDAyV6OxzZRQ?both#Timeline-of-events-in-IST" title="Timeline-of-events-in-IST" style="box-sizing:border-box;background-color:transparent;color:rgb(51,122,183);text-decoration:none;float:left;padding-right:4px;line-height:1" target="_blank"><span class="m_-495277073226778567gmail-octicon m_-495277073226778567gmail-octicon-link" style="box-sizing:border-box;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:1;font-family:octicons;display:inline-block;text-decoration:none;color:rgb(0,0,0);vertical-align:middle"></span></a>Timeline of events (in IST)</h2><p class="m_-495277073226778567gmail-part" style="box-sizing:border-box;margin:0px 0px 16px;color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16px;letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial">1725 - Nigel merges Amar’s patch with the .clang-format file<br style="box-sizing:border-box">1727 - Nigel lands the .clang-format changes to master as gluster-ant. Smoke jobs pass at this point)<br style="box-sizing:border-box">1746 - Amar notices that some files are missing in the clang patch.<br style="box-sizing:border-box">1752 - Nigel lands a new patch with the missing files (all .c files)<br style="box-sizing:border-box">1811 - Amar notices compilation issues after landing the .c changes because it modifes files with the pattern<span> </span><code style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",monospace;font-size:13.6px;padding:0.2em 0px;background-color:rgba(0,0,0,0.04);border-radius:3px;margin:0px;color:inherit">-tmpl.c</code>. Amar starts working on a fix.<br style="box-sizing:border-box">1839 - Nigel notices that the Jenkins job for clang-format doesn’t fail when it’s supposed to fail and goes to fix.<br style="box-sizing:border-box">1855 - Clang-format Jenkins job is now fixed.<br style="box-sizing:border-box">1906 - Amar’s fixes are merged with manual votes for Smoke and Centos Regression from the Infra team. At this point, the builds were passing, but we had voting issues</p><h2 class="m_-495277073226778567gmail-part" id="m_-495277073226778567gmail-What-Went-Wrong" style="box-sizing:border-box;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-weight:600;line-height:1.25;color:rgb(51,51,51);margin-top:24px;margin-bottom:16px;font-size:1.5em;padding-bottom:0.3em;border-bottom:1px solid rgb(238,238,238);letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial"><a class="m_-495277073226778567gmail-anchor m_-495277073226778567gmail-hidden-xs" href="https://hackmd.io/_zKtoPXUT7mDAyV6OxzZRQ?both#What-Went-Wrong" title="What-Went-Wrong" style="box-sizing:border-box;background-color:transparent;color:rgb(51,122,183);text-decoration:none;float:left;padding-right:4px;line-height:1" target="_blank"><span class="m_-495277073226778567gmail-octicon m_-495277073226778567gmail-octicon-link" style="box-sizing:border-box;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:1;font-family:octicons;display:inline-block;text-decoration:none;color:rgb(0,0,0);vertical-align:middle"></span></a>What Went Wrong</h2><ul class="m_-495277073226778567gmail-part" style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;padding-left:2em;color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16px;letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial"><li class="m_-495277073226778567gmail-" style="box-sizing:border-box">We staged the changes on Github on Sept 10th. Given the size of changes, we we missed that the command used to make the changes only caught<span> </span><code style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",monospace;font-size:13.6px;padding:0.2em 0px;background-color:rgba(0,0,0,0.04);border-radius:3px;margin:0px;color:inherit">.h</code><span> </span>files and not<span> </span><code style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",monospace;font-size:13.6px;padding:0.2em 0px;background-color:rgba(0,0,0,0.04);border-radius:3px;margin:0px;color:inherit">.c</code><span> </span>files. The following is the command in question.<br style="box-sizing:border-box"><code style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",monospace;font-size:13.6px;padding:0.2em 0px;background-color:rgba(0,0,0,0.04);border-radius:3px;margin:0px;color:inherit">find . -path ./contrib -prune -o -name '*.c' -or -name '*.h' -print | xargs clang-format -i</code></li><li class="m_-495277073226778567gmail-" style="box-sizing:border-box;margin-top:0.25em">With the changes that we landed, we did run into build bugs<span> </span><a href="https://review.gluster.org/#/c/21130/" style="box-sizing:border-box;background-color:transparent;color:rgb(51,122,183);text-decoration:none" target="_blank">1</a>,<span> </span><a href="https://review.gluster.org/#/c/21128/" style="box-sizing:border-box;background-color:transparent;color:rgb(51,122,183);text-decoration:none" target="_blank">2</a><span> </span>and fixed them. However, we did not verify that all the files were in fact modified or sync up on the find command.</li><li class="m_-495277073226778567gmail-" style="box-sizing:border-box;margin-top:0.25em">We had a general framework of agreement on the steps to do but we looked at it as a code change rather than an infrastructure change. There wasn’t a well defined go/no-go checklist.</li></ul></div></blockquote><div dir="auto"><br></div><div dir="auto">Thank you for writing this up and sharing. The “infrastructure piece” part is a great realisation.</div><div dir="auto"><br></div><div dir="auto">Can we start to use “retrospective” instead of post mortem?</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><ul class="m_-495277073226778567gmail-part" style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;padding-left:2em;color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16px;letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial"><li class="m_-495277073226778567gmail-" style="box-sizing:border-box;margin-top:0.25em"></li><li class="m_-495277073226778567gmail-" style="box-sizing:border-box;margin-top:0.25em">In the middle of this, we had a freebsd-builder enabled that made the smoke job for the final fix not vote. This needed a manual vote from the Infra team.</li></ul><h2 class="m_-495277073226778567gmail-part" id="m_-495277073226778567gmail-What-Went-Well" style="box-sizing:border-box;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-weight:600;line-height:1.25;color:rgb(51,51,51);margin-top:24px;margin-bottom:16px;font-size:1.5em;padding-bottom:0.3em;border-bottom:1px solid rgb(238,238,238);letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial"><a class="m_-495277073226778567gmail-anchor m_-495277073226778567gmail-hidden-xs" href="https://hackmd.io/_zKtoPXUT7mDAyV6OxzZRQ?both#What-Went-Well" title="What-Went-Well" style="box-sizing:border-box;background-color:transparent;color:rgb(51,122,183);text-decoration:none;float:left;padding-right:4px;line-height:1" target="_blank"><span class="m_-495277073226778567gmail-octicon m_-495277073226778567gmail-octicon-link" style="box-sizing:border-box;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:1;font-family:octicons;display:inline-block;text-decoration:none;color:rgb(0,0,0);vertical-align:middle"></span></a>What Went Well</h2><ul class="m_-495277073226778567gmail-part" style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;padding-left:2em;color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16px;letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial"><li class="m_-495277073226778567gmail-" style="box-sizing:border-box">We did reasonably good planning to find potential issues and in fact, did find some potential issues.</li><li class="m_-495277073226778567gmail-" style="box-sizing:border-box;margin-top:0.25em">Nigel and Amar were on hand and available to fix any potential issues that popped up</li><li class="m_-495277073226778567gmail-" style="box-sizing:border-box;margin-top:0.25em">The changes landed at the end of a working day for India the day before a public holiday. While there was impact, it was much less than a similar change performed at working hours.</li></ul><h2 class="m_-495277073226778567gmail-part" id="m_-495277073226778567gmail-Future-recommendations" style="box-sizing:border-box;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-weight:600;line-height:1.25;color:rgb(51,51,51);margin-top:24px;margin-bottom:16px;font-size:1.5em;padding-bottom:0.3em;border-bottom:1px solid rgb(238,238,238);letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial"><a class="m_-495277073226778567gmail-anchor m_-495277073226778567gmail-hidden-xs" href="https://hackmd.io/_zKtoPXUT7mDAyV6OxzZRQ?both#Future-recommendations" title="Future-recommendations" style="box-sizing:border-box;background-color:transparent;color:rgb(51,122,183);text-decoration:none;float:left;padding-right:4px;line-height:1" target="_blank"><span class="m_-495277073226778567gmail-octicon m_-495277073226778567gmail-octicon-link" style="box-sizing:border-box;font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:16px;line-height:1;font-family:octicons;display:inline-block;text-decoration:none;color:rgb(0,0,0);vertical-align:middle"></span></a>Future recommendations</h2><ul class="m_-495277073226778567gmail-part" style="box-sizing:border-box;margin-top:0px;margin-bottom:16px;padding-left:2em;color:rgb(51,51,51);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,"Helvetica Neue",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:16px;letter-spacing:0.35px;text-decoration-style:initial;text-decoration-color:initial"><li class="m_-495277073226778567gmail-" style="box-sizing:border-box">Template files need to be caught by the clang-format job correctly so that they are not checked for formatting. Or the name of the file needs to be changed so that they don’t end with<span> </span><code style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",monospace;font-size:13.6px;padding:0.2em 0px;background-color:rgba(0,0,0,0.04);border-radius:3px;margin:0px;color:inherit">.c</code><span> </span>or<span> </span><code style="box-sizing:border-box;font-family:Menlo,Monaco,Consolas,"Courier New",monospace;font-size:13.6px;padding:0.2em 0px;background-color:rgba(0,0,0,0.04);border-radius:3px;margin:0px;color:inherit">.h</code>.</li><li class="m_-495277073226778567gmail-" style="box-sizing:border-box;margin-top:0.25em">In the future, high impact changes need a good process which has at least an acceptance criteria, go/no-go checklist, and a rollback procedure.</li><li class="m_-495277073226778567gmail-" style="box-sizing:border-box;margin-top:0.25em">This work is currently incomplete and the bug<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1564149#c39" style="box-sizing:border-box;background-color:transparent;color:rgb(51,122,183);text-decoration:none" target="_blank">3</a><span> </span>tracks the remaining action items.</li></ul>-----<div><br></div><div>Thanks Nigel for the postmortem report. </div></div><div dir="ltr"><div><br clear="all"><div><br></div>-- <br><div class="m_-495277073226778567gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div></div>
_______________________________________________<br>
Gluster-infra mailing list<br>
<a href="mailto:Gluster-infra@gluster.org" target="_blank">Gluster-infra@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-infra" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-infra</a></blockquote></div></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature">sankarshan mukhopadhyay<br><<a href="https://about.me/sankarshan.mukhopadhyay" target="_blank">https://about.me/sankarshan.mukhopadhyay</a>><br><br> </div>