Openstack uses Zuul [1], which manages this kinds of issues. Maybe we can explore it. This was brought up sometime before by Prashant Pai (I don't remember if it was on IRC or here). ~kaushal [1] http://ci.openstack.org/zuul/