[Gluster-infra] Automatic nightly builds for glusterfs RPMs?

Niels de Vos ndevos at redhat.com
Wed Jan 15 17:01:47 UTC 2014

On Tue, Jan 14, 2014 at 10:54:05PM +0530, Vijay Bellur wrote:
> On 01/14/2014 05:36 PM, Niels de Vos wrote:
> >On Mon, Jan 13, 2014 at 11:38:54PM +0530, Vijay Bellur wrote:
> >>On 01/13/2014 12:30 AM, Niels de Vos wrote:
> >>>Hi all,
> >>>
> >>>during the last weekly meeting we touched upon creating nightly builds
> >>>from the glusterfs.git repository. It seems that Fedora offers a service
> >>>called COPR [1] that makes this really easy. Of course, this builds only
> >>>RPMs for Fedora and EPEL, but I think it is a good start.
> >>>
> >>>IMHO we should aim for building packages from the main glusterfs.git
> >>>repository, and not pull in anything from the Fedora packaging
> >>>infrastructure (dist-git). There are some differences between the Fedora
> >>>packages and the community ones. In future, these should get minimized,
> >>>but we are a little more flexible with the community packages (less
> >>>strict packaging guidelines, making glusterfs-regression-test possible).
> >>>
> >>>I've put a script together [2] that can be used to create the needed
> >>>SRPM, scp's it to a public download location, and triggers a COPR build.
> >>>This script has been used to create COPR repositories for the master
> >>>branch [3], release-3.5 [4] and release-3.4 [5]. Fedora 18-20 and EPEL-6
> >>>are currently available (x86_64 only). For an unknown reason, EPEL-5
> >>>fails to build :-/
> >>>
> >>>It should be pretty easy to use the COPR repositories for automated
> >>>testing. The RPMs are not intended for general consumption. There are
> >>>some things that aren't too nice (like, master branch packages have
> >>>a version of 'date +%Y%m%d'). I have not tried to install any of the
> >>>generated RPMs yet. The main purpose of this exercise is to show whats
> >>>possible with minimal effort.
> >>>
> >>>Any ideas, thoughts or other feedback is much appreciated. Things I'd
> >>>like to see answered are:
> >>>- Can something like this be used?
> >>
> >>I think we could give this a try.
> >>
> >>>- How/when should these packages get build?
> >>
> >>Nightly if commits have been merged in the last 24 hours.
> >
> >Updated the script [1] with a "-H <HOURS>" switch. It defaults to 24
> >hours and will not trigger a build if 'git log --since=24hours' is
> >empty.
> Nice!
> >
> >>>- Should this be run on build.gluster.org or download.gluster.org?
> >>
> >>We can possibly run the script as a jenkins job or through cron in
> >>build.gluster.org.
> >
> >I can setup a cron-job on build.gluster.org that runs as my user. Any
> >particular time that would be most convenient?
> Don't seem to have a preference. Maybe 00:00 UTC ? :)
> >
> >>>- Sync the repositories to download.gluster.org?
> >>
> >>+1. download.gluster.org is something that we can publish as the
> >>source for all nightly builds.
> >
> >I dont know how (if) automatic pushes to download.gluster.org are done.
> >Maybe just setup an rsync or similar job that copies the COPR
> >repositories over?

Things that need to be done:
1. make sure that the following works on the server
   - git clone
   - ./autogen.sh
   - make dist
   - rpmbuild -ts *.tar.gz
   -> build.gluster.org is fine for this
2. install copr-cli on the system
   - depends on python-argparse and python-requests (from EPEL-6?)
   - el6 package from http://people.fedoraproject.org/~devos/copr/
   -> no el6 builds in Fedora EPEL (requested in Bug 1053142)
3. copy the API-key from http://copr.fedoraproject.org/api
   -> done on my user account on build.gluster.org
4. setup a cronjob to trigger the builds
   - start time 00:00 UTC
   - use a fixed working directory for git caching (and verification?)
   - use a shell script for sequential builds
   (optionally the script can wait until a build has finished)
5. pick a system where the packages should be located
   - pull the packages on download.gluster.org
   - or, push them to download.gluster.org
   - what target directory?
   - add a README.WARNING to make clear that the packages are for 
     testing ONLY
6. trigger a sync of the repos
   - use lftp or something similar, this works for me:
     $ lftp -e 'mirror --only-newer' \

Current blocking points:
- I need help/permission/agreement to do step 2
- Any input, recommendation for step 5 + 6


More information about the Gluster-infra mailing list