[Gluster-infra] Automatic nightly builds for glusterfs RPMs?

[Vijay Bellur]

On 01/15/2014 10:31 PM, Niels de Vos wrote:
> On Tue, Jan 14, 2014 at 10:54:05PM +0530, Vijay Bellur wrote:
>> On 01/14/2014 05:36 PM, Niels de Vos wrote:
>>> On Mon, Jan 13, 2014 at 11:38:54PM +0530, Vijay Bellur wrote:
>>>> On 01/13/2014 12:30 AM, Niels de Vos wrote:
>>>>> Hi all,
>>>>>
>>>>> during the last weekly meeting we touched upon creating nightly builds
>>>> >from the glusterfs.git repository. It seems that Fedora offers a service
>>>>> called COPR [1] that makes this really easy. Of course, this builds only
>>>>> RPMs for Fedora and EPEL, but I think it is a good start.
>>>>>
>>>>> IMHO we should aim for building packages from the main glusterfs.git
>>>>> repository, and not pull in anything from the Fedora packaging
>>>>> infrastructure (dist-git). There are some differences between the Fedora
>>>>> packages and the community ones. In future, these should get minimized,
>>>>> but we are a little more flexible with the community packages (less
>>>>> strict packaging guidelines, making glusterfs-regression-test possible).
>>>>>
>>>>> I've put a script together [2] that can be used to create the needed
>>>>> SRPM, scp's it to a public download location, and triggers a COPR build.
>>>>> This script has been used to create COPR repositories for the master
>>>>> branch [3], release-3.5 [4] and release-3.4 [5]. Fedora 18-20 and EPEL-6
>>>>> are currently available (x86_64 only). For an unknown reason, EPEL-5
>>>>> fails to build :-/
>>>>>
>>>>> It should be pretty easy to use the COPR repositories for automated
>>>>> testing. The RPMs are not intended for general consumption. There are
>>>>> some things that aren't too nice (like, master branch packages have
>>>>> a version of 'date +%Y%m%d'). I have not tried to install any of the
>>>>> generated RPMs yet. The main purpose of this exercise is to show whats
>>>>> possible with minimal effort.
>>>>>
>>>>> Any ideas, thoughts or other feedback is much appreciated. Things I'd
>>>>> like to see answered are:
>>>>> - Can something like this be used?
>>>>
>>>> I think we could give this a try.
>>>>
>>>>> - How/when should these packages get build?
>>>>
>>>> Nightly if commits have been merged in the last 24 hours.
>>>
>>> Updated the script [1] with a "-H <HOURS>" switch. It defaults to 24
>>> hours and will not trigger a build if 'git log --since=24hours' is
>>> empty.
>>
>> Nice!
>>
>>>
>>>>> - Should this be run on build.gluster.org or download.gluster.org?
>>>>
>>>> We can possibly run the script as a jenkins job or through cron in
>>>> build.gluster.org.
>>>
>>> I can setup a cron-job on build.gluster.org that runs as my user. Any
>>> particular time that would be most convenient?
>>
>> Don't seem to have a preference. Maybe 00:00 UTC ? :)
>>
>>>
>>>>> - Sync the repositories to download.gluster.org?
>>>>
>>>> +1. download.gluster.org is something that we can publish as the
>>>> source for all nightly builds.
>>>
>>> I dont know how (if) automatic pushes to download.gluster.org are done.
>>> Maybe just setup an rsync or similar job that copies the COPR
>>> repositories over?
>
> Things that need to be done:
> 1. make sure that the following works on the server
>     - git clone
>     - ./autogen.sh
>     - make dist
>     - rpmbuild -ts *.tar.gz
>     -> build.gluster.org is fine for this
> 2. install copr-cli on the system
>     - depends on python-argparse and python-requests (from EPEL-6?)
>     - el6 package from http://people.fedoraproject.org/~devos/copr/

Done.

[root at build ~]# rpm -qa | grep copr
copr-cli-1.15-1.el6.noarch

>     -> no el6 builds in Fedora EPEL (requested in Bug 1053142)
> 3. copy the API-key from http://copr.fedoraproject.org/api
>     -> done on my user account on build.gluster.org
> 4. setup a cronjob to trigger the builds
>     - start time 00:00 UTC
>     - use a fixed working directory for git caching (and verification?)
>     - use a shell script for sequential builds
>     (optionally the script can wait until a build has finished)
> 5. pick a system where the packages should be located
>     - pull the packages on download.gluster.org
>     - or, push them to download.gluster.org
>     - what target directory?

pub/gluster/glusterfs/nightly/<version>/ is one possibility.

>     - add a README.WARNING to make clear that the packages are for
>       testing ONLY
> 6. trigger a sync of the repos
>     - use lftp or something similar, this works for me:
>       $ lftp -e 'mirror --only-newer' \
>         http://copr-be.cloud.fedoraproject.org/results/devos/glusterfs-3.5/

Looks good to me.

Thanks,
Vijay

>
> Current blocking points:
> - I need help/permission/agreement to do step 2
> - Any input, recommendation for step 5 + 6
>
> Thanks,
> Niels
>