[Gluster-infra] Reducing the number of builders in the cage

[Michael Scherer]

Le lundi 15 octobre 2018 à 14:54 +0200, Michael Scherer a écrit :
> Le lundi 15 octobre 2018 à 17:36 +0530, Nigel Babu a écrit :
> > I think it might we worth pulling out some utilization numbers to
> > see
> > how
> > many to pull.
> 
> So, I think the first step would be to clean the list of jobs and
> labels. For example, this one didn't run since 7 months, should we
> keep
> it ?
> 
> https://build.gluster.org/label/glusto/
> 
> The same go for this one:
> https://build.gluster.org/label/master/
> 
> I am looking at the jobs, and I think the only 2 group of projects
> that
> we should look at are:
> https://build.gluster.org/label/rpm7/

So, a rough estimate is 50 build per jobs ( we have 24 today after 12h,
so I will just multiply by 2), with 15 minutes per jobs. There is 4
jobs for that group, so for today, we would have used 50h of cpu time.
That's roughly 2 builders, so let's make that 3 or 4. 

We currently have 39 there, so that's around 90% unused. 

> https://build.gluster.org/label/smoke7/

So removing the jobs that run less than 2 times per day, the only jobs
that I looked are smoke and gd2-smoke.

I will again count 15 minutes per jobs to simplify the stats.

We have also 24 jobs for gd2 and smoke. So 50 jobs per day, so 100
smokes job. So we would need 25h of cpu time, aka 1 builder. Let's us
use 2 to cope with the load, or even 3 for redundancy.

So we would need a total of 7 VMs, and we would still be chunking
things as fast as now. Make it 10 and we could cope with twice as much
load of patches.

So I suggest to shutdown 75% of the builders with the label
smoke7/rpm7. 


> and maybe:
> https://build.gluster.org/label/regression7/
> (even if I guess the goal long term is to have this outside of the
> cage)
> 
> The rest is either using specialized builders (like bugziller, debian
> ones, freebsd), or stuff that may need to be removed, or stuff
> outside
> the DC.
> 
> 
> >  If we can get the freebsd builder working, that would
> > eliminate the need to run it on rackspace and having two of them
> > would
> > increase the speed at which we process the smoke queue.
> 
> The freebsd builder is working: 
> 
> https://build.gluster.org/job/freebsd-non-voting-smoke/
> 
> Also
> On Mon, Oct 15, 2018 at 5:32 PM Michael Scherer <mscherer at redhat.com>
> > wrote:
> > 
> > > Le lundi 15 octobre 2018 à 15:29 +0530, Sankarshan Mukhopadhyay a
> > > écrit :
> > > > On Mon, Oct 15, 2018 at 3:19 PM Michael Scherer <mscherer at redha
> > > > t.
> > > > com>
> > > > wrote:
> > > > 
> > > > > so we currently have 50 builders in the cage, and I think
> > > > > that's
> > > > > too
> > > > > much. While that's not a huge issue, having too much VMs do
> > > > > cause
> > > > > slowdown on ansible, consume ressources for nothing (disk
> > > > > space,
> > > > > CPU,
> > > > > bw). When I look on the graph on https://munin.gluster.org/ ,
> > > > > there
> > > > > isn't much load.
> > > > > 
> > > > > So I would like to start remove some of them and see how that
> > > > > go.
> > > > 
> > > > What is the value of this "some of them"? 10%, 20% ...
> > > 
> > > It depend on the builders, but I think i would remove 10% for a
> > > start,
> > > so around 4.
> > > 
> > > Now, not all builders are equal, I am not gonna remove the debian
> > > nor
> > > the freebsd one, of course.
> > > 
> > > I think the easiest would be to mark them offline in jenkins, see
> > > if
> > > that create issues (such as the queue becoming large), and if
> > > nothing
> > > happen, remove them.
> > > 
> > > AFAIK, thoses are mostly here for smoke tests, format, warnings,
> > > etc.
> > > So fast jobs.
> > > --
> > > Michael Scherer
> > > Sysadmin, Community Infrastructure and Platform, OSAS
> > > 
> > > _______________________________________________
> > > Gluster-infra mailing list
> > > Gluster-infra at gluster.org
> > > https://lists.gluster.org/mailman/listinfo/gluster-infra
> > 
> > 
> > 
> 
> _______________________________________________
> Gluster-infra mailing list
> Gluster-infra at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-infra
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-infra/attachments/20181015/516d697b/attachment.sig>