[Gluster-devel] Gluster builder being hit by too much process

[Michael Scherer]

Hi,

so on the last 2 days, I have been contacted by people because some
builders were failling. Upon investigation ( https://bugzilla.redhat.co
m/show_bug.cgi?id=1498390 ), the main issue seems to be the following:

Each build failed had a set of glusterd process (around 300) that where
started by jenkins, to test regression for this change:
https://review.gluster.org/#/c/18271/ 

(found due to environment variable of the process)

But upon closer inspection of the patch, it doesn't seems buggy, so my
suspicion are on the test case, who is also quite simple (and likely
bug free), but who also seems to start a ton of of volume (around 1000
) if I am not wrong, and this do seems to result into a large number of
process being created.

See https://review.gluster.org/#/c/18271/5/tests/bugs/cli/bug-1490853.t


Could it be that the test case do uncover a bug in the test suite, or a
bug in gluster ?

Looking at the test suite, I see that the cleanup function is
conveniently ignoring a ton of error:
https://github.com/gluster/glusterfs/blob/master/tests/include.rc#L465
which do not help to see what is going wrong.

I also do not see out of memory errors, but munin graph seems to stop
right at the same time, so maybe that's just a ressource issue.

So my questions are:
- is gluster supposed to scale gracefully with 1000 volumes on 1 single
 node ?
- how much ressources should we plan for that ? (right now, we have 2G
VM, we can't increase much without reinstalling the whole set of
servers)

If you see any builders not working, please ping me on irc.

-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20171006/25d4624b/attachment.sig>