[Gluster-infra] Distributed Testing and Memory issues

[Deepshikha Khandelwal]

On Sun, Mar 18, 2018 at 12:12 PM, Karthikeyan Radhakrishnan <krad at fb.com>
wrote:

> Hi Nigel,
>
>
>
> This is awesome!
>
>
>
> MemoryError is very weird. We @Facebook have never seen that. The test
> server/client is super thin to cause memory pressure, but the tests they
> run can cause such issues. How much memory does the machine you are running
> have?
>
I'm running this on machines having 2GB memory. And I think this is enough
to have this distributed test framework setup for us.

> Is the machine under pressure when you see the errors? The best way would
> be to add a rpc to query memory stat and observe.
>
These are newly created machines running just XMLRPC server process. I
checked with top and got to know that this process is utilizing about 77%
of memory at initial stage itself when the tester part of code scans and
skip kicking the host/server for availability.
RPC is a new thing for me, so I'm not aware of RPC query calls. If you can
brief me more about this, it would be helpful.

>
>
> Let me accelerate setting up some common space (like aws) where can re-pro
> such problems.
>
It would be great.

>
>
> Thanks!
>
> -Karthik
>
>
>
> *From: *Nigel Babu <nigelb at redhat.com>
> *Date: *Saturday, March 17, 2018 at 7:03 AM
> *To: *Karthikeyan Radhakrishnan <krad at fb.com>
> *Cc: *gluster-infra <gluster-infra at gluster.org>, Deepshikha Khandelwal <
> dkhandel at redhat.com>, Jeff Darcy <jeff at pl.atyp.us>
> *Subject: *Distributed Testing and Memory issues
>
>
>
> Hey Karthik,
>
>
>
> Deepshikha has been working on testing the distributed test framework that
> you contributed (thank you!). Instead of writing our own code to chunk the
> tests, we've decided to just consume what you've written so we can work on
> making it run both at FB and upstream.
>
>
>
> We're running into MemoryError exception from the threads. Do you know
> what's the best way to debug or let us know how much memory your machines
> have? That'll help us figure out solving this sooner upstream.
>
>
>
> PS: This email is CC'd to gluster-infra and is archived publicly.
>
>
> --
>
> nigelb
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-infra/attachments/20180320/79641ddd/attachment.html>