<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Mar 18, 2018 at 12:12 PM, Karthikeyan Radhakrishnan <span dir="ltr"><<a href="mailto:krad@fb.com" target="_blank">krad@fb.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="#0563C1" vlink="#954F72" lang="EN-US">
<div class="m_3427284749503738446m_5316614938492719640WordSection1">
<p class="MsoNormal">Hi Nigel,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">This is awesome!<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">MemoryError is very weird. We @Facebook have never seen that. The test server/client is super thin to cause memory pressure, but the tests they run can cause such issues. How much memory does the machine you are running have? </p></div></div></blockquote><div>I'm running this on machines having 2GB memory. And I think this is enough to have this distributed test framework setup for us. <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="#0563C1" vlink="#954F72" lang="EN-US"><div class="m_3427284749503738446m_5316614938492719640WordSection1"><p class="MsoNormal">Is the machine
under pressure when you see the errors? The best way would be to add a rpc to query memory stat and observe.</p></div></div></blockquote><div>These are newly created machines running just XMLRPC server process. I checked with top and got to know that this process is utilizing about 77% of memory at initial stage itself when the tester part of code scans and skip kicking the host/server for availability.</div><div>RPC is a new thing for me, so I'm not aware of RPC query calls. If you can brief me more about this, it would be helpful. <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="#0563C1" vlink="#954F72" lang="EN-US"><div class="m_3427284749503738446m_5316614938492719640WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Let me accelerate setting up some common space (like aws) where can re-pro such problems.</p></div></div></blockquote><div>It would be great. <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div link="#0563C1" vlink="#954F72" lang="EN-US"><div class="m_3427284749503738446m_5316614938492719640WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks!<u></u><u></u></p>
<p class="MsoNormal">-Karthik <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Nigel Babu <<a href="mailto:nigelb@redhat.com" target="_blank">nigelb@redhat.com</a>><br>
<b>Date: </b>Saturday, March 17, 2018 at 7:03 AM<br>
<b>To: </b>Karthikeyan Radhakrishnan <<a href="mailto:krad@fb.com" target="_blank">krad@fb.com</a>><br>
<b>Cc: </b>gluster-infra <<a href="mailto:gluster-infra@gluster.org" target="_blank">gluster-infra@gluster.org</a>>, Deepshikha Khandelwal <<a href="mailto:dkhandel@redhat.com" target="_blank">dkhandel@redhat.com</a>>, Jeff Darcy <<a href="mailto:jeff@pl.atyp.us" target="_blank">jeff@pl.atyp.us</a>><br>
<b>Subject: </b>Distributed Testing and Memory issues<u></u><u></u></span></p>
</div><div><div class="m_3427284749503738446h5">
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><a name="m_3427284749503738446_m_5316614938492719640__MailOriginalBody">Hey Karthik,<u></u><u></u></a></p>
</div>
<div>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span>Deepshikha has been working on testing the distributed test framework that you contributed (thank you!). Instead of writing our own code to chunk the tests, we've decided to just consume what
you've written so we can work on making it run both at FB and upstream.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span>We're running into MemoryError exception from the threads. Do you know what's the best way to debug or let us know how much memory your machines have? That'll help us figure out solving this
sooner upstream.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span>PS: This email is CC'd to gluster-infra and is archived publicly.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span><br>
-- <u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal"><span>nigelb<u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br></div></div>