<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><a href="https://github.com/gluster/glusterfs/blob/master/extras/distributed-testing/distributed-test-runner.py#L334">https://github.com/gluster/glusterfs/blob/master/extras/distributed-testing/distributed-test-runner.py#L334</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This call can be expensive in terms of memory because it sends the entire zip of the files from client to server. This call be optimized to send in batches if there is memory issues. You can try the change or let me know if you want me
to make the change. Otherwise this code should be very slim on memory.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12.0pt;color:black">From: </span></b><span style="font-size:12.0pt;color:black">Deepshikha Khandelwal <dkhandel@redhat.com><br>
<b>Date: </b>Monday, March 19, 2018 at 9:24 PM<br>
<b>To: </b>Karthikeyan Radhakrishnan <krad@fb.com><br>
<b>Cc: </b>Nigel Babu <nigelb@redhat.com>, gluster-infra <gluster-infra@gluster.org>, Jeff Darcy <jeff@pl.atyp.us><br>
<b>Subject: </b>Re: Distributed Testing and Memory issues<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><a name="_MailOriginalBody"><o:p> </o:p></a></p>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">On Sun, Mar 18, 2018 at 12:12 PM, Karthikeyan Radhakrishnan <</span><a href="mailto:krad@fb.com" target="_blank"><span style="mso-bookmark:_MailOriginalBody">krad@fb.com</span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody">>
wrote:<o:p></o:p></span></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">Hi Nigel,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">This is awesome!<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">MemoryError is very weird. We @Facebook have never seen that. The test server/client is super thin to cause memory pressure, but the
tests they run can cause such issues. How much memory does the machine you are running have?
<o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">I'm running this on machines having 2GB memory. And I think this is enough to have this distributed test framework setup for us.
<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">Is the machine under pressure when you see the errors? The best way would be to add a rpc to query memory stat and observe.<o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">These are newly created machines running just XMLRPC server process. I checked with top and got to know that this process is utilizing about 77% of memory at initial stage itself when the tester
part of code scans and skip kicking the host/server for availability.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">RPC is a new thing for me, so I'm not aware of RPC query calls. If you can brief me more about this, it would be helpful.
<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">Let me accelerate setting up some common space (like aws) where can re-pro such problems.<o:p></o:p></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody">It would be great.
<o:p></o:p></span></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"> <o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">Thanks!<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">-Karthik
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"> <o:p></o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"><b><span style="font-size:12.0pt;color:black">From:
</span></b></span><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt;color:black">Nigel Babu <</span></span><a href="mailto:nigelb@redhat.com" target="_blank"><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt">nigelb@redhat.com</span></span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt;color:black">><br>
<b>Date: </b>Saturday, March 17, 2018 at 7:03 AM<br>
<b>To: </b>Karthikeyan Radhakrishnan <</span></span><a href="mailto:krad@fb.com" target="_blank"><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt">krad@fb.com</span></span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt;color:black">><br>
<b>Cc: </b>gluster-infra <</span></span><a href="mailto:gluster-infra@gluster.org" target="_blank"><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt">gluster-infra@gluster.org</span></span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt;color:black">>,
Deepshikha Khandelwal <</span></span><a href="mailto:dkhandel@redhat.com" target="_blank"><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt">dkhandel@redhat.com</span></span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt;color:black">>,
Jeff Darcy <</span></span><a href="mailto:jeff@pl.atyp.us" target="_blank"><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt">jeff@pl.atyp.us</span></span><span style="mso-bookmark:_MailOriginalBody"></span></a><span style="mso-bookmark:_MailOriginalBody"><span style="font-size:12.0pt;color:black">><br>
<b>Subject: </b>Distributed Testing and Memory issues</span><o:p></o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"> <o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"><a name="m_3427284749503738446_m_5316614938492719">Hey Karthik,</a><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">Deepshikha has been working on testing the distributed test framework that you contributed (thank you!). Instead of writing our own
code to chunk the tests, we've decided to just consume what you've written so we can work on making it run both at FB and upstream.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">We're running into MemoryError exception from the threads. Do you know what's the best way to debug or let us know how much memory your
machines have? That'll help us figure out solving this sooner upstream.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"> <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">PS: This email is CC'd to gluster-infra and is archived publicly.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody"><br>
-- <o:p></o:p></span></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="mso-bookmark:_MailOriginalBody">nigelb<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="mso-bookmark:_MailOriginalBody"><o:p> </o:p></span></p>
</div>
</div>
</div>
</body>
</html>