<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Result for a single ls on a directory with 10k directories inside
(16*3 volume)</p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-33e4f081-7fff-9afd-5c71-bdcbae1cffcc"><br>
</b></p>
<div dir="ltr" style="margin-left:0pt;" align="left">
<table
style="border:none;border-collapse:collapse;width:468pt;table-layout:fixed">
<colgroup><col width="199"><col width="113"><col><col></colgroup><tbody>
<tr style="height:21.75pt">
<td style="border-left:solid #000000 1pt;border-right:solid
#000000 1pt;border-bottom:solid #000000
1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Configuration</span></p>
</td>
<td style="border-left:solid #000000 1pt;border-right:solid
#000000 1pt;border-bottom:solid #000000
1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Plain volume</span></p>
</td>
<td style="border-left:solid #000000 1pt;border-right:solid
#000000 1pt;border-bottom:solid #000000
1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Parallel-readdir</span></p>
</td>
<td style="border-left:solid #000000 1pt;border-right:solid
#000000 1pt;border-bottom:solid #000000
1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Proposed Solution</span></p>
</td>
</tr>
<tr style="height:0pt">
<td style="border-left:solid #000000 1pt;border-right:solid
#000000 1pt;border-bottom:solid #000000
1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Single Dir ls (Seconds)</span></p>
</td>
<td style="border-left:solid #000000 1pt;border-right:solid
#000000 1pt;border-bottom:solid #000000
1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">-----</span></p>
</td>
<td style="border-left:solid #000000 1pt;border-right:solid
#000000 1pt;border-bottom:solid #000000
1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">135</span></p>
</td>
<td style="border-left:solid #000000 1pt;border-right:solid
#000000 1pt;border-bottom:solid #000000
1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">32.744</span></p>
</td>
</tr>
</tbody>
</table>
</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">It is showing 321% improvements.</div>
<div class="moz-cite-prefix"><br>
</div>
<div class="moz-cite-prefix">Regards</div>
<div class="moz-cite-prefix">Rafi KC</div>
<div class="moz-cite-prefix">On 27/05/20 11:26 am, RAFI KC wrote:<br>
</div>
<blockquote type="cite"
cite="mid:9e1d43b8-98cc-9dae-193e-75faf6793e59@redhat.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<p>Hi All,</p>
<p>I have been working on POC to improve readdirp performance
improvement. At the end of the experiment, The results are
showing promising result in performance, overall there is a 104%
improvement for full filesystem crawl compared to the existing
solution. Here is the short test numbers. The tests were carried
out in 16*3 setup with 1.5 Million dentries (Both files and
dir). The system also contains some empty directories. <b>In
the result the proposed solution is 287% faster than the plane
volume and 104% faster than the parallel-readdir based
solution.</b><br>
</p>
<p><b style="font-weight:normal;"
id="docs-internal-guid-8644ede0-7fff-09be-a0d9-5ee943340f2c"><br>
</b></p>
<div dir="ltr" style="margin-left:0pt;" align="left">
<table
style="border:none;border-collapse:collapse;width:468pt;table-layout:fixed">
<colgroup><col width="199"><col width="113"><col><col></colgroup><tbody>
<tr style="height:0pt">
<td style="border-left:solid #000000
1pt;border-right:solid #000000 1pt;border-bottom:solid
#000000 1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Configuration</span></p>
</td>
<td style="border-left:solid #000000
1pt;border-right:solid #000000 1pt;border-bottom:solid
#000000 1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Plain volume</span></p>
</td>
<td style="border-left:solid #000000
1pt;border-right:solid #000000 1pt;border-bottom:solid
#000000 1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Parallel-readdir</span></p>
</td>
<td style="border-left:solid #000000
1pt;border-right:solid #000000 1pt;border-bottom:solid
#000000 1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">Proposed Solution</span></p>
</td>
</tr>
<tr style="height:0pt">
<td style="border-left:solid #000000
1pt;border-right:solid #000000 1pt;border-bottom:solid
#000000 1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">FS Crawl Time in Seconds</span></p>
</td>
<td style="border-left:solid #000000
1pt;border-right:solid #000000 1pt;border-bottom:solid
#000000 1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">16497.523</span></p>
</td>
<td style="border-left:solid #000000
1pt;border-right:solid #000000 1pt;border-bottom:solid
#000000 1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">8717.872</span></p>
</td>
<td style="border-left:solid #000000
1pt;border-right:solid #000000 1pt;border-bottom:solid
#000000 1pt;border-top:solid #000000
1pt;vertical-align:top;padding:5pt 5pt 5pt
5pt;overflow:hidden;overflow-wrap:break-word;">
<p dir="ltr" style="line-height:1.2;text-align:
center;margin-top:0pt;margin-bottom:0pt;"><span style="font-size:10pt;font-family:Arial;color:#000000;background-color:#ffffff;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre;white-space:pre-wrap;">4261.401 </span></p>
</td>
</tr>
</tbody>
</table>
</div>
<p>In short, the basic idea behind the proposal is the efficient
managing of readdir buffer in gluster along with prefetching the
dentries for intelligent switch-over to the next buffer. The
detailed problem description, deign description and results are
available in the doc.<a
href="https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit"
moz-do-not-send="true">https://docs.google.com/document/d/10z4T5Sd_-wCFrmDrzyQtlWOGLang1_g17wO8VUxSiJ8/edit</a>
<br>
</p>
<p><br>
</p>
<p>If anybody can help with the testing on a different kind of
workloads, I would be very happy to assist. If wanted to test
the patch and run a performance test on your setup, I could help
with back-porting the patch to the version of your choice.<br>
</p>
<p><br>
</p>
<p><a class="moz-txt-link-freetext"
href="https://review.gluster.org/24469" moz-do-not-send="true">https://review.gluster.org/24469</a></p>
<p><a class="moz-txt-link-freetext"
href="https://review.gluster.org/24470" moz-do-not-send="true">https://review.gluster.org/24470</a><br>
</p>
<p><br>
</p>
<p>Regards</p>
<p>Rafi KC<br>
</p>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">________
Community Meeting Calendar:
Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: <a class="moz-txt-link-freetext" href="https://bluejeans.com/441850968">https://bluejeans.com/441850968</a>
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</body>
</html>