<div dir="ltr"><div>Hi,</div><div><br></div>I think there could be multiple reasons for not getting similar improvement as we were getting.<br><br><div>1) You are executing a test case in a VM environment. The resources are limited as compared to a physical machine.<br>2) The improvement was showing after enable tcmalloc and disable glusterfs based mempool, I am not<br> sure how the memory managed in case of vm environment. On physical server there is no doubt we were<br> getting significant improvement. <br>3) Can you share what gluster options you have enabled to run the test ?<br> gluster v info<br>4) What is smallfile parameters you have used to run the test case (no_of_file and no_of_threads)<br><br></div><div>Thanks,<br>Mohit Agrawal<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Jan 20, 2022 at 12:24 PM 박현승 <<a href="mailto:hspark0582@gluesys.com">hspark0582@gluesys.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<img src="https://mail.gluesys.com/mail/mail_confirm.php?host=gluesys.com&id=hspark0582&sid=16426616340088041004&rcptto=gluster-devel%40gluster.org" style="display: block;" width="1" height="1">
<p style="font-size:10pt;margin:0px"><span style="font-family:Arial">Dear Gluster developers,</span></p>
<p style="font-size:10pt;margin:0px"> </p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">This is Hyunseung Park at Gluesys, South Korea.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial"> </span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">We are trying to replicate the test in <a href="https://github.com/gluster/glusterfs/issues/2771" target="_blank">https://github.com/gluster/glusterfs/issues/2771</a> but to no avail.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">In our experiments, Gluster version 10 unfortunately did not perform noticably better than version 9.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial"> </span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"> </p>
<table border="1" cellpadding="0" cellspacing="0" dir="ltr" style="table-layout:fixed;font-size:10pt;font-family:Arial;width:0px;border-collapse:collapse;border:none">
<colgroup>
<col width="100">
<col width="100">
<col width="100">
<col width="100">
<col width="100">
</colgroup>
<tbody>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">v9 2x2</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">average</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">create</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">3399.99</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">3484.79</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2702.57</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">3195.783333</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">ls -l</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">65605.2</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">64930.6</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">72018.7</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">67518.16667</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">chmod</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">4858.95</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">4965.29</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">5597.73</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">5140.656667</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">stat</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7334.88</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7755.89</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">8335.11</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7808.626667</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">read</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7015.64</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">8255.48</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7007.01</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7426.043333</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">append</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2554.93</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2777.65</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2572.57</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2635.05</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">mkdir</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1800.29</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1865.07</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1805.48</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1823.613333</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">rmdir</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1854.09</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1722.89</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1876.81</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1817.93</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">cleanup</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2402.02</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2447.36</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2438.71</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2429.363333</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">v10 2x2</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial"> </span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">average</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">create</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">3741.39</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">3174.82</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">3234.42</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">3383.543333</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">ls -l</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">71543.7</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">67275.9</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">72975.1</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">70598.23333</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">chmod</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">5441.11</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">5109.22</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">5004.08</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">5184.803333</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">stat</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7746.37</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7677.99</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7885.72</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7770.026667</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">read</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7061.12</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7165.21</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7121.07</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">7115.8</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">append</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">3458.93</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2641.84</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2887.46</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2996.076667</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">mkdir</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2685.22</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1879.35</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1970.91</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2178.493333</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">rmdir</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2240.11</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1648.37</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1602.16</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">1830.213333</span></td>
</tr>
<tr style="height:21px">
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom"><span style="font-family:Arial">cleanup</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">3739.68</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2407.57</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2403.48</span></td>
<td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right"><span style="font-family:Arial">2850.243333</span></td>
</tr>
</tbody>
</table>
<p> </p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">The result above is from the test that deployed 32 threads on each of the 4 clients.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">Some results were better than others, but it is not good enough when compared to the result in the aforementioned link.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">We are wondering what we can do to get the full potential of the new version.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"> </p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">We have been running tests with varying file sizes, number of threads, different volume topology, etc. but we were not able to see data conclusive enough.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">We were also not able to find meaningful output from running tests using other benchmark tools such as bonnie++ and FIO.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"> </p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">To find the potential cause we tried to look into the program by calling malloc_stats() and using perf.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">However, we also could not find something noteworthy from the result.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">Here is the data recorded during one set of smallfile test (from create to cleanup): <a href="https://drive.google.com/drive/folders/1NMXNjgOZ7svDd4-YvKCU4UAp43tm15dC?usp=sharing" target="_blank">https://drive.google.com/drive/folders/1NMXNjgOZ7svDd4-YvKCU4UAp43tm15dC?usp=sharing</a></span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial"> </span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">Below is our test environment:</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial"> </span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">Basic HW info: VM (vSphere), 2 core CPU, 4G RAM. 4 servers and 4 clients.</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">OS: Centos 7</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">kernel version: 3.10.0-1160</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">Gluster version: built from git, checked out from branch 'release-9' and 'release-10' respectively (commits 7094da and a804f8)</span></p>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"><span style="font-family:Arial">build option: default setting except "./configure --disable-linux-io_uring" (kernel does not support io_uring)</span></p>
<div style="font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt">
<p><span style="font-family:Arial">Configuration result (case of version 10):</span></p>
<p><span style="font-family:Arial">GlusterFS configure summary<br>
===========================<br>
FUSE client : yes<br>
epoll IO multiplex : yes<br>
fusermount : yes<br>
readline : no<br>
georeplication : yes<br>
Linux-AIO : yes<br>
Linux io_uring : no<br>
Use liburing : no<br>
Enable Debug : no<br>
Run with Valgrind : no<br>
Sanitizer enabled : none<br>
XML output : yes<br>
Unit Tests : no<br>
Track priv ports : yes<br>
POSIX ACLs : yes<br>
SELinux features : yes<br>
firewalld-config : no<br>
Events : yes<br>
EC dynamic support : x64 sse avx<br>
Use memory pools : no<br>
Nanosecond m/atimes : yes<br>
Server components : yes<br>
Legacy gNFS server : no<br>
IPV6 default : no<br>
Use TIRPC : yes<br>
With Python : 3.6<br>
Cloudsync : yes<br>
Metadata dispersal : no<br>
Link with TCMALLOC : yes<br>
Enable Brick Mux : no<br>
Building with LTO : no</span></p>
<p><span style="font-family:Arial"> </span></p>
<p><span style="font-family:Arial"> </span></p>
</div>
<p style="margin:0px;font-family:"\00b9d1\00c740 \00ace0\00b515";font-size:10pt"> </p>
-------<br>
<br>
Community Meeting Calendar:<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
<br>
</blockquote></div>