<img src="https://mail.gluesys.com/mail/mail_confirm.php?host=gluesys.com&id=hspark0582&sid=16426616340088041004&rcptto=gluster-devel%40gluster.org" style="display:block" width="1" height="1">

<p style="font-size:10pt;margin:0px;"><span style="font-family:Arial;">Dear Gluster developers,</span></p>

<p style="font-size:10pt;margin:0px;"> </p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">This is Hyunseung Park at Gluesys, South Korea.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">  </span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">We are trying to replicate the test in https://github.com/gluster/glusterfs/issues/2771 but to no avail.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">In our experiments, Gluster version 10 unfortunately did not perform noticably better than version 9.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">  </span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"> </p>

<table border="1" cellpadding="0" cellspacing="0" dir="ltr" style="table-layout:fixed;font-size:10pt;font-family:Arial;width:0px;border-collapse:collapse;border:none" xmlns="http://www.w3.org/1999/xhtml">
        <colgroup>
                <col width="100" />
                <col width="100" />
                <col width="100" />
                <col width="100" />
                <col width="100" />
        </colgroup>
        <tbody>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">v9 2x2</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">average</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">create</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3399.99</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3484.79</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2702.57</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3195.783333</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">ls -l</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">65605.2</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">64930.6</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">72018.7</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">67518.16667</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">chmod</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">4858.95</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">4965.29</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5597.73</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5140.656667</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">stat</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7334.88</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7755.89</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">8335.11</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7808.626667</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">read</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7015.64</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">8255.48</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7007.01</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7426.043333</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">append</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2554.93</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2777.65</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2572.57</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2635.05</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">mkdir</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1800.29</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1865.07</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1805.48</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1823.613333</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">rmdir</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1854.09</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1722.89</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1876.81</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1817.93</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">cleanup</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2402.02</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2447.36</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2438.71</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2429.363333</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">v10 2x2</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">  </span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">average</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">create</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3741.39</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3174.82</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3234.42</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3383.543333</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">ls -l</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">71543.7</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">67275.9</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">72975.1</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">70598.23333</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">chmod</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5441.11</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5109.22</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5004.08</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5184.803333</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">stat</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7746.37</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7677.99</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7885.72</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7770.026667</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">read</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7061.12</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7165.21</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7121.07</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7115.8</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">append</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3458.93</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2641.84</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2887.46</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2996.076667</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">mkdir</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2685.22</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1879.35</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1970.91</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2178.493333</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">rmdir</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2240.11</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1648.37</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1602.16</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1830.213333</span></td>
                </tr>
                <tr style="height:21px;">
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">cleanup</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3739.68</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2407.57</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2403.48</span></td>
                        <td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2850.243333</span></td>
                </tr>
        </tbody>
</table>

<p> </p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">The result above is from the test that deployed 32 threads on each of the 4 clients.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Some results were better than others, but it is not good enough when compared to the result in the aforementioned link.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">We are wondering what we can do to get the full potential of the new version.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"> </p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">We have been running tests with varying file sizes, number of threads, different volume topology, etc. but we were not able to see data conclusive enough.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">We were also not able to find meaningful output from running tests using other benchmark tools such as bonnie++ and FIO.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"> </p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">To find the potential cause we tried to look into the program by calling malloc_stats() and using perf.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">However, we also could not find something noteworthy from the result.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Here is the data recorded during one set of smallfile test (from create to cleanup): https://drive.google.com/drive/folders/1NMXNjgOZ7svDd4-YvKCU4UAp43tm15dC?usp=sharing</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">  </span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Below is our test environment:</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">  </span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Basic HW info: VM (vSphere), 2 core CPU, 4G RAM. 4 servers and 4 clients.</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">OS: Centos 7</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">kernel version: 3.10.0-1160</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Gluster version: built from git, checked out from branch 'release-9' and 'release-10' respectively (commits 7094da and a804f8)</span></p>

<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">build option: default setting except "./configure --disable-linux-io_uring" (kernel does not support io_uring)</span></p>

<div style="font-family:맑은 고딕; font-size:10pt">
<p><span style="font-family:Arial;">Configuration result (case of version 10):</span></p>

<p><span style="font-family:Arial;">GlusterFS configure summary<br />
===========================<br />
FUSE client          : yes<br />
epoll IO multiplex   : yes<br />
fusermount           : yes<br />
readline             : no<br />
georeplication       : yes<br />
Linux-AIO            : yes<br />
Linux io_uring       : no<br />
Use liburing         : no<br />
Enable Debug         : no<br />
Run with Valgrind    : no<br />
Sanitizer enabled    : none<br />
XML output           : yes<br />
Unit Tests           : no<br />
Track priv ports     : yes<br />
POSIX ACLs           : yes<br />
SELinux features     : yes<br />
firewalld-config     : no<br />
Events               : yes<br />
EC dynamic support   : x64 sse avx<br />
Use memory pools     : no<br />
Nanosecond m/atimes  : yes<br />
Server components    : yes<br />
Legacy gNFS server   : no<br />
IPV6 default         : no<br />
Use TIRPC            : yes<br />
With Python          : 3.6<br />
Cloudsync            : yes<br />
Metadata dispersal   : no<br />
Link with TCMALLOC   : yes<br />
Enable Brick Mux     : no<br />
Building with LTO    : no</span></p>

<p><span style="font-family:Arial;">  </span></p>

<p><span style="font-family:Arial;">  </span></p>
</div>

<p style="margin:0px;font-family:맑은 고딕; font-size:10pt"> </p>