<img src="https://mail.gluesys.com/mail/mail_confirm.php?host=gluesys.com&id=hspark0582&sid=16426616340088041004&rcptto=gluster-devel%40gluster.org" style="display:block" width="1" height="1">
<p style="font-size:10pt;margin:0px;"><span style="font-family:Arial;">Dear Gluster developers,</span></p>
<p style="font-size:10pt;margin:0px;"> </p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">This is Hyunseung Park at Gluesys, South Korea.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;"> </span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">We are trying to replicate the test in https://github.com/gluster/glusterfs/issues/2771 but to no avail.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">In our experiments, Gluster version 10 unfortunately did not perform noticably better than version 9.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;"> </span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"> </p>
<table border="1" cellpadding="0" cellspacing="0" dir="ltr" style="table-layout:fixed;font-size:10pt;font-family:Arial;width:0px;border-collapse:collapse;border:none" xmlns="http://www.w3.org/1999/xhtml">
<colgroup>
<col width="100" />
<col width="100" />
<col width="100" />
<col width="100" />
<col width="100" />
</colgroup>
<tbody>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">v9 2x2</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">average</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">create</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3399.99</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3484.79</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2702.57</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3195.783333</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">ls -l</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">65605.2</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">64930.6</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">72018.7</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">67518.16667</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">chmod</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">4858.95</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">4965.29</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5597.73</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5140.656667</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">stat</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7334.88</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7755.89</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">8335.11</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7808.626667</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">read</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7015.64</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">8255.48</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7007.01</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7426.043333</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">append</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2554.93</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2777.65</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2572.57</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2635.05</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">mkdir</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1800.29</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1865.07</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1805.48</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1823.613333</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">rmdir</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1854.09</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1722.89</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1876.81</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1817.93</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">cleanup</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2402.02</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2447.36</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2438.71</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2429.363333</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">v10 2x2</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;"> </span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">average</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">create</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3741.39</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3174.82</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3234.42</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3383.543333</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">ls -l</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">71543.7</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">67275.9</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">72975.1</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">70598.23333</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">chmod</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5441.11</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5109.22</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5004.08</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">5184.803333</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">stat</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7746.37</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7677.99</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7885.72</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7770.026667</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">read</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7061.12</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7165.21</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7121.07</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">7115.8</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">append</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3458.93</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2641.84</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2887.46</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2996.076667</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">mkdir</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2685.22</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1879.35</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1970.91</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2178.493333</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">rmdir</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2240.11</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1648.37</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1602.16</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">1830.213333</span></td>
</tr>
<tr style="height:21px;">
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;"><span style="font-family:Arial;">cleanup</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">3739.68</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2407.57</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2403.48</span></td>
<td style="overflow:hidden;padding:2px 3px 2px 3px;vertical-align:bottom;text-align:right;"><span style="font-family:Arial;">2850.243333</span></td>
</tr>
</tbody>
</table>
<p> </p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">The result above is from the test that deployed 32 threads on each of the 4 clients.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Some results were better than others, but it is not good enough when compared to the result in the aforementioned link.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">We are wondering what we can do to get the full potential of the new version.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"> </p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">We have been running tests with varying file sizes, number of threads, different volume topology, etc. but we were not able to see data conclusive enough.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">We were also not able to find meaningful output from running tests using other benchmark tools such as bonnie++ and FIO.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"> </p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">To find the potential cause we tried to look into the program by calling malloc_stats() and using perf.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">However, we also could not find something noteworthy from the result.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Here is the data recorded during one set of smallfile test (from create to cleanup): https://drive.google.com/drive/folders/1NMXNjgOZ7svDd4-YvKCU4UAp43tm15dC?usp=sharing</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;"> </span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Below is our test environment:</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;"> </span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Basic HW info: VM (vSphere), 2 core CPU, 4G RAM. 4 servers and 4 clients.</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">OS: Centos 7</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">kernel version: 3.10.0-1160</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">Gluster version: built from git, checked out from branch 'release-9' and 'release-10' respectively (commits 7094da and a804f8)</span></p>
<p style="margin:0px; font-family:맑은 고딕; font-size:10pt"><span style="font-family:Arial;">build option: default setting except "./configure --disable-linux-io_uring" (kernel does not support io_uring)</span></p>
<div style="font-family:맑은 고딕; font-size:10pt">
<p><span style="font-family:Arial;">Configuration result (case of version 10):</span></p>
<p><span style="font-family:Arial;">GlusterFS configure summary<br />
===========================<br />
FUSE client : yes<br />
epoll IO multiplex : yes<br />
fusermount : yes<br />
readline : no<br />
georeplication : yes<br />
Linux-AIO : yes<br />
Linux io_uring : no<br />
Use liburing : no<br />
Enable Debug : no<br />
Run with Valgrind : no<br />
Sanitizer enabled : none<br />
XML output : yes<br />
Unit Tests : no<br />
Track priv ports : yes<br />
POSIX ACLs : yes<br />
SELinux features : yes<br />
firewalld-config : no<br />
Events : yes<br />
EC dynamic support : x64 sse avx<br />
Use memory pools : no<br />
Nanosecond m/atimes : yes<br />
Server components : yes<br />
Legacy gNFS server : no<br />
IPV6 default : no<br />
Use TIRPC : yes<br />
With Python : 3.6<br />
Cloudsync : yes<br />
Metadata dispersal : no<br />
Link with TCMALLOC : yes<br />
Enable Brick Mux : no<br />
Building with LTO : no</span></p>
<p><span style="font-family:Arial;"> </span></p>
<p><span style="font-family:Arial;"> </span></p>
</div>
<p style="margin:0px;font-family:맑은 고딕; font-size:10pt"> </p>