<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hi David,</div><div><br></div><div>i haven't tested samba but glusterfs fuse, i have posted the results few months ago, tests conducted using gluster 4.1.5:</div><div><br><b>Options Reconfigured:</b><br>client.event-threads 3<br>performance.cache-size 8GB<br>performance.io-thread-count 24<br>network.inode-lru-limit 1048576<br>performance.parallel-readdir on<br>performance.cache-invalidation on<br>performance.md-cache-timeout 600<br>features.cache-invalidation on<br>features.cache-invalidation-timeout 600<br>performance.client-io-threads on</div><div><br></div><div><table dir="ltr" style="table-layout:fixed;font-size:10pt;font-family:arial,sans,sans-serif;width:0px;border-collapse:collapse;border:medium none" cellspacing="0" cellpadding="0" border="1"><colgroup><col width="234"><col width="165"><col width="100"></colgroup><tbody><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">nr of clients</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">6</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Network</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">10Gb</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Clients Mem</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">128GB</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Clients Cores</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">22</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Centos</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">7.5.1804</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Kernel</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">3.10.0-862.14.4.el7.x86_64</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">nr of servers/bricks per volume</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">3</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Network</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">100Gb</td><td style="border-color:rgb(204,204,204) transparent rgb(204,204,204) rgb(204,204,204);border-style:solid;border-width:1px;overflow:visible;padding:2px 0px;vertical-align:bottom"><div style="white-space:nowrap;overflow:hidden;width:297px"><div style="float:left">*node to node is 100Gb, cleints 10Gb</div></div></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Server Mem</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">377GB</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Server Cores</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">56</td><td style="overflow:hidden;padding:2px 0px;vertical-align:bottom;border:1px solid rgb(204,204,204)">*Intel 5120 CPU</td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Storage</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">4x8TB NVME</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Centos</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">7.5.1804</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr><tr style="height:21px"><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;font-weight:bold;border:1px solid rgb(204,204,204)">Kernel</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;text-align:right;border:1px solid rgb(204,204,204)">3.10.0-862.14.4.el7.x86_64</td><td style="overflow:hidden;padding:2px 3px;vertical-align:bottom;border:1px solid rgb(204,204,204)"><br></td></tr></tbody></table></div><div><br></div><div>This for example are FOPS with 128K IO size (cnsidered sweet spot for glusterfs according to documentation). In Blue 8threads per client and red 4threads for client<br></div><div><div><img src="cid:ii_jr7hj3cu0" alt="image.png" width="535" height="371"></div><div>Below 4K</div><div><div><img src="cid:ii_jr7hqsu01" alt="image.png" width="535" height="371"></div><div>and 1MB</div><div><div><img src="cid:ii_jr7hr4j02" alt="image.png" width="535" height="371"><br></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_5469492897407800284gmail_attr">On Tue, Jan 22, 2019 at 9:09 AM David Spisla <<a href="mailto:spisla80@gmail.com" target="_blank">spisla80@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello Amar,</div><div>thank you for the advice. We already use nl-cache option and a bunch of other settings. At the moment we try the samba-vfs-glusterfs plugin to access a gluster volume via samba. The performance increase now. <br></div><div>Additionally we are looking for some performance measurements to compare with. Maybe someone in the community also does performance tests. Does Redhat has some official reference measurement?</div><div><br></div><div>Regards</div><div>David Spisla<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_5469492897407800284gmail-m_1491705826103984582gmail_attr">Am Di., 22. Jan. 2019 um 07:14 Uhr schrieb Amar Tumballi Suryanarayan <<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">For Samba usecase, please make sure you have nl-cache (ie, 'negative-lookup cache') enabled. We have seen some improvements from this value.<div><br></div><div>-Amar</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_5469492897407800284gmail-m_1491705826103984582gmail-m_-5162899884277905295gmail_attr">On Tue, Dec 18, 2018 at 8:23 PM David Spisla <<a href="mailto:spisla80@gmail.com" target="_blank">spisla80@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Dear Gluster Community,</div><div><br></div><div>it is a known fact that Samba+Gluster has a bad smallfile performance. We now have some test measurements created by this setup: 2-Node-Cluster on real hardware with Replica-2 Volume (just one subvolume), Gluster v.4.1.6, Samba v4.7. Samba writes to Gluster via FUSE. Files created by fio. We used a Windows System as Client which is in the same network like the servers.<br></div><div><br></div><div>The measurements are as follows. In each test case 400 files were written:</div><div><br></div><div> 64KiB_x_400 files 1MiB_x_400 files 10MiB_x_400 files<br></div><div>1 Thread 0,77 MiB/s 8,05 MiB/s 72,67 MiB/s<br></div><div>4 Threads 0,86 MiB/s 8,92 MiB/s 90,38 MiB/s <br></div><div>8 Threads 0,87 MiB/s 8,92 MiB/s 94,75 MiB/s</div><div><br></div><div>
<pre class="gmail-m_5469492897407800284gmail-m_1491705826103984582gmail-m_-5162899884277905295gmail-m_-6519672765833847115gmail-tw-data-text gmail-m_5469492897407800284gmail-m_1491705826103984582gmail-m_-5162899884277905295gmail-m_-6519672765833847115gmail-tw-ta gmail-m_5469492897407800284gmail-m_1491705826103984582gmail-m_-5162899884277905295gmail-m_-6519672765833847115gmail-tw-text-small" id="gmail-m_5469492897407800284gmail-m_1491705826103984582gmail-m_-5162899884277905295gmail-m_-6519672765833847115gmail-tw-target-text" style="text-align:left;height:216px" dir="ltr"><span style="font-family:arial,helvetica,sans-serif"><span lang="en">Does anyone have measurements that are in a similar range or are significantly different?<br>We do not know which values can still be considered "normal" and which are not. <br>We also know that there are options to improve performance. But first of all we are interested <br>in whether there are reference values.<br>
<span style="font-family:arial,helvetica,sans-serif"><span lang="en">Regards<br>David Spisla</span></span></span></span></pre></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_5469492897407800284gmail-m_1491705826103984582gmail-m_-5162899884277905295gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</blockquote></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_5469492897407800284gmail_signature"><span style="display:block;font-size:11pt;font-family:Century Gothic;color:rgb(0,53,128)"><div style="color:rgb(0,53,128);font-family:Arial,Helvetica,sans-serif;font-weight:bold;font-size:15px"><div>Davide Obbi</div><div style="font-weight:normal;font-size:13px;color:rgb(0,174,239)">Senior System Administrator<br><br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">Booking.com B.V.<br>Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><span style="color:rgb(0,174,239)">Direct </span>+31207031558<br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><div style="font-weight:bold;font-size:16px;color:rgb(0,53,128)"><a href="https://www.booking.com/" style="color:rgb(0,127,255);background-image:initial;background-position:initial;background-repeat:initial" target="_blank"><img src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com" title="Booking.com"></a></div><div><span style="font-size:11px">Empowering people to experience the world since 1996</span></div><div><span style="font-size:11px">43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 million reported listings <br></span><span style="font-size:11px">Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span></div></div></div></span></div>