<div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Dear Gluster Community,</div><div><br></div><div>I also have a issue concerning performance. The last days I updated our test cluster from GlusterFS v5.5 to v7.0 . The setup in general:</div><div><br></div><div>2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with 2 Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did several tests to ensure that Samba don't causes the fall.</div><div>The setup ist completely the same except the Gluster Version<br></div><div>Here are my results:</div><div>64KiB 1MiB 10MiB (Filesize)</div><div>
<div style="box-sizing:border-box;font-family:"Segoe UI","Helvetica Neue","Apple Color Emoji","Segoe UI Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"></div><div style="box-sizing:border-box;font-family:"Segoe UI","Helvetica Neue","Apple Color Emoji","Segoe UI Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div style="box-sizing:border-box;font-family:"Segoe UI","Helvetica Neue","Apple Color Emoji","Segoe UI Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">3,49 47,41 300,50
(Values in MiB/s with GlusterFS v5.5) <br></div><div style="box-sizing:border-box;font-family:"Segoe UI","Helvetica Neue","Apple Color Emoji","Segoe UI Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
0,16 2,61 76,63 (Values in MiB/s with GlusterFS v7.0) <br></div><div style="box-sizing:border-box;font-family:"Segoe UI","Helvetica Neue","Apple Color Emoji","Segoe UI Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="box-sizing:border-box;font-family:"Segoe UI","Helvetica Neue","Apple Color Emoji","Segoe UI Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">We use this volume options (GlusterFS 7.0):</div><div style="box-sizing:border-box;font-family:"Segoe UI","Helvetica Neue","Apple Color Emoji","Segoe UI Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br></div><div style="box-sizing:border-box;font-family:"Segoe UI","Helvetica Neue","Apple Color Emoji","Segoe UI Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Volume Name: archive1<br>Type: Distributed-Replicate<br>Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 2 x 2 = 4<br>Transport-type: tcp<br>Bricks:<br>Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick<br>Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick<br>Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick<br>Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick<br>Options Reconfigured:<br>performance.client-io-threads: off<br>nfs.disable: on<br>storage.fips-mode-rchecksum: on<br>transport.address-family: inet<br>user.smb: disable<br>features.read-only: off<br>features.worm: off<br>features.worm-file-level: on<br>features.retention-mode: enterprise<br>features.default-retention-period: 120<br>network.ping-timeout: 10<br>features.cache-invalidation: on<br>features.cache-invalidation-timeout: 600<br>performance.nl-cache: on<br>performance.nl-cache-timeout: 600<br>client.event-threads: 32<br>server.event-threads: 32<br>cluster.lookup-optimize: on<br>performance.stat-prefetch: on<br>performance.cache-invalidation: on<br>performance.md-cache-timeout: 600<br>performance.cache-samba-metadata: on<br>performance.cache-ima-xattrs: on<br>performance.io-thread-count: 64<br>cluster.use-compound-fops: on<br>performance.cache-size: 512MB<br>performance.cache-refresh-timeout: 10<br>performance.read-ahead: off<br>performance.write-behind-window-size: 4MB<br>performance.write-behind: on<br>storage.build-pgfid: on<br>features.ctime: on<br>cluster.quorum-type: fixed<br>cluster.quorum-count: 1<br>features.bitrot: on<br>features.scrub: Active<br>features.scrub-freq: daily<br></div>
</div>
</div><div><br></div><div>For GlusterFS 5.5 its nearly the same except the fact that there were 2 options to enable ctime feature. <br></div><div>Our optimization for Samba looks like this (for every version):</div><div><br></div><div>[global]<br> workgroup = SAMBA<br> netbios name = CLUSTER<br> kernel share modes = no<br> aio read size = 1<br> aio write size = 1<br> kernel oplocks = no<br> max open files = 100000<br> nt acl support = no<br> security = user<br> server min protocol = SMB2<br> store dos attributes = no<br> strict locking = no<br> full_audit:failure = pwrite_send pwrite_recv pwrite offload_write_send offload_write_recv create_file open unlink connect disconnect rename chown fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate <br> full_audit:success = pwrite_send pwrite_recv pwrite offload_write_send offload_write_recv create_file open unlink connect disconnect rename chown fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate <br> full_audit:facility = local5<br> durable handles = yes<br> posix locking = no<br> log level = 2<br> max log size = 100000<br> debug pid = yes<br></div><div><br></div><div>What can be the cause for this rapid falling of the performance for small files? Are some of our vol options not recommended anymore? <br></div><div>There were some patches concerning performance for small files in v6.0 und v7.0 :<br></div><div>
<p class="MsoNormal" style="margin:0cm 0cm 0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:9pt;font-family:"Verdana",sans-serif"><a href="https://bugzilla.redhat.com/1670031" style="color:blue;text-decoration:underline"><span style="font-size:12pt;font-family:"\000026quot",serif;color:rgb(63,81,181);text-decoration:none">#1670031</span></a></span><span style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white none repeat scroll 0% 0%"><span style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px">: performance regression seen with smallfile
workload tests</span></span><span style="font-size:9pt;font-family:"Verdana",sans-serif"><span></span></span></p>
</div><div>
<p style="margin:12pt 0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:12pt;font-family:"\000026quot",serif"><a href="https://bugzilla.redhat.com/1659327" style="color:blue;text-decoration:underline"><span style="color:rgb(63,81,181)">#1659327</span></a></span><span style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px"><span style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white none repeat scroll 0% 0%">: 43% regression in small-file sequential read performance</span></span></p><p style="margin:12pt 0cm;font-size:11pt;font-family:"Calibri",sans-serif">And one patch for the io-cache:</p><p style="margin:12pt 0cm;font-size:11pt;font-family:"Calibri",sans-serif">
</p><p style="margin:12pt 0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="font-size:12pt;font-family:"\000026quot",serif"><a href="https://bugzilla.redhat.com/1659869" style="color:blue;text-decoration:underline"><span style="color:rgb(63,81,181)">#1659869</span></a></span><span style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px"><span style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white none repeat scroll 0% 0%">: improvements to io-cache</span></span></p><p style="margin:12pt 0cm;font-size:11pt;font-family:"Calibri",sans-serif">Regards</p><p style="margin:12pt 0cm;font-size:11pt;font-family:"Calibri",sans-serif">David Spisla<br></p><p style="margin:12pt 0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px"><span style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white none repeat scroll 0% 0%"></span></span><span style="font-size:12pt;font-family:"\000026quot",serif"><span></span></span></p>
<p style="margin:12pt 0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px"><span style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white none repeat scroll 0% 0%"></span></span><span style="font-size:12pt;font-family:"\000026quot",serif"><span></span></span></p>
</div></div></div></div>