<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<div class="moz-cite-prefix">On 11/4/19 8:46 PM, David Spisla wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJyj9j_Of=_Fsh4oTYZYzt-Z3p6vC0CYTiZaGHZ=fYRCJFqsdg@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Dear Gluster Community,</div>
<div><br>
</div>
<div>I also have a issue concerning performance. The last
days I updated our test cluster from GlusterFS v5.5 to
v7.0 . The setup in general:</div>
<div><br>
</div>
<div>2 HP DL380 Servers with 10Gbit NICs, 1
Distribute-Replica 2 Volume with 2 Replica Pairs. Client
is SMB Samba (access via vfs_glusterfs) . I did several
tests to ensure that Samba don't causes the fall.</div>
<div>The setup ist completely the same except the Gluster
Version<br>
</div>
<div>Here are my results:</div>
<div>64KiB 1MiB 10MiB
(Filesize)</div>
<div>
<div style="box-sizing:border-box;font-family:"Segoe
UI","Helvetica Neue","Apple Color
Emoji","Segoe UI
Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div
style="box-sizing:border-box;font-family:"Segoe
UI","Helvetica Neue","Apple Color
Emoji","Segoe UI
Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">3,49
47,41 300,50 (Values in
MiB/s with GlusterFS v5.5) <br>
</div>
<div
style="box-sizing:border-box;font-family:"Segoe
UI","Helvetica Neue","Apple Color
Emoji","Segoe UI
Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">0,16
2,61 76,63 (Values
in MiB/s with GlusterFS v7.0) <br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p>Can you please share the profile information [1] for both
versions? Also it would be really helpful if you can mention the
io patterns that used for this tests.<br>
</p>
<p>[1] :
<a class="moz-txt-link-freetext" href="https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/">https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/</a><br>
</p>
<blockquote type="cite"
cite="mid:CAJyj9j_Of=_Fsh4oTYZYzt-Z3p6vC0CYTiZaGHZ=fYRCJFqsdg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>
<div style="box-sizing:border-box;font-family:"Segoe
UI","Helvetica Neue","Apple Color
Emoji","Segoe UI
Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div
style="box-sizing:border-box;font-family:"Segoe
UI","Helvetica Neue","Apple Color
Emoji","Segoe UI
Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br>
</div>
<div
style="box-sizing:border-box;font-family:"Segoe
UI","Helvetica Neue","Apple Color
Emoji","Segoe UI
Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">We
use this volume options (GlusterFS 7.0):</div>
<div
style="box-sizing:border-box;font-family:"Segoe
UI","Helvetica Neue","Apple Color
Emoji","Segoe UI
Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br>
</div>
<div
style="box-sizing:border-box;font-family:"Segoe
UI","Helvetica Neue","Apple Color
Emoji","Segoe UI
Emoji",Helvetica,Arial,sans-serif;font-size:14px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Volume
Name: archive1<br>
Type: Distributed-Replicate<br>
Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 2 x 2 = 4<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick<br>
Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick<br>
Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick<br>
Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick<br>
Options Reconfigured:<br>
performance.client-io-threads: off<br>
nfs.disable: on<br>
storage.fips-mode-rchecksum: on<br>
transport.address-family: inet<br>
user.smb: disable<br>
features.read-only: off<br>
features.worm: off<br>
features.worm-file-level: on<br>
features.retention-mode: enterprise<br>
features.default-retention-period: 120<br>
network.ping-timeout: 10<br>
features.cache-invalidation: on<br>
features.cache-invalidation-timeout: 600<br>
performance.nl-cache: on<br>
performance.nl-cache-timeout: 600<br>
client.event-threads: 32<br>
server.event-threads: 32<br>
cluster.lookup-optimize: on<br>
performance.stat-prefetch: on<br>
performance.cache-invalidation: on<br>
performance.md-cache-timeout: 600<br>
performance.cache-samba-metadata: on<br>
performance.cache-ima-xattrs: on<br>
performance.io-thread-count: 64<br>
cluster.use-compound-fops: on<br>
performance.cache-size: 512MB<br>
performance.cache-refresh-timeout: 10<br>
performance.read-ahead: off<br>
performance.write-behind-window-size: 4MB<br>
performance.write-behind: on<br>
storage.build-pgfid: on<br>
features.ctime: on<br>
cluster.quorum-type: fixed<br>
cluster.quorum-count: 1<br>
features.bitrot: on<br>
features.scrub: Active<br>
features.scrub-freq: daily<br>
</div>
</div>
</div>
<div><br>
</div>
<div>For GlusterFS 5.5 its nearly the same except the fact
that there were 2 options to enable ctime feature. <br>
</div>
</div>
</div>
</div>
</blockquote>
<p><br>
</p>
<p><br>
Ctime stores additional metadata information as an extended
attributes which sometimes exceeds the default inode size. In such
scenarios the additional xattrs won't fit into the default size.
This will result in additional blocks to be used to store xattrs
in the inide, which will effect the latency. This is purely based
on the i/o operations and the total xattrs size stored in the
inode.<br>
<br>
Is it possible for you to repeat the test by disabling ctime or
increasing the inode size to a higher value say 1024KB?<br>
</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:CAJyj9j_Of=_Fsh4oTYZYzt-Z3p6vC0CYTiZaGHZ=fYRCJFqsdg@mail.gmail.com">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div>Our optimization for Samba looks like this (for every
version):</div>
<div><br>
</div>
<div>[global]<br>
workgroup = SAMBA<br>
netbios name = CLUSTER<br>
kernel share modes = no<br>
aio read size = 1<br>
aio write size = 1<br>
kernel oplocks = no<br>
max open files = 100000<br>
nt acl support = no<br>
security = user<br>
server min protocol = SMB2<br>
store dos attributes = no<br>
strict locking = no<br>
full_audit:failure = pwrite_send pwrite_recv pwrite
offload_write_send offload_write_recv create_file open
unlink connect disconnect rename chown fchown lchown chmod
fchmod mkdir rmdir ntimes ftruncate fallocate <br>
full_audit:success = pwrite_send pwrite_recv pwrite
offload_write_send offload_write_recv create_file open
unlink connect disconnect rename chown fchown lchown chmod
fchmod mkdir rmdir ntimes ftruncate fallocate <br>
full_audit:facility = local5<br>
durable handles = yes<br>
posix locking = no<br>
log level = 2<br>
max log size = 100000<br>
debug pid = yes<br>
</div>
<div><br>
</div>
<div>What can be the cause for this rapid falling of the
performance for small files? Are some of our vol options
not recommended anymore? <br>
</div>
<div>There were some patches concerning performance for
small files in v6.0 und v7.0 :<br>
</div>
<div>
<p class="MsoNormal" style="margin:0cm 0cm
0.0001pt;font-size:11pt;font-family:"Calibri",sans-serif"><span
style="font-size:9pt;font-family:"Verdana",sans-serif"><a
href="https://bugzilla.redhat.com/1670031"
style="color:blue;text-decoration:underline"
moz-do-not-send="true"><span
style="font-size:12pt;font-family:"\000026quot",serif;color:rgb(63,81,181);text-decoration:none">#1670031</span></a></span><span
style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white
none repeat scroll 0% 0%"><span
style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px">:
performance regression seen with smallfile
workload tests</span></span><span
style="font-size:9pt;font-family:"Verdana",sans-serif"><span></span></span></p>
</div>
<div>
<p style="margin:12pt
0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span
style="font-size:12pt;font-family:"\000026quot",serif"><a
href="https://bugzilla.redhat.com/1659327"
style="color:blue;text-decoration:underline"
moz-do-not-send="true"><span
style="color:rgb(63,81,181)">#1659327</span></a></span><span
style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px"><span
style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white
none repeat scroll 0% 0%">: 43% regression in
small-file sequential read performance</span></span></p>
<p style="margin:12pt
0cm;font-size:11pt;font-family:"Calibri",sans-serif">And
one patch for the io-cache:</p>
<p style="margin:12pt
0cm;font-size:11pt;font-family:"Calibri",sans-serif">
</p>
<p style="margin:12pt
0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span
style="font-size:12pt;font-family:"\000026quot",serif"><a
href="https://bugzilla.redhat.com/1659869"
style="color:blue;text-decoration:underline"
moz-do-not-send="true"><span
style="color:rgb(63,81,181)">#1659869</span></a></span><span
style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px"><span
style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white
none repeat scroll 0% 0%">: improvements to io-cache</span></span></p>
<p style="margin:12pt
0cm;font-size:11pt;font-family:"Calibri",sans-serif">Regards</p>
<p style="margin:12pt
0cm;font-size:11pt;font-family:"Calibri",sans-serif">David
Spisla<br>
</p>
<p style="margin:12pt
0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span
style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px"><span
style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white
none repeat scroll 0% 0%"></span></span><span
style="font-size:12pt;font-family:"\000026quot",serif"><span></span></span></p>
<p style="margin:12pt
0cm;font-size:11pt;font-family:"Calibri",sans-serif"><span
style="color:rgba(0,0,0,0.87);float:none;word-spacing:0px"><span
style="font-size:12pt;font-family:"Helvetica",sans-serif;background:white
none repeat scroll 0% 0%"></span></span><span
style="font-size:12pt;font-family:"\000026quot",serif"><span></span></span></p>
</div>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">________
Community Meeting Calendar:
APAC Schedule -
Every 2nd and 4th Tuesday at 11:30 AM IST
Bridge: <a class="moz-txt-link-freetext" href="https://bluejeans.com/118564314">https://bluejeans.com/118564314</a>
NA/EMEA Schedule -
Every 1st and 3rd Tuesday at 01:00 PM EDT
Bridge: <a class="moz-txt-link-freetext" href="https://bluejeans.com/118564314">https://bluejeans.com/118564314</a>
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</body>
</html>