<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0cm;
margin-bottom:.0001pt;
font-size:8.5pt;
font-family:Menlo;
color:black;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0cm;
margin-bottom:.0001pt;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-SG" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">I have setup oVirt hyperconverged with 3xDL325 with 3 NVMe SSDs on each server for each Gluster volumes: engine, vmstore and data in replica 3.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">I am getting almost 4x performance difference when I am benchmarking with DD on gluster disk:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Below output is from one of the gluster host:<o:p></o:p></span></p>
<p class="p1"><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">time dd if=/dev/zero of=/gluster/vmstore/testfile
bs=1M count=1000 oflag=direct</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1000+0 records in</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1000+0 records out</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1048576000 bytes (1.0 GB) copied, 0.546909 s, 1.9 GB/s</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">real</span></span><span class="apple-tab-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
</span></span><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">0m0.552s</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">user</span></span><span class="apple-tab-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
</span></span><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">0m0.001s</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">sys</span></span><span class="apple-tab-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
</span></span><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">0m0.165s</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt">Below output is from VM.<o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"># time dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1000+0 records in</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1000+0 records out</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1048576000 bytes (1.0 GB) copied, 1.89253 s, 554 MB/s</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">real</span></span><span class="apple-tab-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
</span></span><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">0m1.896s</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">user</span></span><span class="apple-tab-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
</span></span><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">0m0.002s</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">sys</span></span><span class="apple-tab-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
</span></span><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">0m0.247s</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Earlier I was getting around ~270MB in that VM, It improved after setting the
</span><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">engine-config -s LibgfApiSupported=true --cver=4.3</span></span><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">I am using default tuned profile(virtual-host). Backend network is 10GbE, tested with iPerf and it gives above
</span><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">9.40 Gbits/se</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> bandwidth<o:p></o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"># gluster v info vmstore</span></span><span class="apple-converted-space"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> </span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Volume Name: vmstore</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Type: Replicate</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Volume ID: b1f0c1fb-8b8b-479a-9117-61d45692047a</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Status: Started</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Snapshot Count: 0</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Number of Bricks: 1 x 3 = 3</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Transport-type: tcp</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Bricks:</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Brick1: daal-gluster.mbi.nus.edu.sg:/gluster/vmstore/brick1</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Brick2: naan-gluster.mbi.nus.edu.sg:/gluster/vmstore/brick1</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Brick3: dosa-gluster.mbi.nus.edu.sg:/gluster/vmstore/brick1</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Options Reconfigured:</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">server.event-threads: 4</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">client.event-threads: 4</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">performance.client-io-threads: off</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">nfs.disable: on</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">transport.address-family: inet</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">performance.quick-read: off</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">performance.read-ahead: off</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">performance.io-cache: on</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">performance.low-prio-threads: 32</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">network.remote-dio: enable</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cluster.eager-lock: enable</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cluster.quorum-type: auto</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cluster.server-quorum-type: server</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cluster.data-self-heal-algorithm: full</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cluster.locking-scheme: granular</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cluster.shd-max-threads: 8</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">cluster.shd-wait-qlength: 10000</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">features.shard: on</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">user.cifs: off</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">storage.owner-uid: 36</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">storage.owner-gid: 36</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">features.shard-block-size: 512MB</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">performance.cache-size: 1GB</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">performance.io-thread-count: 32</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span class="s1"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">performance.write-behind-window-size: 4MB</span></span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Is there any way I can improve my gluster disk’s performance?<o:p></o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks for helping out.<o:p></o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p> </o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Regards,<o:p></o:p></span></p>
<p class="p1"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">Sandeep</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</div>
<br>
<hr>
<font face="Arial" color="Black" size="1"><br>
Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you.<br>
</font>
</body>
</html>