<html><head></head><body><div class="ydpef75d35cyahoo-style-wrap" style="font-family: courier new, courier, monaco, monospace, sans-serif; font-size: 16px;"><div></div>
<div>Dear Darrell,</div><div><br></div><div>I found the issue and now I can reach the maximum of the network with a fuse client.</div><div>Here is a short overview:</div><div>1. I noticed that working with a new gluster volume is reaching my network speed - I was quite excited.</div><div>2. Then I have destroyed my gluster volume and created a new one and started adding features from ovirt</div><div><br></div><div>Once I have added features.shard on -> I hit the same performance from before. Increasing the shard size to 16MB didn't help at all.</div><div><br></div><div>For my case where I have 2 Virtualization hosts with single data gluster volume - sharding is not neccessary, but for larger setups it will be a problem.</div><div><br></div><div>As this looks to me as a bug - can someone tell me where I can report it ?</div><div><br></div><div>Thanks to all who guided me in this journey of GlusterFS ! I have learned so much , as my prior knowledge was only in Ceph.</div><div><br></div><div>Best Regards,</div><div>Strahil Nikolov<br></div><div><br></div>
</div><div id="ydp658cfa08yahoo_quoted_9323618734" class="ydp658cfa08yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
В четвъртък, 24 януари 2019 г., 17:53:50 ч. Гринуич+2, Darrell Budic <budic@onholyground.com> написа:
</div>
<div><br></div>
<div><br></div>
<div><div id="ydp658cfa08yiv2833557303"><div>Strahil-<div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">The fuse client is what it is, it’s limited by operating in user land and waiting for the gluster servers to acknowledge all the writes. I noted you're using ovirt, you should look into enabling the libgfapi engine setting to run your VMs with libgf natively. You can’t test directly from the host with that, but you can run your tests inside the VMs. I saw significant throughput and latency improvements that way. It’s still somewhat beta, so you’ll probably need to search the overt-users mailing list to find info on enabling it. </div><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">Good luck!</div><div class="ydp658cfa08yiv2833557303"><div><br class="ydp658cfa08yiv2833557303" clear="none"><blockquote class="ydp658cfa08yiv2833557303" type="cite"><div class="ydp658cfa08yiv2833557303yqt8646561959" id="ydp658cfa08yiv2833557303yqt97590"><div class="ydp658cfa08yiv2833557303">On Jan 24, 2019, at 4:32 AM, Strahil Nikolov <<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>> wrote:</div><br class="ydp658cfa08yiv2833557303Apple-interchange-newline" clear="none"><div class="ydp658cfa08yiv2833557303"><div class="ydp658cfa08yiv2833557303"><div class="ydp658cfa08yiv2833557303" style="background-color:rgb(255, 255, 255);"><div class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7272"><span class="ydp658cfa08yiv2833557303">Dear Amar, Community,</span></div><div class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7274"><span class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></span></div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7275"><span class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7299">it seems the issue is in the fuse client itself</span>.</div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7384"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7383">Here is the latest update:</div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7300">1. I have added the following:</div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7385">server.event-threads: 4<br class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7416" clear="none">client.event-threads: 4<br class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7417" clear="none">performance.stat-prefetch: on</div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7610">performance.strict-o-direct: off<br class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7418" clear="none"></div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7608">Results: no change</div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7645"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7662">2. Allowed nfs and connected ovirt1 to the gluster volume:</div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7661">nfs.disable: off<br class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7665" clear="none"></div><div class="ydp658cfa08yiv2833557303" dir="ltr">Results: Drastic improvement in performance as follows:</div><div class="ydp658cfa08yiv2833557303" dir="ltr"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7788">[root@ovirt1 data]# dd if=/dev/zero of=largeio bs=1M count=5000 status=progress<br class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7800" clear="none">5000+0 records in<br class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7801" clear="none">5000+0 records out<br class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7802" clear="none">5242880000 bytes (5.2 GB) copied, 53.0443 s, 98.8 MB/s</div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7841"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7961">So I would be happy if anyone guide me in order to fix the situation as the fuse client is the best way to use glusterfs, and it seems the glusterfs-server is not the guilty one.</div><div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7962"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303" dir="ltr">Thanks in advance for your guidance.I have learned so much.</div><div class="ydp658cfa08yiv2833557303" dir="ltr"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303" dir="ltr">Best Regards,</div><div class="ydp658cfa08yiv2833557303" dir="ltr">Strahil Nikolov<br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303qtdSeparateBR"><br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303yahoo_quoted" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7305" style="display:block;"> <div class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7304" style="font-family:Courier New, courier, monaco, monospace, sans-serif;font-size:16px;"> <div class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7303" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;"> <div class="ydp658cfa08yiv2833557303" dir="ltr" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7302"> <font class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7301" size="2" face="Arial"> </font><hr class="ydp658cfa08yiv2833557303" size="1"> <b class="ydp658cfa08yiv2833557303"><span class="ydp658cfa08yiv2833557303" style="font-weight:bold;">От:</span></b> Strahil <<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>><br class="ydp658cfa08yiv2833557303" clear="none"> <b class="ydp658cfa08yiv2833557303"><span class="ydp658cfa08yiv2833557303" style="font-weight:bold;">До:</span></b> Amar Tumballi Suryanarayan <<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:atumball@redhat.com" rel="nofollow" target="_blank">atumball@redhat.com</a>> <br class="ydp658cfa08yiv2833557303" clear="none"><b class="ydp658cfa08yiv2833557303"><span class="ydp658cfa08yiv2833557303" style="font-weight:bold;">Копие:</span></b> Gluster-users <<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:gluster-users@gluster.org" rel="nofollow" target="_blank">gluster-users@gluster.org</a>><br class="ydp658cfa08yiv2833557303" clear="none"> <b class="ydp658cfa08yiv2833557303"><span class="ydp658cfa08yiv2833557303" style="font-weight:bold;">Изпратен:</span></b> сряда, 23 януари 2019 г. 18:44<br class="ydp658cfa08yiv2833557303" clear="none"> <b class="ydp658cfa08yiv2833557303"><span class="ydp658cfa08yiv2833557303" style="font-weight:bold;">Тема:</span></b> Re: [Gluster-users] Gluster performance issues - need advise<br class="ydp658cfa08yiv2833557303" clear="none"> </div> <div class="ydp658cfa08yiv2833557303y_msg_container" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7382"><br class="ydp658cfa08yiv2833557303" clear="none"><div class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303"><div class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7381"><div class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7380"><div class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7379">Dear Amar,<div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303" id="ydp658cfa08yiv2833557303yui_3_16_0_ym19_1_1548325460022_7378">Thanks for your email.</div><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">Actually my concerns were on both topics.</div><div class="ydp658cfa08yiv2833557303">Would you recommend any perf options that will be suitable ?</div><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">After mentioning the network usage, I just checked it and it seems duringthe test session, ovirt1 (both client and host) is using no more than 455Mbit/s which is half the network bandwidth.</div><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">I'm still in the middle of nowhere, so any ideas are welcome.</div><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">Best Regards,</div><div class="ydp658cfa08yiv2833557303">Strahil Nikolov</div></div><div class="ydp658cfa08yiv2833557303yqt8450526150" id="ydp658cfa08yiv2833557303yqt03107"><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"><div class="ydp658cfa08yiv2833557303elided-text">On Jan 23, 2019 17:49, Amar Tumballi Suryanarayan <<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:atumball@redhat.com" rel="nofollow" target="_blank">atumball@redhat.com</a>> wrote:<br class="ydp658cfa08yiv2833557303" clear="none"><blockquote class="ydp658cfa08yiv2833557303" style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex;"><div class="ydp658cfa08yiv2833557303" dir="ltr">I didn't understand the issue properly. Mostly I missed something.<div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">Are you concerned the performance is 49MB/s with and without perf options? or are you expecting it to be 123MB/s as over the n/w you get that speed?</div><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">If it is the first problem, then you are actually having 'performance.write-behind on' in both options, and it is the only perf xlator which comes into action during the test you ran.</div><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">If it is the second, then please be informed that gluster does client side replication, which means, n/w would be split in half for write operations (like write(), creat() etc), so the number you are getting is almost the maximum with 1GbE.</div><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div><div class="ydp658cfa08yiv2833557303">Regards,</div><div class="ydp658cfa08yiv2833557303">Amar</div></div><br class="ydp658cfa08yiv2833557303" clear="none"><div class="ydp658cfa08yiv2833557303elided-text"><div class="ydp658cfa08yiv2833557303" dir="ltr">On Wed, Jan 23, 2019 at 8:38 PM Strahil Nikolov <<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br class="ydp658cfa08yiv2833557303" clear="none"></div><blockquote class="ydp658cfa08yiv2833557303" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb( 204 , 204 , 204 );padding-left:1ex;">Hello Community,<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
recently I have built a new lab based on oVirt and CentOS 7.<br class="ydp658cfa08yiv2833557303" clear="none">
During deployment I had some hicups, but now the engine is up and running - but gluster is causing me trouble.<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Symptoms: Slow VM install from DVD, poor write performance. The latter has been tested via:<br class="ydp658cfa08yiv2833557303" clear="none">
dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_data bs=1M count=1000 status=progress<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
The reported speed is 60MB/s which is way too low for my setup.<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
My lab design:<br class="ydp658cfa08yiv2833557303" clear="none">
<a shape="rect" class="ydp658cfa08yiv2833557303" href="https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharing" rel="nofollow" target="_blank">https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharing</a><br class="ydp658cfa08yiv2833557303" clear="none">
Gluster version is 3.12.15<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
So far I have done:<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
1. Added 'server.allow-insecure on' (with 'option rpc-auth-allow-insecure on' in glusterd.vol)<br class="ydp658cfa08yiv2833557303" clear="none">
Volume info after that change:<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Volume Name: data<br class="ydp658cfa08yiv2833557303" clear="none">
Type: Replicate<br class="ydp658cfa08yiv2833557303" clear="none">
Volume ID: 9b06a1e9-8102-4cd7-bc56-84960a1efaa2<br class="ydp658cfa08yiv2833557303" clear="none">
Status: Started<br class="ydp658cfa08yiv2833557303" clear="none">
Snapshot Count: 0<br class="ydp658cfa08yiv2833557303" clear="none">
Number of Bricks: 1 x (2 + 1) = 3<br class="ydp658cfa08yiv2833557303" clear="none">
Transport-type: tcp<br class="ydp658cfa08yiv2833557303" clear="none">
Bricks:<br class="ydp658cfa08yiv2833557303" clear="none">
Brick1: ovirt1.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">
Brick2: ovirt2.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">
Brick3: ovirt3.localdomain:/gluster_bricks/data/data (arbiter)<br class="ydp658cfa08yiv2833557303" clear="none">
Options Reconfigured:<br class="ydp658cfa08yiv2833557303" clear="none">
performance.client-io-threads: off<br class="ydp658cfa08yiv2833557303" clear="none">
nfs.disable: on<br class="ydp658cfa08yiv2833557303" clear="none">
transport.address-family: inet<br class="ydp658cfa08yiv2833557303" clear="none">
performance.quick-read: off<br class="ydp658cfa08yiv2833557303" clear="none">
performance.read-ahead: off<br class="ydp658cfa08yiv2833557303" clear="none">
<a shape="rect" class="ydp658cfa08yiv2833557303" href="http://performance.io" rel="nofollow" target="_blank">performance.io</a>-cache: off<br class="ydp658cfa08yiv2833557303" clear="none">
performance.low-prio-threads: 32<br class="ydp658cfa08yiv2833557303" clear="none">
network.remote-dio: off<br class="ydp658cfa08yiv2833557303" clear="none">
cluster.eager-lock: enable<br class="ydp658cfa08yiv2833557303" clear="none">
cluster.quorum-type: auto<br class="ydp658cfa08yiv2833557303" clear="none">
cluster.server-quorum-type: server<br class="ydp658cfa08yiv2833557303" clear="none">
cluster.data-self-heal-algorithm: full<br class="ydp658cfa08yiv2833557303" clear="none">
cluster.locking-scheme: granular<br class="ydp658cfa08yiv2833557303" clear="none">
cluster.shd-max-threads: 8<br class="ydp658cfa08yiv2833557303" clear="none">
cluster.shd-wait-qlength: 10000<br class="ydp658cfa08yiv2833557303" clear="none">
features.shard: on<br class="ydp658cfa08yiv2833557303" clear="none">
user.cifs: off<br class="ydp658cfa08yiv2833557303" clear="none">
storage.owner-uid: 36<br class="ydp658cfa08yiv2833557303" clear="none">
storage.owner-gid: 36<br class="ydp658cfa08yiv2833557303" clear="none">
network.ping-timeout: 30<br class="ydp658cfa08yiv2833557303" clear="none">
performance.strict-o-direct: on<br class="ydp658cfa08yiv2833557303" clear="none">
cluster.granular-entry-heal: enable<br class="ydp658cfa08yiv2833557303" clear="none">
server.allow-insecure: on<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Seems no positive or negative effect so far.<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
2. Tested with tmpfs on all bricks -> ovirt1 mounted gluster volume -> max 60MB/s (bs=1M without 'oflag=direct')<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
[root@ovirt1 data]# dd if=/dev/zero of=large_io bs=1M count=4000 status=progress<br class="ydp658cfa08yiv2833557303" clear="none">
4177526784 bytes (4.2 GB) copied, 70.843409 s, 59.0 MB/s<br class="ydp658cfa08yiv2833557303" clear="none">
4000+0 records in<br class="ydp658cfa08yiv2833557303" clear="none">
4000+0 records out<br class="ydp658cfa08yiv2833557303" clear="none">
4194304000 bytes (4.2 GB) copied, 71.1407 s, 59.0 MB/s<br class="ydp658cfa08yiv2833557303" clear="none">
[root@ovirt1 data]# rm -f large_io<br class="ydp658cfa08yiv2833557303" clear="none">
[root@ovirt1 data]# gluster volume profile data info<br class="ydp658cfa08yiv2833557303" clear="none">
Brick: ovirt1.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">
---------------------------------------------------<br class="ydp658cfa08yiv2833557303" clear="none">
Cumulative Stats:<br class="ydp658cfa08yiv2833557303" clear="none">
Block Size: 131072b+ <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Reads: 8 <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Writes: 44968 <br class="ydp658cfa08yiv2833557303" clear="none">
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">
--------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 3 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 35 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 78.00 us 78.00 us 78.00 us 1 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 35.67 us 26.00 us 73.00 us 6 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 324.00 us 324.00 us 324.00 us 1 XATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 45.80 us 38.00 us 54.00 us 10 STAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 227.67 us 216.00 us 242.00 us 3 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 113.38 us 68.00 us 381.00 us 8 READ<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 59.97 us 45.00 us 113.00 us 32 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 24.41 us 13.00 us 89.00 us 161 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 43.43 us 28.00 us 214.00 us 93 STATFS<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 246.35 us 11.00 us 1155.00 us 20 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 283.00 us 233.00 us 353.00 us 18 READDIRP<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 153.23 us 122.00 us 259.00 us 87 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">
0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
0.77 124.24 us 87.00 us 604.00 us 44968 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">
0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">
1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">
96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Duration: 380 seconds<br class="ydp658cfa08yiv2833557303" clear="none">
Data Read: 1048576 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
Data Written: 5894045696 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Interval 0 Stats:<br class="ydp658cfa08yiv2833557303" clear="none">
Block Size: 131072b+ <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Reads: 8 <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Writes: 44968 <br class="ydp658cfa08yiv2833557303" clear="none">
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">
--------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 3 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 35 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 78.00 us 78.00 us 78.00 us 1 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 35.67 us 26.00 us 73.00 us 6 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 324.00 us 324.00 us 324.00 us 1 XATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 45.80 us 38.00 us 54.00 us 10 STAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 227.67 us 216.00 us 242.00 us 3 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 113.38 us 68.00 us 381.00 us 8 READ<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 59.97 us 45.00 us 113.00 us 32 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 24.41 us 13.00 us 89.00 us 161 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 43.43 us 28.00 us 214.00 us 93 STATFS<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 246.35 us 11.00 us 1155.00 us 20 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 283.00 us 233.00 us 353.00 us 18 READDIRP<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 153.23 us 122.00 us 259.00 us 87 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">
0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
0.77 124.24 us 87.00 us 604.00 us 44968 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">
0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">
1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">
96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Duration: 380 seconds<br class="ydp658cfa08yiv2833557303" clear="none">
Data Read: 1048576 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
Data Written: 5894045696 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Brick: ovirt3.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">
---------------------------------------------------<br class="ydp658cfa08yiv2833557303" clear="none">
Cumulative Stats:<br class="ydp658cfa08yiv2833557303" clear="none">
Block Size: 1b+ <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Reads: 0 <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Writes: 39328 <br class="ydp658cfa08yiv2833557303" clear="none">
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">
--------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 2 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 12 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 101.00 us 101.00 us 101.00 us 1 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 51.50 us 20.00 us 81.00 us 4 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 219.50 us 188.00 us 251.00 us 2 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 62.30 us 38.00 us 119.00 us 10 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 24.60 us 12.00 us 64.00 us 40 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">
0.02 176.30 us 10.00 us 765.00 us 10 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.07 63.08 us 39.00 us 133.00 us 78 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">
0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">
0.13 126.89 us 99.00 us 179.00 us 76 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">
0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">
28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">
40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Duration: 189 seconds<br class="ydp658cfa08yiv2833557303" clear="none">
Data Read: 0 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
Data Written: 39328 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Interval 0 Stats:<br class="ydp658cfa08yiv2833557303" clear="none">
Block Size: 1b+ <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Reads: 0 <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Writes: 39328 <br class="ydp658cfa08yiv2833557303" clear="none">
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">
--------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 2 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 12 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 101.00 us 101.00 us 101.00 us 1 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 51.50 us 20.00 us 81.00 us 4 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 219.50 us 188.00 us 251.00 us 2 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 62.30 us 38.00 us 119.00 us 10 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 24.60 us 12.00 us 64.00 us 40 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">
0.02 176.30 us 10.00 us 765.00 us 10 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.07 63.08 us 39.00 us 133.00 us 78 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">
0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">
0.13 126.89 us 99.00 us 179.00 us 76 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">
0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">
28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">
40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Duration: 189 seconds<br class="ydp658cfa08yiv2833557303" clear="none">
Data Read: 0 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
Data Written: 39328 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Brick: ovirt2.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">
---------------------------------------------------<br class="ydp658cfa08yiv2833557303" clear="none">
Cumulative Stats:<br class="ydp658cfa08yiv2833557303" clear="none">
Block Size: 512b+ 131072b+ <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Reads: 0 0 <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Writes: 36 76758 <br class="ydp658cfa08yiv2833557303" clear="none">
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">
--------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 6 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 87 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 36.18 us 22.00 us 62.00 us 11 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 82.56 us 59.00 us 138.00 us 9 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 89.42 us 67.00 us 161.00 us 12 SETATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 272.40 us 235.00 us 296.00 us 5 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 154.28 us 88.00 us 320.00 us 18 XATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 86.69 us 30.00 us 379.00 us 62 STAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 64.30 us 47.00 us 169.00 us 84 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">
0.02 107.34 us 23.00 us 273.00 us 73 READDIRP<br class="ydp658cfa08yiv2833557303" clear="none">
0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.03 128.51 us 27.00 us 338.00 us 96 FSYNC<br class="ydp658cfa08yiv2833557303" clear="none">
0.03 240.75 us 14.00 us 1943.00 us 52 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.04 65.59 us 26.00 us 293.00 us 279 STATFS<br class="ydp658cfa08yiv2833557303" clear="none">
0.06 180.77 us 118.00 us 306.00 us 148 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">
0.14 37.98 us 17.00 us 192.00 us 1598 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">
0.67 91.68 us 12.00 us 1141.00 us 3186 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">
10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">
19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">
25.21 142.92 us 62.00 us 593.00 us 76794 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">
32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Duration: 1206 seconds<br class="ydp658cfa08yiv2833557303" clear="none">
Data Read: 0 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
Data Written: 10060843008 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Interval 0 Stats:<br class="ydp658cfa08yiv2833557303" clear="none">
Block Size: 512b+ 131072b+ <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Reads: 0 0 <br class="ydp658cfa08yiv2833557303" clear="none">
No. of Writes: 36 76758 <br class="ydp658cfa08yiv2833557303" clear="none">
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">
--------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 6 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 87 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 36.18 us 22.00 us 62.00 us 11 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 82.56 us 59.00 us 138.00 us 9 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 89.42 us 67.00 us 161.00 us 12 SETATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.00 272.40 us 235.00 us 296.00 us 5 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 154.28 us 88.00 us 320.00 us 18 XATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 86.69 us 30.00 us 379.00 us 62 STAT<br class="ydp658cfa08yiv2833557303" clear="none">
0.01 64.30 us 47.00 us 169.00 us 84 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">
0.02 107.34 us 23.00 us 273.00 us 73 READDIRP<br class="ydp658cfa08yiv2833557303" clear="none">
0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE<br class="ydp658cfa08yiv2833557303" clear="none">
0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">
0.03 128.51 us 27.00 us 338.00 us 96 FSYNC<br class="ydp658cfa08yiv2833557303" clear="none">
0.03 240.75 us 14.00 us 1943.00 us 52 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">
0.04 65.59 us 26.00 us 293.00 us 279 STATFS<br class="ydp658cfa08yiv2833557303" clear="none">
0.06 180.77 us 118.00 us 306.00 us 148 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">
0.14 37.98 us 17.00 us 192.00 us 1598 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">
0.67 91.66 us 12.00 us 1141.00 us 3186 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">
10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">
11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">
19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">
25.21 142.92 us 62.00 us 593.00 us 76794 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">
32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Duration: 1206 seconds<br class="ydp658cfa08yiv2833557303" clear="none">
Data Read: 0 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
Data Written: 10060843008 bytes<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
This indicates to me that it's not a problem in Disk/LVM/FileSystem layout.<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Most probably I haven't created the volume properly or some option/feature is disabled ?!?<br class="ydp658cfa08yiv2833557303" clear="none">
Network shows OK for a gigabit:<br class="ydp658cfa08yiv2833557303" clear="none">
[root@ovirt1 data]# dd if=/dev/zero status=progress | nc ovirt2 9999<br class="ydp658cfa08yiv2833557303" clear="none">
3569227264 bytes (3.6 GB) copied, 29.001052 s, 123 MB/s^C<br class="ydp658cfa08yiv2833557303" clear="none">
7180980+0 records in<br class="ydp658cfa08yiv2833557303" clear="none">
7180979+0 records out<br class="ydp658cfa08yiv2833557303" clear="none">
3676661248 bytes (3.7 GB) copied, 29.8739 s, 123 MB/s<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
I'm looking for any help... you can share your volume info also.<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Thanks in advance.<br class="ydp658cfa08yiv2833557303" clear="none">
<br class="ydp658cfa08yiv2833557303" clear="none">
Best Regards,<br class="ydp658cfa08yiv2833557303" clear="none">
Strahil Nikolov<br class="ydp658cfa08yiv2833557303" clear="none">
_______________________________________________<br class="ydp658cfa08yiv2833557303" clear="none">
Gluster-users mailing list<br class="ydp658cfa08yiv2833557303" clear="none">
<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:Gluster-users@gluster.org" rel="nofollow" target="_blank">Gluster-users@gluster.org</a><br class="ydp658cfa08yiv2833557303" clear="none">
<a shape="rect" class="ydp658cfa08yiv2833557303" href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="nofollow" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br class="ydp658cfa08yiv2833557303" clear="none">
</blockquote></div><br class="ydp658cfa08yiv2833557303" clear="all"><div class="ydp658cfa08yiv2833557303"><br class="ydp658cfa08yiv2833557303" clear="none"></div>-- <br class="ydp658cfa08yiv2833557303" clear="none"><div class="ydp658cfa08yiv2833557303" dir="ltr"><div class="ydp658cfa08yiv2833557303" dir="ltr"><div class="ydp658cfa08yiv2833557303"><div class="ydp658cfa08yiv2833557303" dir="ltr"><div class="ydp658cfa08yiv2833557303">Amar Tumballi (amarts)<br class="ydp658cfa08yiv2833557303" clear="none"></div></div></div></div></div>
</blockquote></div><br class="ydp658cfa08yiv2833557303" clear="none"></div></div></div></div></div><div class="ydp658cfa08yiv2833557303yqt8450526150" id="ydp658cfa08yiv2833557303yqt99954">Dear Amar,<br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none">Thanks for your email.<br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none">Actually my concerns were on both topics.<br class="ydp658cfa08yiv2833557303" clear="none">Would you recommend any perf options that will be suitable ?<br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none">After mentioning the network usage, I just checked it and it seems duringthe test session, ovirt1 (both client and host) is using no more than 455Mbit/s which is half the network bandwidth.<br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none">I'm still in the middle of nowhere, so any ideas are welcome.<br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none">Best Regards,<br class="ydp658cfa08yiv2833557303" clear="none">Strahil Nikolov<br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none">On Jan 23, 2019 17:49, Amar Tumballi Suryanarayan <<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:atumball@redhat.com" rel="nofollow" target="_blank">atumball@redhat.com</a>> wrote:<br class="ydp658cfa08yiv2833557303" clear="none">><br class="ydp658cfa08yiv2833557303" clear="none">> I didn't understand the issue properly. Mostly I missed something.<br class="ydp658cfa08yiv2833557303" clear="none">><br class="ydp658cfa08yiv2833557303" clear="none">> Are you concerned the performance is 49MB/s with and without perf options? or are you expecting it to be 123MB/s as over the n/w you get that speed?<br class="ydp658cfa08yiv2833557303" clear="none">><br class="ydp658cfa08yiv2833557303" clear="none">> If it is the first problem, then you are actually having 'performance.write-behind on' in both options, and it is the only perf xlator which comes into action during the test you ran.<br class="ydp658cfa08yiv2833557303" clear="none">><br class="ydp658cfa08yiv2833557303" clear="none">> If it is the second, then please be informed that gluster does client side replication, which means, n/w would be split in half for write operations (like write(), creat() etc), so the number you are getting is almost the maximum with 1GbE.<br class="ydp658cfa08yiv2833557303" clear="none">><br class="ydp658cfa08yiv2833557303" clear="none">> Regards,<br class="ydp658cfa08yiv2833557303" clear="none">> Amar<br class="ydp658cfa08yiv2833557303" clear="none">><br class="ydp658cfa08yiv2833557303" clear="none">> On Wed, Jan 23, 2019 at 8:38 PM Strahil Nikolov <<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>> wrote:<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Hello Community,<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> recently I have built a new lab based on oVirt and CentOS 7.<br class="ydp658cfa08yiv2833557303" clear="none">>> During deployment I had some hicups, but now the engine is up and running - but gluster is causing me trouble.<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Symptoms: Slow VM install from DVD, poor write performance. The latter has been tested via:<br class="ydp658cfa08yiv2833557303" clear="none">>> dd if=/dev/zero of=/rhev/data-center/mnt/glusterSD/ovirt1.localdomain:_data bs=1M count=1000 status=progress<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> The reported speed is 60MB/s which is way too low for my setup.<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> My lab design:<br class="ydp658cfa08yiv2833557303" clear="none">>> <a shape="rect" class="ydp658cfa08yiv2833557303" href="https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharing" rel="nofollow" target="_blank">https://drive.google.com/file/d/1SiW21ASPXHRAEuE_jZ50R3FoO-NcnFqT/view?usp=sharing</a><br class="ydp658cfa08yiv2833557303" clear="none">>> Gluster version is 3.12.15<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> So far I have done:<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> 1. Added 'server.allow-insecure on' (with 'option rpc-auth-allow-insecure on' in glusterd.vol)<br class="ydp658cfa08yiv2833557303" clear="none">>> Volume info after that change:<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Volume Name: data<br class="ydp658cfa08yiv2833557303" clear="none">>> Type: Replicate<br class="ydp658cfa08yiv2833557303" clear="none">>> Volume ID: 9b06a1e9-8102-4cd7-bc56-84960a1efaa2<br class="ydp658cfa08yiv2833557303" clear="none">>> Status: Started<br class="ydp658cfa08yiv2833557303" clear="none">>> Snapshot Count: 0<br class="ydp658cfa08yiv2833557303" clear="none">>> Number of Bricks: 1 x (2 + 1) = 3<br class="ydp658cfa08yiv2833557303" clear="none">>> Transport-type: tcp<br class="ydp658cfa08yiv2833557303" clear="none">>> Bricks:<br class="ydp658cfa08yiv2833557303" clear="none">>> Brick1: ovirt1.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">>> Brick2: ovirt2.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">>> Brick3: ovirt3.localdomain:/gluster_bricks/data/data (arbiter)<br class="ydp658cfa08yiv2833557303" clear="none">>> Options Reconfigured:<br class="ydp658cfa08yiv2833557303" clear="none">>> performance.client-io-threads: off<br class="ydp658cfa08yiv2833557303" clear="none">>> nfs.disable: on<br class="ydp658cfa08yiv2833557303" clear="none">>> transport.address-family: inet<br class="ydp658cfa08yiv2833557303" clear="none">>> performance.quick-read: off<br class="ydp658cfa08yiv2833557303" clear="none">>> performance.read-ahead: off<br class="ydp658cfa08yiv2833557303" clear="none">>> <a shape="rect" class="ydp658cfa08yiv2833557303" href="http://performance.io" rel="nofollow" target="_blank">performance.io</a>-cache: off<br class="ydp658cfa08yiv2833557303" clear="none">>> performance.low-prio-threads: 32<br class="ydp658cfa08yiv2833557303" clear="none">>> network.remote-dio: off<br class="ydp658cfa08yiv2833557303" clear="none">>> cluster.eager-lock: enable<br class="ydp658cfa08yiv2833557303" clear="none">>> cluster.quorum-type: auto<br class="ydp658cfa08yiv2833557303" clear="none">>> cluster.server-quorum-type: server<br class="ydp658cfa08yiv2833557303" clear="none">>> cluster.data-self-heal-algorithm: full<br class="ydp658cfa08yiv2833557303" clear="none">>> cluster.locking-scheme: granular<br class="ydp658cfa08yiv2833557303" clear="none">>> cluster.shd-max-threads: 8<br class="ydp658cfa08yiv2833557303" clear="none">>> cluster.shd-wait-qlength: 10000<br class="ydp658cfa08yiv2833557303" clear="none">>> features.shard: on<br class="ydp658cfa08yiv2833557303" clear="none">>> user.cifs: off<br class="ydp658cfa08yiv2833557303" clear="none">>> storage.owner-uid: 36<br class="ydp658cfa08yiv2833557303" clear="none">>> storage.owner-gid: 36<br class="ydp658cfa08yiv2833557303" clear="none">>> network.ping-timeout: 30<br class="ydp658cfa08yiv2833557303" clear="none">>> performance.strict-o-direct: on<br class="ydp658cfa08yiv2833557303" clear="none">>> cluster.granular-entry-heal: enable<br class="ydp658cfa08yiv2833557303" clear="none">>> server.allow-insecure: on<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Seems no positive or negative effect so far.<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> 2. Tested with tmpfs on all bricks -> ovirt1 mounted gluster volume -> max 60MB/s (bs=1M without 'oflag=direct')<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> [<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> data]# dd if=/dev/zero of=large_io bs=1M count=4000 status=progress<br class="ydp658cfa08yiv2833557303" clear="none">>> 4177526784 bytes (4.2 GB) copied, 70.843409 s, 59.0 MB/s<br class="ydp658cfa08yiv2833557303" clear="none">>> 4000+0 records in<br class="ydp658cfa08yiv2833557303" clear="none">>> 4000+0 records out<br class="ydp658cfa08yiv2833557303" clear="none">>> 4194304000 bytes (4.2 GB) copied, 71.1407 s, 59.0 MB/s<br class="ydp658cfa08yiv2833557303" clear="none">>> [<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> data]# rm -f large_io<br class="ydp658cfa08yiv2833557303" clear="none">>> [<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> data]# gluster volume profile data info<br class="ydp658cfa08yiv2833557303" clear="none">>> Brick: ovirt1.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">>> ---------------------------------------------------<br class="ydp658cfa08yiv2833557303" clear="none">>> Cumulative Stats:<br class="ydp658cfa08yiv2833557303" clear="none">>> Block Size: 131072b+ <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Reads: 8 <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Writes: 44968 <br class="ydp658cfa08yiv2833557303" clear="none">>> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">>> --------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 3 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 35 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 78.00 us 78.00 us 78.00 us 1 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 35.67 us 26.00 us 73.00 us 6 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 324.00 us 324.00 us 324.00 us 1 XATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 45.80 us 38.00 us 54.00 us 10 STAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 227.67 us 216.00 us 242.00 us 3 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 113.38 us 68.00 us 381.00 us 8 READ<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 59.97 us 45.00 us 113.00 us 32 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 24.41 us 13.00 us 89.00 us 161 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 43.43 us 28.00 us 214.00 us 93 STATFS<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 246.35 us 11.00 us 1155.00 us 20 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 283.00 us 233.00 us 353.00 us 18 READDIRP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 153.23 us 122.00 us 259.00 us 87 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.77 124.24 us 87.00 us 604.00 us 44968 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">>> 1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">>> 96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Duration: 380 seconds<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Read: 1048576 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Written: 5894045696 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Interval 0 Stats:<br class="ydp658cfa08yiv2833557303" clear="none">>> Block Size: 131072b+ <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Reads: 8 <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Writes: 44968 <br class="ydp658cfa08yiv2833557303" clear="none">>> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">>> --------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 3 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 35 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 28 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 78.00 us 78.00 us 78.00 us 1 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 35.67 us 26.00 us 73.00 us 6 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 324.00 us 324.00 us 324.00 us 1 XATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 45.80 us 38.00 us 54.00 us 10 STAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 227.67 us 216.00 us 242.00 us 3 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 113.38 us 68.00 us 381.00 us 8 READ<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 39.82 us 1.00 us 148.00 us 28 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 67.54 us 10.00 us 283.00 us 24 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 59.97 us 45.00 us 113.00 us 32 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 24.41 us 13.00 us 89.00 us 161 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 43.43 us 28.00 us 214.00 us 93 STATFS<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 246.35 us 11.00 us 1155.00 us 20 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 283.00 us 233.00 us 353.00 us 18 READDIRP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 153.23 us 122.00 us 259.00 us 87 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 99.77 us 10.00 us 258.00 us 442 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.31 49.22 us 27.00 us 540.00 us 45620 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.77 124.24 us 87.00 us 604.00 us 44968 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.93 15767.71 us 15.00 us 305833.00 us 431 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">>> 1.99 160711.39 us 3332.00 us 406037.00 us 90 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">>> 96.00 5167.82 us 18.00 us 55972.00 us 135349 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Duration: 380 seconds<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Read: 1048576 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Written: 5894045696 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Brick: ovirt3.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">>> ---------------------------------------------------<br class="ydp658cfa08yiv2833557303" clear="none">>> Cumulative Stats:<br class="ydp658cfa08yiv2833557303" clear="none">>> Block Size: 1b+ <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Reads: 0 <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Writes: 39328 <br class="ydp658cfa08yiv2833557303" clear="none">>> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">>> --------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 2 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 12 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 101.00 us 101.00 us 101.00 us 1 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 51.50 us 20.00 us 81.00 us 4 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 219.50 us 188.00 us 251.00 us 2 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 62.30 us 38.00 us 119.00 us 10 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 24.60 us 12.00 us 64.00 us 40 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.02 176.30 us 10.00 us 765.00 us 10 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.07 63.08 us 39.00 us 133.00 us 78 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.13 126.89 us 99.00 us 179.00 us 76 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">>> 28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">>> 40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Duration: 189 seconds<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Read: 0 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Written: 39328 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Interval 0 Stats:<br class="ydp658cfa08yiv2833557303" clear="none">>> Block Size: 1b+ <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Reads: 0 <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Writes: 39328 <br class="ydp658cfa08yiv2833557303" clear="none">>> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">>> --------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 2 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 12 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 17 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 101.00 us 101.00 us 101.00 us 1 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 51.50 us 20.00 us 81.00 us 4 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 219.50 us 188.00 us 251.00 us 2 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 43.45 us 11.00 us 90.00 us 11 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 62.30 us 38.00 us 119.00 us 10 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 50.59 us 1.00 us 102.00 us 17 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 24.60 us 12.00 us 64.00 us 40 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.02 176.30 us 10.00 us 765.00 us 10 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.07 63.08 us 39.00 us 133.00 us 78 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.13 27.35 us 10.00 us 91.00 us 333 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.13 126.89 us 99.00 us 179.00 us 76 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.42 116.70 us 8.00 us 8661.00 us 261 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">>> 28.73 51.79 us 22.00 us 2574.00 us 39822 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 29.52 53.87 us 16.00 us 3290.00 us 39328 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">>> 40.92 24.71 us 10.00 us 3224.00 us 118864 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Duration: 189 seconds<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Read: 0 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Written: 39328 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Brick: ovirt2.localdomain:/gluster_bricks/data/data<br class="ydp658cfa08yiv2833557303" clear="none">>> ---------------------------------------------------<br class="ydp658cfa08yiv2833557303" clear="none">>> Cumulative Stats:<br class="ydp658cfa08yiv2833557303" clear="none">>> Block Size: 512b+ 131072b+ <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Reads: 0 0 <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Writes: 36 76758 <br class="ydp658cfa08yiv2833557303" clear="none">>> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">>> --------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 6 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 87 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 36.18 us 22.00 us 62.00 us 11 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 82.56 us 59.00 us 138.00 us 9 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 89.42 us 67.00 us 161.00 us 12 SETATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 272.40 us 235.00 us 296.00 us 5 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 154.28 us 88.00 us 320.00 us 18 XATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 86.69 us 30.00 us 379.00 us 62 STAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 64.30 us 47.00 us 169.00 us 84 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.02 107.34 us 23.00 us 273.00 us 73 READDIRP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.03 128.51 us 27.00 us 338.00 us 96 FSYNC<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.03 240.75 us 14.00 us 1943.00 us 52 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.04 65.59 us 26.00 us 293.00 us 279 STATFS<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.06 180.77 us 118.00 us 306.00 us 148 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.14 37.98 us 17.00 us 192.00 us 1598 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.67 91.68 us 12.00 us 1141.00 us 3186 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">>> 10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">>> 19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">>> 25.21 142.92 us 62.00 us 593.00 us 76794 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">>> 32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Duration: 1206 seconds<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Read: 0 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Written: 10060843008 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Interval 0 Stats:<br class="ydp658cfa08yiv2833557303" clear="none">>> Block Size: 512b+ 131072b+ <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Reads: 0 0 <br class="ydp658cfa08yiv2833557303" clear="none">>> No. of Writes: 36 76758 <br class="ydp658cfa08yiv2833557303" clear="none">>> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop<br class="ydp658cfa08yiv2833557303" clear="none">>> --------- ----------- ----------- ----------- ------------ ----<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 6 FORGET<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 87 RELEASE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 0.00 us 0.00 us 0.00 us 96 RELEASEDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 100.50 us 80.00 us 121.00 us 2 REMOVEXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 101.00 us 101.00 us 101.00 us 2 SETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 36.18 us 22.00 us 62.00 us 11 FLUSH<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 57.44 us 42.00 us 77.00 us 9 FTRUNCATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 82.56 us 59.00 us 138.00 us 9 FSTAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 89.42 us 67.00 us 161.00 us 12 SETATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.00 272.40 us 235.00 us 296.00 us 5 CREATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 154.28 us 88.00 us 320.00 us 18 XATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 45.29 us 1.00 us 319.00 us 96 OPENDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 86.69 us 30.00 us 379.00 us 62 STAT<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.01 64.30 us 47.00 us 169.00 us 84 OPEN<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.02 107.34 us 23.00 us 273.00 us 73 READDIRP<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.02 4688.00 us 86.00 us 9290.00 us 2 TRUNCATE<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.02 59.29 us 13.00 us 394.00 us 165 GETXATTR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.03 128.51 us 27.00 us 338.00 us 96 FSYNC<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.03 240.75 us 14.00 us 1943.00 us 52 READDIR<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.04 65.59 us 26.00 us 293.00 us 279 STATFS<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.06 180.77 us 118.00 us 306.00 us 148 MKNOD<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.14 37.98 us 17.00 us 192.00 us 1598 INODELK<br class="ydp658cfa08yiv2833557303" clear="none">>> 0.67 91.66 us 12.00 us 1141.00 us 3186 LOOKUP<br class="ydp658cfa08yiv2833557303" clear="none">>> 10.10 55.92 us 28.00 us 1658.00 us 78608 FXATTROP<br class="ydp658cfa08yiv2833557303" clear="none">>> 11.89 6814.76 us 18.00 us 301246.00 us 760 ENTRYLK<br class="ydp658cfa08yiv2833557303" clear="none">>> 19.44 36.55 us 14.00 us 2353.00 us 231535 FINODELK<br class="ydp658cfa08yiv2833557303" clear="none">>> 25.21 142.92 us 62.00 us 593.00 us 76794 WRITE<br class="ydp658cfa08yiv2833557303" clear="none">>> 32.28 91283.68 us 28.00 us 316658.00 us 154 UNLINK<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Duration: 1206 seconds<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Read: 0 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>> Data Written: 10060843008 bytes<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> This indicates to me that it's not a problem in Disk/LVM/FileSystem layout.<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Most probably I haven't created the volume properly or some option/feature is disabled ?!?<br class="ydp658cfa08yiv2833557303" clear="none">>> Network shows OK for a gigabit:<br class="ydp658cfa08yiv2833557303" clear="none">>> [<a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:root@ovirt1" rel="nofollow" target="_blank">root@ovirt1</a> data]# dd if=/dev/zero status=progress | nc ovirt2 9999<br class="ydp658cfa08yiv2833557303" clear="none">>> 3569227264 bytes (3.6 GB) copied, 29.001052 s, 123 MB/s^C<br class="ydp658cfa08yiv2833557303" clear="none">>> 7180980+0 records in<br class="ydp658cfa08yiv2833557303" clear="none">>> 7180979+0 records out<br class="ydp658cfa08yiv2833557303" clear="none">>> 3676661248 bytes (3.7 GB) copied, 29.8739 s, 123 MB/s<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> I'm looking for any help... you can share your volume info also.<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Thanks in advance.<br class="ydp658cfa08yiv2833557303" clear="none">>><br class="ydp658cfa08yiv2833557303" clear="none">>> Best Regards,<br class="ydp658cfa08yiv2833557303" clear="none">>> Strahil Nikolov<br class="ydp658cfa08yiv2833557303" clear="none">>> _______________________________________________<br class="ydp658cfa08yiv2833557303" clear="none">>> Gluster-users mailing list<br class="ydp658cfa08yiv2833557303" clear="none">>> <a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:Gluster-users@gluster.org" rel="nofollow" target="_blank">Gluster-users@gluster.org</a><br class="ydp658cfa08yiv2833557303" clear="none">>> <a shape="rect" class="ydp658cfa08yiv2833557303" href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="nofollow" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br class="ydp658cfa08yiv2833557303" clear="none">><br class="ydp658cfa08yiv2833557303" clear="none">><br class="ydp658cfa08yiv2833557303" clear="none">><br class="ydp658cfa08yiv2833557303" clear="none">> -- <br class="ydp658cfa08yiv2833557303" clear="none">> Amar Tumballi (amarts)<br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none"></div><br class="ydp658cfa08yiv2833557303" clear="none"><br class="ydp658cfa08yiv2833557303" clear="none"></div> </div> </div> </div></div></div>_______________________________________________<br class="ydp658cfa08yiv2833557303" clear="none">Gluster-users mailing list<br class="ydp658cfa08yiv2833557303" clear="none"><a shape="rect" class="ydp658cfa08yiv2833557303" href="mailto:Gluster-users@gluster.org" rel="nofollow" target="_blank">Gluster-users@gluster.org</a><br class="ydp658cfa08yiv2833557303" clear="none">https://lists.gluster.org/mailman/listinfo/gluster-users</div></div></blockquote></div><br class="ydp658cfa08yiv2833557303" clear="none"></div></div></div></div>
</div>
</div></body></html>