<div dir="auto"><div>Following changes resolved the perf issue:<div dir="auto"><br><div dir="auto"><span style="font-family:sans-serif;font-size:13.696px">Added the option</span><br style="font-family:sans-serif;font-size:13.696px"><span style="font-family:sans-serif;font-size:13.696px">/etc/glusterfs/glusterd.vol : </span><br style="font-family:sans-serif;font-size:13.696px"><span style="font-family:sans-serif;font-size:13.696px">option rpc-auth-allow-insecure on</span><br style="font-family:sans-serif;font-size:13.696px"><br style="font-family:sans-serif;font-size:13.696px"><span style="font-family:sans-serif;font-size:13.696px">restarted glusterd</span><br style="font-family:sans-serif;font-size:13.696px"><br style="font-family:sans-serif;font-size:13.696px"><span style="font-family:sans-serif;font-size:13.696px">Then set the volume option: </span><br style="font-family:sans-serif;font-size:13.696px"><span style="font-family:sans-serif;font-size:13.696px">gluster volume set vms server.allow-insecure on</span><br></div><div dir="auto"><br></div><div dir="auto">I am reaching now the max network bandwidth and performance of VMs is quite good. </div><div dir="auto"><br></div><div dir="auto">Did not upgrade the glusterd.</div><div dir="auto"><br></div><div dir="auto">As a next try I am thinking to upgrade gluster to 3.12 + test libgfapi integration of qemu by upgrading to ovirt 4.1.5 and check vm perf.</div></div><br><div class="gmail_extra"><br><div class="gmail_quote">On Sep 6, 2017 1:20 PM, "Abi Askushi" <<a href="mailto:rightkicktech@gmail.com">rightkicktech@gmail.com</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>I tried to follow step from <a href="https://wiki.centos.org/SpecialInterestGroup/Storage" target="_blank">https://wiki.centos.org/<wbr>SpecialInterestGroup/Storage</a> to install latest gluster on the first node.<br></div>It installed 3.10 and not 3.11. I am not sure how to install 3.11 without compiling it. </div><div>Then when tried to start the gluster on the node the bricks were reported down (the other 2 nodes have still 3.8). No sure why. The logs were showing the below (even after rebooting the server): <br></div><div><br></div><div><span style="font-family:monospace,monospace">[2017-09-06 10:56:09.023777] E [rpcsvc.c:557:rpcsvc_check_<wbr>and_reply_error] 0-rpcsvc: rpc actor failed to complete successfully<br>[2017-09-06 10:56:09.024122] E [server-helpers.c:395:server_<wbr>alloc_frame] (-->/lib64/libgfrpc.so.0(<wbr>rpcsvc_handle_rpc_call+0x325) [0x7f2d0ec20905] -->/usr/lib64/glusterfs/3.10.<wbr>5/xlator/protocol/server.so(+<wbr>0x3006b) [0x7f2cfa4bf06b] -->/usr/lib64/glusterfs/3.10.<wbr>5/xlator/protocol/server.so(+<wbr>0xdb34) [0x7f2cfa49cb34] ) 0-server: invalid argument: client [Invalid argument]<br></span></div><div><br></div><div>Do I need to upgrade all nodes before I attempt to start the gluster services?</div><div>I reverted the first node back to 3.8 at the moment and all restored. <br></div><div>Also tests with eager lock disabled did not make any difference. <br></div><div><br></div><div><br></div><div><br></div></div><div class="elided-text"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 6, 2017 at 11:15 AM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Do you see any improvement with 3.11.1 as that has a patch that improves perf for this kind of a workload<br><br></div><div>Also, could you disable eager-lock and check if that helps? I see that max time is being spent in acquiring locks.<span class="m_8350782179552582065HOEnZb"><font color="#888888"><br></font></span></div><span class="m_8350782179552582065HOEnZb"><font color="#888888"><div><br></div><div>-Krutika<br></div></font></span></div><div class="m_8350782179552582065HOEnZb"><div class="m_8350782179552582065h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 6, 2017 at 1:38 PM, Abi Askushi <span dir="ltr"><<a href="mailto:rightkicktech@gmail.com" target="_blank">rightkicktech@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">Hi Krutika,<div dir="auto"><br></div><div dir="auto">Is it anything in the profile indicating what is causing this bottleneck? In case i can collect any other info let me know.</div><div dir="auto"><br></div><div dir="auto">Thanx</div></div><div class="m_8350782179552582065m_6836400769176734828HOEnZb"><div class="m_8350782179552582065m_6836400769176734828h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sep 5, 2017 13:27, "Abi Askushi" <<a href="mailto:rightkicktech@gmail.com" target="_blank">rightkicktech@gmail.com</a>> wrote:<br type="attribution"><blockquote class="m_8350782179552582065m_6836400769176734828m_-9174049062882710918quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hi Krutika, <br><br></div>Attached the profile stats. I enabled profiling then ran some dd tests. Also 3 Windows VMs are running on top this volume but did not do any stress testing on the VMs. I have left the profiling enabled in case more time is needed for useful stats. <br><br></div>Thanx<br></div><div class="m_8350782179552582065m_6836400769176734828m_-9174049062882710918elided-text"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 5, 2017 at 12:48 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>OK my understanding is that with preallocated disks the performance with and without shard will be the same.<br><br></div>In any case, please attach the volume profile[1], so we can see what else is slowing things down.<br><br></div>-Krutika<br><br>[1] - <a href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/#running-glusterfs-volume-profile-command" target="_blank">https://gluster.readthedocs.io<wbr>/en/latest/Administrator%20Gui<wbr>de/Monitoring%20Workload/#runn<wbr>ing-glusterfs-volume-profile-c<wbr>ommand</a><br></div><div class="m_8350782179552582065m_6836400769176734828m_-9174049062882710918m_9090603865708594746HOEnZb"><div class="m_8350782179552582065m_6836400769176734828m_-9174049062882710918m_9090603865708594746h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 5, 2017 at 2:32 PM, Abi Askushi <span dir="ltr"><<a href="mailto:rightkicktech@gmail.com" target="_blank">rightkicktech@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div>Hi Krutika, <br><br></div>I already have a preallocated disk on VM. <br></div>Now I am checking performance with dd on the hypervisors which have the gluster volume configured. <br><br></div>I tried also several values of shard-block-size and I keep getting the same low values on write performance. <br></div>Enabling client-io-threads also did not have any affect. <br><br></div>The version of gluster I am using is glusterfs 3.8.12 built on May 11 2017 18:46:20. <br></div>The setup is a set of 3 Centos 7.3 servers and ovirt 4.1, using gluster as storage. <br></div><div><br></div><div>Below are the current settings: <br></div><div><br></div><div><span><br>Volume Name: vms<br>Type: Replicate<br>Volume ID: 4513340d-7919-498b-bfe0-d836b5<wbr>cea40b<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 1 x (2 + 1) = 3<br>Transport-type: tcp<br>Bricks:<br>Brick1: gluster0:/gluster/vms/brick<br>Brick2: gluster1:/gluster/vms/brick<br>Brick3: gluster2:/gluster/vms/brick (arbiter)<br>Options Reconfigured:<br></span>server.event-threads: 4<br>client.event-threads: 4<br>performance.client-io-threads: on<br>features.shard-block-size: 512MB<span><br>cluster.granular-entry-heal: enable<br>performance.strict-o-direct: on<br>network.ping-timeout: 30<br>storage.owner-gid: 36<br>storage.owner-uid: 36<br>user.cifs: off<br>features.shard: on<br>cluster.shd-wait-qlength: 10000<br>cluster.shd-max-threads: 8<br>cluster.locking-scheme: granular<br>cluster.data-self-heal-algorit<wbr>hm: full<br>cluster.server-quorum-type: server<br>cluster.quorum-type: auto<br>cluster.eager-lock: enable<br>network.remote-dio: off<br>performance.low-prio-threads: 32<br></span>performance.stat-prefetch: on<span><br>performance.io-cache: off<br>performance.read-ahead: off<br>performance.quick-read: off<br>transport.address-family: inet<br>performance.readdir-ahead: on<br>nfs.disable: on<br>nfs.export-volumes: on<br><br></span></div><div><br></div>I observed that when testing with dd if=/dev/zero of=testfile bs=1G count=1 I get 65MB/s on the vms gluster volume (and the network traffic between the servers reaches ~ 500Mbps), while when testing with dd if=/dev/zero of=testfile bs=1G count=1 <b>oflag=direct </b>I get a consistent 10MB/s<b> </b>and the network traffic hardly reaching 100Mbps. <br><br>Any other things one can do?<br></div><div class="m_8350782179552582065m_6836400769176734828m_-9174049062882710918m_9090603865708594746m_874780393017638440HOEnZb"><div class="m_8350782179552582065m_6836400769176734828m_-9174049062882710918m_9090603865708594746m_874780393017638440h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 5, 2017 at 5:57 AM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>I'm assuming you are using this volume to store vm images, because I see shard in the options list.<br><br></div>Speaking from shard translator's POV, one thing you can do to improve performance is to use preallocated images.<br></div>This will at least eliminate the need for shard to perform multiple steps as part of the writes - such as creating the shard and then writing to it and then updating the aggregated file size - all of which require one network call each, which further get blown up once they reach AFR (replicate) into many more network calls.<br><br></div>Second, I'm assuming you're using the default shard block size of 4MB (you can confirm this using `gluster volume get <VOL> shard-block-size`). In our tests, we've found that larger shard sizes perform better. So maybe change the shard-block-size to 64MB (`gluster volume set <VOL> shard-block-size 64MB`).<br><br></div><div>Third, keep stat-prefetch enabled. We've found that qemu sends quite a lot of [f]stats which can be served from the (md)cache to improve performance. So enable that.<br><br></div><div>Also, could you also enable client-io-threads and see if that improves performance?<br><br></div>Which version of gluster are you using BTW?<br><br></div>-Krutika<br><br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_8350782179552582065m_6836400769176734828m_-9174049062882710918m_9090603865708594746m_874780393017638440m_7174052037606786489h5">On Tue, Sep 5, 2017 at 4:32 AM, Abi Askushi <span dir="ltr"><<a href="mailto:rightkicktech@gmail.com" target="_blank">rightkicktech@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_8350782179552582065m_6836400769176734828m_-9174049062882710918m_9090603865708594746m_874780393017638440m_7174052037606786489h5"><div dir="ltr"><div><div><div>Hi all, <br><br></div>I have a gluster volume used to host several VMs (managed through oVirt). <br></div><div>The volume is a replica 3 with arbiter and the 3 servers use 1 Gbit network for the storage.</div><div><br></div><div>When testing with dd (dd if=/dev/zero of=testfile bs=1G count=1 oflag=direct) out of the volume (e.g. writing at /root/) the performance of the dd is reported to be ~ 700MB/s, which is quite decent. When testing the dd on the gluster volume I get ~ 43 MB/s which way lower from the previous. When testing with dd the gluster volume, the network traffic was not exceeding 450 Mbps on the network interface. I would expect to reach near 900 Mbps considering that there is 1 Gbit of bandwidth available. This results having VMs with very slow performance (especially on their write operations). <br></div><div><br></div><div>The full details of the volume are below. Any advise on what can be tweaked will be highly appreciated. <br></div></div><div><br></div><div>Volume Name: vms<br>Type: Replicate<br>Volume ID: 4513340d-7919-498b-bfe0-d836b5<wbr>cea40b<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 1 x (2 + 1) = 3<br>Transport-type: tcp<br>Bricks:<br>Brick1: gluster0:/gluster/vms/brick<br>Brick2: gluster1:/gluster/vms/brick<br>Brick3: gluster2:/gluster/vms/brick (arbiter)<br>Options Reconfigured:<br>cluster.granular-entry-heal: enable<br>performance.strict-o-direct: on<br>network.ping-timeout: 30<br>storage.owner-gid: 36<br>storage.owner-uid: 36<br>user.cifs: off<br>features.shard: on<br>cluster.shd-wait-qlength: 10000<br>cluster.shd-max-threads: 8<br>cluster.locking-scheme: granular<br>cluster.data-self-heal-algorit<wbr>hm: full<br>cluster.server-quorum-type: server<br>cluster.quorum-type: auto<br>cluster.eager-lock: enable<br>network.remote-dio: off<br>performance.low-prio-threads: 32<br>performance.stat-prefetch: off<br>performance.io-cache: off<br>performance.read-ahead: off<br>performance.quick-read: off<br>transport.address-family: inet<br>performance.readdir-ahead: on<br>nfs.disable: on<br>nfs.export-volumes: on<br></div><div><br></div><div><br></div><div>Thanx, <br></div><div>Alex<br></div></div>
<br></div></div>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></blockquote></div><br></div></div></div>