[Gluster-users] Slow performance of gluster volume

Wed Sep 6 08:08:18 UTC 2017

Hi Krutika,

Is it anything in the profile indicating what is causing this bottleneck?
In case i can collect any other info let me know.

Thanx

On Sep 5, 2017 13:27, "Abi Askushi" <rightkicktech at gmail.com> wrote:

Hi Krutika,

Attached the profile stats. I enabled profiling then ran some dd tests.
Also 3 Windows VMs are running on top this volume but did not do any stress
testing on the VMs. I have left the profiling enabled in case more time is
needed for useful stats.

Thanx

On Tue, Sep 5, 2017 at 12:48 PM, Krutika Dhananjay <kdhananj at redhat.com>
wrote:

> OK my understanding is that with preallocated disks the performance with
> and without shard will be the same.
>
> In any case, please attach the volume profile[1], so we can see what else
> is slowing things down.
>
> -Krutika
>
> [1] - https://gluster.readthedocs.io/en/latest/Administrator%20Gui
> de/Monitoring%20Workload/#running-glusterfs-volume-profile-command
>
> On Tue, Sep 5, 2017 at 2:32 PM, Abi Askushi <rightkicktech at gmail.com>
> wrote:
>
>> Hi Krutika,
>>
>> I already have a preallocated disk on VM.
>> Now I am checking performance with dd on the hypervisors which have the
>> gluster volume configured.
>>
>> I tried also several values of shard-block-size and I keep getting the
>> same low values on write performance.
>> Enabling client-io-threads also did not have any affect.
>>
>> The version of gluster I am using is glusterfs 3.8.12 built on May 11
>> 2017 18:46:20.
>> The setup is a set of 3 Centos 7.3 servers and ovirt 4.1, using gluster
>> as storage.
>>
>> Below are the current settings:
>>
>>
>> Volume Name: vms
>> Type: Replicate
>> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster0:/gluster/vms/brick
>> Brick2: gluster1:/gluster/vms/brick
>> Brick3: gluster2:/gluster/vms/brick (arbiter)
>> Options Reconfigured:
>> server.event-threads: 4
>> client.event-threads: 4
>> performance.client-io-threads: on
>> features.shard-block-size: 512MB
>> cluster.granular-entry-heal: enable
>> performance.strict-o-direct: on
>> network.ping-timeout: 30
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> user.cifs: off
>> features.shard: on
>> cluster.shd-wait-qlength: 10000
>> cluster.shd-max-threads: 8
>> cluster.locking-scheme: granular
>> cluster.data-self-heal-algorithm: full
>> cluster.server-quorum-type: server
>> cluster.quorum-type: auto
>> cluster.eager-lock: enable
>> network.remote-dio: off
>> performance.low-prio-threads: 32
>> performance.stat-prefetch: on
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>> transport.address-family: inet
>> performance.readdir-ahead: on
>> nfs.disable: on
>> nfs.export-volumes: on
>>
>>
>> I observed that when testing with dd if=/dev/zero of=testfile bs=1G
>> count=1 I get 65MB/s on the vms gluster volume (and the network traffic
>> between the servers reaches ~ 500Mbps), while when testing with dd
>> if=/dev/zero of=testfile bs=1G count=1 *oflag=direct *I get a consistent
>> 10MB/s and the network traffic hardly reaching 100Mbps.
>>
>> Any other things one can do?
>>
>> On Tue, Sep 5, 2017 at 5:57 AM, Krutika Dhananjay <kdhananj at redhat.com>
>> wrote:
>>
>>> I'm assuming you are using this volume to store vm images, because I see
>>> shard in the options list.
>>>
>>> Speaking from shard translator's POV, one thing you can do to improve
>>> performance is to use preallocated images.
>>> This will at least eliminate the need for shard to perform multiple
>>> steps as part of the writes - such as creating the shard and then writing
>>> to it and then updating the aggregated file size - all of which require one
>>> network call each, which further get blown up once they reach AFR
>>> (replicate) into many more network calls.
>>>
>>> Second, I'm assuming you're using the default shard block size of 4MB
>>> (you can confirm this using `gluster volume get <VOL> shard-block-size`).
>>> In our tests, we've found that larger shard sizes perform better. So maybe
>>> change the shard-block-size to 64MB (`gluster volume set <VOL>
>>> shard-block-size 64MB`).
>>>
>>> Third, keep stat-prefetch enabled. We've found that qemu sends quite a
>>> lot of [f]stats which can be served from the (md)cache to improve
>>> performance. So enable that.
>>>
>>> Also, could you also enable client-io-threads and see if that improves
>>> performance?
>>>
>>> Which version of gluster are you using BTW?
>>>
>>> -Krutika
>>>
>>>
>>> On Tue, Sep 5, 2017 at 4:32 AM, Abi Askushi <rightkicktech at gmail.com>
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have a gluster volume used to host several VMs (managed through
>>>> oVirt).
>>>> The volume is a replica 3 with arbiter and the 3 servers use 1 Gbit
>>>> network for the storage.
>>>>
>>>> When testing with dd (dd if=/dev/zero of=testfile bs=1G count=1
>>>> oflag=direct) out of the volume (e.g. writing at /root/) the performance of
>>>> the dd is reported to be ~ 700MB/s, which is quite decent. When testing the
>>>> dd on the gluster volume I get ~ 43 MB/s which way lower from the previous.
>>>> When testing with dd the gluster volume, the network traffic was not
>>>> exceeding 450 Mbps on the network interface. I would expect to reach near
>>>> 900 Mbps considering that there is 1 Gbit of bandwidth available. This
>>>> results having VMs with very slow performance (especially on their write
>>>> operations).
>>>>
>>>> The full details of the volume are below. Any advise on what can be
>>>> tweaked will be highly appreciated.
>>>>
>>>> Volume Name: vms
>>>> Type: Replicate
>>>> Volume ID: 4513340d-7919-498b-bfe0-d836b5cea40b
>>>> Status: Started
>>>> Snapshot Count: 0
>>>> Number of Bricks: 1 x (2 + 1) = 3
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: gluster0:/gluster/vms/brick
>>>> Brick2: gluster1:/gluster/vms/brick
>>>> Brick3: gluster2:/gluster/vms/brick (arbiter)
>>>> Options Reconfigured:
>>>> cluster.granular-entry-heal: enable
>>>> performance.strict-o-direct: on
>>>> network.ping-timeout: 30
>>>> storage.owner-gid: 36
>>>> storage.owner-uid: 36
>>>> user.cifs: off
>>>> features.shard: on
>>>> cluster.shd-wait-qlength: 10000
>>>> cluster.shd-max-threads: 8
>>>> cluster.locking-scheme: granular
>>>> cluster.data-self-heal-algorithm: full
>>>> cluster.server-quorum-type: server
>>>> cluster.quorum-type: auto
>>>> cluster.eager-lock: enable
>>>> network.remote-dio: off
>>>> performance.low-prio-threads: 32
>>>> performance.stat-prefetch: off
>>>> performance.io-cache: off
>>>> performance.read-ahead: off
>>>> performance.quick-read: off
>>>> transport.address-family: inet
>>>> performance.readdir-ahead: on
>>>> nfs.disable: on
>>>> nfs.export-volumes: on
>>>>
>>>>
>>>> Thanx,
>>>> Alex
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170906/a915ddbe/attachment.html>