[Gluster-users] Gluster performance on NVMe SSDs

[Kumar Sandeep]

Hi,

I have setup oVirt hyperconverged with 3xDL325 with 3 NVMe SSDs on each server for each Gluster volumes: engine, vmstore and data in replica 3.

I am getting almost 4x performance difference when I am benchmarking with DD on gluster disk:

Below output is from one of the gluster host:

 time dd if=/dev/zero of=/gluster/vmstore/testfile bs=1M count=1000 oflag=direct

1000+0 records in

1000+0 records out

1048576000 bytes (1.0 GB) copied, 0.546909 s, 1.9 GB/s



real        0m0.552s

user       0m0.001s

sys          0m0.165s

Below output is from VM.

# time dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct

1000+0 records in

1000+0 records out

1048576000 bytes (1.0 GB) copied, 1.89253 s, 554 MB/s



real        0m1.896s

user       0m0.002s

sys          0m0.247s



Earlier I was getting around ~270MB in that VM, It improved after setting the engine-config -s LibgfApiSupported=true --cver=4.3



I am using default tuned profile(virtual-host). Backend network is 10GbE, tested with iPerf and it gives above 9.40 Gbits/se bandwidth



# gluster v info vmstore

Volume Name: vmstore

Type: Replicate

Volume ID: b1f0c1fb-8b8b-479a-9117-61d45692047a

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 3 = 3

Transport-type: tcp

Bricks:

Brick1: daal-gluster.mbi.nus.edu.sg:/gluster/vmstore/brick1

Brick2: naan-gluster.mbi.nus.edu.sg:/gluster/vmstore/brick1

Brick3: dosa-gluster.mbi.nus.edu.sg:/gluster/vmstore/brick1

Options Reconfigured:

server.event-threads: 4

client.event-threads: 4

performance.client-io-threads: off

nfs.disable: on

transport.address-family: inet

performance.quick-read: off

performance.read-ahead: off

performance.io-cache: on

performance.low-prio-threads: 32

network.remote-dio: enable

cluster.eager-lock: enable

cluster.quorum-type: auto

cluster.server-quorum-type: server

cluster.data-self-heal-algorithm: full

cluster.locking-scheme: granular

cluster.shd-max-threads: 8

cluster.shd-wait-qlength: 10000

features.shard: on

user.cifs: off

storage.owner-uid: 36

storage.owner-gid: 36

features.shard-block-size: 512MB

performance.cache-size: 1GB

performance.io-thread-count: 32

performance.write-behind-window-size: 4MB





Is there any way I can improve my gluster disk’s performance?



Thanks for helping out.



Regards,

Sandeep

________________________________

Important: This email is confidential and may be privileged. If you are not the intended recipient, please delete it and notify us immediately; you should not copy or use it for any purpose, nor disclose its contents to any other person. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190613/24a05a6f/attachment.html>