[Gluster-users] Slow performance over rsync in Replicated-Distributed Setup

Shubhank Gaur shubhank008pp at gmail.com
Sat Mar 6 14:27:43 UTC 2021

Hello users,

I have started using gluster just a few weeks ago and I am rocking a
Replicated-Distributed setup with arbiters (A) and SATA Volumes (V). I have
around 6 volumes and 3 arbiters in this setup:
V+V+A | V+V+A | V+V+A

All these volumes are spread across 3 different nodes, all of them being
1Gbit. Due to hardware limitations, SSD or 10Gbit network is not

But even then, testing via iperf and normal rsync of files between servers,
I am easily able to achieve 700Mbps~
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  49.9 MBytes   419 Mbits/sec   21    132 KBytes
[  4]   1.00-2.00   sec  80.0 MBytes   671 Mbits/sec    0    214 KBytes
[  4]   2.00-3.00   sec  87.0 MBytes   730 Mbits/sec    3    228 KBytes
[  4]   3.00-4.00   sec  91.6 MBytes   769 Mbits/sec   15    215 KBytes

But when rsyncing data from same server to another node with mounted
glusterVolume, I am getting measly 50Mbps (7MBps).

All servers have 64GB Ram and their memory usage is around 50% and CPU
usage less than 10%.
All bricks are zfs volumes, no Raid setup or anything.  All volumes are
direct hard disks formatted as ZFS (JBOD setup).

My Gluster Vol Info

gluster vol info

Volume Name: glusterStore
Type: Distributed-Replicate
Volume ID: c7ac8094-f379-45fc-8cfd-f2937355e03d
Status: Started
Snapshot Count: 0
Number of Bricks: 3 x (2 + 1) = 9
Transport-type: tcp
Brick3: :/home/glusterArbiter (arbiter)
Brick4: :/zpool2/proxmox
Brick5: :/zpool2/proxmox
Brick6: (arbiter)
Brick8: :/zpool3/proxmox
Brick9: (arbiter)
Options Reconfigured:
performance.readdir-ahead: enable
cluster.rsync-hash-regex: none
client.event-threads: 16
server.event-threads: 16
network.ping-timeout: 5
performance.normal-prio-threads: 64
performance.high-prio-threads: 64
performance.io-thread-count: 64
performance.cache-size: 1GB
performance.read-ahead: off
performance.io-cache: off
performance.flush-behind: off
performance.quick-read: on
network.frame-timeout: 60
storage.batch-fsync-delay-usec: 0
server.allow-insecure: on
performance.stat-prefetch: off
cluster.lookup-optimize: on
performance.write-behind: on
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210306/98865a7e/attachment.html>

More information about the Gluster-users mailing list