[Gluster-users] Slow performance over rsync in Replicated-Distributed Setup

[Felix Kölzow]

Dear Shubhank,

this small file performance appears to be slow on glusterfs usually.

Can you provide more details according to your setup? (zfs settings,
bonding, tuned-adm profile, etc, ...)


 From a gluster point of view, setting performance.write-behind-window
to 128MB increases performance.

I was able to hit the cpu limit using smallfile benchmark tool
(available on github) and native glusterfs-client with that knob.


Furthermore, throughput increases if you increase the number of rsync
processes ( -> github -> msrsync works well here).


Regards,

Felix


On 06/03/2021 15:27, Shubhank Gaur wrote:
> Hello users,
>
> I have started using gluster just a few weeks ago and I am rocking a
> Replicated-Distributed setup with arbiters (A) and SATA Volumes (V). I
> have around 6 volumes and 3 arbiters in this setup:
> V+V+A | V+V+A | V+V+A
>
> All these volumes are spread across 3 different nodes, all of them
> being 1Gbit. Due to hardware limitations, SSD or 10Gbit network is not
> available.
>
> But even then, testing via iperf and normal rsync of files between
> servers, I am easily able to achieve 700Mbps~
> [ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
> [  4]   0.00-1.00   sec  49.9 MBytes   419 Mbits/sec   21  132 KBytes
> [  4]   1.00-2.00   sec  80.0 MBytes   671 Mbits/sec    0  214 KBytes
> [  4]   2.00-3.00   sec  87.0 MBytes   730 Mbits/sec    3  228 KBytes
> [  4]   3.00-4.00   sec  91.6 MBytes   769 Mbits/sec   15  215 KBytes
>
> But when rsyncing data from same server to another node with mounted
> glusterVolume, I am getting measly 50Mbps (7MBps).
>
> All servers have 64GB Ram and their memory usage is around 50% and CPU
> usage less than 10%.
> All bricks are zfs volumes, no Raid setup or anything.  All volumes
> are direct hard disks formatted as ZFS (JBOD setup).
>
>
> My Gluster Vol Info
>
> gluster vol info
>
> Volume Name: glusterStore
> Type: Distributed-Replicate
> Volume ID: c7ac8094-f379-45fc-8cfd-f2937355e03d
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 3 x (2 + 1) = 9
> Transport-type: tcp
> Bricks:
> Brick1: 62.0.0.1:/zpool1/proxmox
> Brick2: 5.0.0.1:/zpool1/proxmox
> Brick3: 62.0.0.1 :/home/glusterArbiter (arbiter)
> Brick4: 62.0.0.1 :/zpool2/proxmox
> Brick5: 5.0.0.1 :/zpool2/proxmox
> Brick6: 62.0.0.2:/home/glusterArbiter2 (arbiter)
> Brick7: 62.0.0.2:/zpool/proxmox
> Brick8: 5.0.0.1 :/zpool3/proxmox
> Brick9: 62.0.0.2:/home/glusterArbiter (arbiter)
> Options Reconfigured:
> performance.readdir-ahead: enable
> cluster.rsync-hash-regex: none
> client.event-threads: 16
> server.event-threads: 16
> network.ping-timeout: 5
> performance.normal-prio-threads: 64
> performance.high-prio-threads: 64
> performance.io-thread-count: 64
> performance.cache-size: 1GB
> performance.read-ahead: off
> performance.io-cache: off
> performance.flush-behind: off
> performance.quick-read: on
> network.frame-timeout: 60
> storage.batch-fsync-delay-usec: 0
> server.allow-insecure: on
> performance.stat-prefetch: off
> cluster.lookup-optimize: on
> performance.write-behind: on
> cluster.granular-entry-heal: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
>
>
> Regards
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210306/895ecb67/attachment.html>