[Gluster-users] Gluster Performance - 12 Gbps SSDs and 10 Gbps NIC

Tue Dec 12 18:10:31 UTC 2023

Sorry, I noticed that too after I posted, so I instantly upgraded to 10.
Issue remains.

On Tue, Dec 12, 2023 at 1:09 PM Gilberto Ferreira <
gilberto.nunes32 at gmail.com> wrote:

> I strongly suggest you update to version 10 or higher.
> It's come with significant improvement regarding performance.
> ---
> Gilberto Nunes Ferreira
> (47) 99676-7530 - Whatsapp / Telegram
>
>
>
>
>
>
> Em ter., 12 de dez. de 2023 às 13:03, Danny <dbray925+gluster at gmail.com>
> escreveu:
>
>> MTU is already 9000, and as you can see from the IPERF results, I've got
>> a nice, fast connection between the nodes.
>>
>> On Tue, Dec 12, 2023 at 9:49 AM Strahil Nikolov <hunter86_bg at yahoo.com>
>> wrote:
>>
>>> Hi,
>>>
>>> Let’s try the simple things:
>>>
>>> Check if you can use MTU9000 and if it’s possible, set it on the Bond
>>> Slaves and the bond devices:
>>>  ping GLUSTER_PEER -c 10 -M do -s 8972
>>>
>>> Then try to follow up the recommendations from
>>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance
>>>
>>>
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>> On Monday, December 11, 2023, 3:32 PM, Danny <dbray925+gluster at gmail.com>
>>> wrote:
>>>
>>> Hello list, I'm hoping someone can let me know what setting I missed.
>>>
>>> Hardware:
>>> Dell R650 servers, Dual 24 Core Xeon 2.8 GHz, 1 TB RAM
>>> 8x SSD s Negotiated Speed 12 Gbps
>>> PERC H755 Controller - RAID 6
>>> Created virtual "data" disk from the above 8 SSD drives, for a ~20 TB
>>> /dev/sdb
>>>
>>> OS:
>>> CentOS Stream
>>> kernel-4.18.0-526.el8.x86_64
>>> glusterfs-7.9-1.el8.x86_64
>>>
>>> IPERF Test between nodes:
>>> [ ID] Interval           Transfer     Bitrate         Retr
>>> [  5]   0.00-10.00  sec  11.5 GBytes  9.90 Gbits/sec    0
>>> sender
>>> [  5]   0.00-10.04  sec  11.5 GBytes  9.86 Gbits/sec
>>>  receiver
>>>
>>> All good there. ~10 Gbps, as expected.
>>>
>>> LVM Install:
>>> export DISK="/dev/sdb"
>>> sudo parted --script $DISK "mklabel gpt"
>>> sudo parted --script $DISK "mkpart primary 0% 100%"
>>> sudo parted --script $DISK "set 1 lvm on"
>>> sudo pvcreate --dataalignment 128K /dev/sdb1
>>> sudo vgcreate --physicalextentsize 128K gfs_vg /dev/sdb1
>>> sudo lvcreate -L 16G -n gfs_pool_meta gfs_vg
>>> sudo lvcreate -l 95%FREE -n gfs_pool gfs_vg
>>> sudo lvconvert --chunksize 1280K --thinpool gfs_vg/gfs_pool
>>> --poolmetadata gfs_vg/gfs_pool_meta
>>> sudo lvchange --zero n gfs_vg/gfs_pool
>>> sudo lvcreate -V 19.5TiB --thinpool gfs_vg/gfs_pool -n gfs_lv
>>> sudo mkfs.xfs -f -i size=512 -n size=8192 -d su=128k,sw=10
>>> /dev/mapper/gfs_vg-gfs_lv
>>> sudo vim /etc/fstab
>>> /dev/mapper/gfs_vg-gfs_lv   /gluster/data/brick   xfs
>>> rw,inode64,noatime,nouuid 0 0
>>>
>>> sudo systemctl daemon-reload && sudo mount -a
>>> fio --name=test --filename=/gluster/data/brick/wow --size=1G
>>> --readwrite=write
>>>
>>> Run status group 0 (all jobs):
>>>   WRITE: bw=2081MiB/s (2182MB/s), 2081MiB/s-2081MiB/s
>>> (2182MB/s-2182MB/s), io=1024MiB (1074MB), run=492-492msec
>>>
>>> All good there. 2182MB/s =~ 17.5 Gbps. Nice!
>>>
>>>
>>> Gluster install:
>>> export NODE1='10.54.95.123'
>>> export NODE2='10.54.95.124'
>>> export NODE3='10.54.95.125'
>>> sudo gluster peer probe $NODE2
>>> sudo gluster peer probe $NODE3
>>> sudo gluster volume create data replica 3 arbiter 1
>>> $NODE1:/gluster/data/brick $NODE2:/gluster/data/brick
>>> $NODE3:/gluster/data/brick force
>>> sudo gluster volume set data network.ping-timeout 5
>>> sudo gluster volume set data performance.client-io-threads on
>>> sudo gluster volume set data group metadata-cache
>>> sudo gluster volume start data
>>> sudo gluster volume info all
>>>
>>> Volume Name: data
>>> Type: Replicate
>>> Volume ID: b52b5212-82c8-4b1a-8db3-52468bc0226e
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x (2 + 1) = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: 10.54.95.123:/gluster/data/brick
>>> Brick2: 10.54.95.124:/gluster/data/brick
>>> Brick3: 10.54.95.125:/gluster/data/brick (arbiter)
>>> Options Reconfigured:
>>> network.inode-lru-limit: 200000
>>> performance.md-cache-timeout: 600
>>> performance.cache-invalidation: on
>>> performance.stat-prefetch: on
>>> features.cache-invalidation-timeout: 600
>>> features.cache-invalidation: on
>>> network.ping-timeout: 5
>>> transport.address-family: inet
>>> storage.fips-mode-rchecksum: on
>>> nfs.disable: on
>>> performance.client-io-threads: on
>>>
>>> sudo vim /etc/fstab
>>> localhost:/data             /data                 glusterfs
>>> defaults,_netdev      0 0
>>>
>>> sudo systemctl daemon-reload && sudo mount -a
>>> fio --name=test --filename=/data/wow --size=1G --readwrite=write
>>>
>>> Run status group 0 (all jobs):
>>>   WRITE: bw=109MiB/s (115MB/s), 109MiB/s-109MiB/s (115MB/s-115MB/s),
>>> io=1024MiB (1074MB), run=9366-9366msec
>>>
>>> Oh no, what's wrong? From 2182MB/s down to only 115MB/s? What am I
>>> missing? I'm not expecting the above ~17 Gbps, but I'm thinking it should
>>> at least be close(r) to ~10 Gbps.
>>>
>>> Any suggestions?
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://meet.google.com/cpu-eiue-hvk
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://meet.google.com/cpu-eiue-hvk
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20231212/81f80c55/attachment.html>