[Gluster-users] replica performance and brick size best practice

[Strahil Nikolov]

Hi,
First you need to identify what kind of workload you will have.Some optimizations for one workload can prevent better performance in another type.
If you plan to use the volume for small files , this document is a good start https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/small_file_performance_enhancements
Replica 2 volumes are prone to split brain and it's always recommended to have a replica 3 or an arbiter .
As a best practice - always test the volume with the application that will use it as synthetic benchmarks are just synthetics.
I always start performance review from the bottom (infrastructure) and end on application level.
You can start with https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.5/html/administration_guide/chap-configuring_red_hat_storage_for_enhancing_performance as the storage is one of the most important parts.
What kind of HW raid , how many disks, stripe size and stripe width that you used in LVM?Do you use thin LVM ?How did you create your XFS (isize is critical) ?
Have you used gluster's tuned profile ?
Jumbo frames ?
Then you will need to optimize the volume for small files (see the link above).
Does you app allow you to use libgfapi ? Based on my observations in the oVirt list, libgfapi it used to provide some performance benefits compared to fuse .
Also, if you work with very small files, it would make sense to combine them in some container (like in VM disks).
Keep in mind that GlusterFS performance scales with the size of the cluster and the number of clients. For ultra high performance for a few clients -> there are other options.
Best Regards,Strahil Nikolov  
 
  On Wed, Nov 9, 2022 at 12:05, beer Ll<llcfhllml at gmail.com> wrote:   Hi 

I have 2 gluster server (backup1 , backup2)  connected with 10Gbit linkglusterfs version 10.3-1  server and client

Each server is  with 44T disk raid 6 with 1 partition used for the brick

 /dev/VG_BT_BRICK1/LV_BT_BRICK1   [      43.66 TiB] 
/dev/mapper/VG_BT_BRICK1-LV_BT_BRICK1 on /mnt/glusterfs type xfs (rw,noatime,nouuid,attr2,inode64,sunit=256,swidth=1536,noquota)

I create a replica volume named backup

Volume Name: backup
Type: Replicate
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: backup-1:/mnt/glusterfs/brick1
Brick2: backup-2:/mnt/glusterfs/brick1
Options Reconfigured:
cluster.granular-entry-heal: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
cluster.enable-shared-storage: enable

The volume backup is mounted with gluster client on /mnt/share

backup-1:/backup on /mnt/share type fuse.glusterfs (rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)


Test with smallfile utility 

XFS
with filesystem xfs on  /mnt/glusterfs 

total threads = 8
total files = 800
100.00% of requested files processed, warning threshold is  70.00%
elapsed time =     0.009
files/sec = 120927.642211

GLUSTERFS CLIENT
with  glusterfs on /mnt/share
total threads = 8
total files = 800
100.00% of requested files processed, warning threshold is  70.00%
elapsed time =     3.014
files/sec = 284.975861



How is it possible to increase the performance in glusterfs volume ? 
How is best practice of brick size and replica management ?Is better 1 big brick per server or more small bricks distributed ?
Many Thanks 





________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://meet.google.com/cpu-eiue-hvk
Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20221113/cd13d6b1/attachment.html>