[Gluster-users] Understanding gluster performance

[Gionatan Danti]

Hi all,
I would like to better understand gluster performance and how to 
profile/analyze them.

I set up two old test machine each with a quad-core i7 CPU, 8 GB RAM and 
4x 5400 RPM disks in software RAID 10. OS is CentOS 8.1 and I am using 
Gluster 6.7. To avoid being limited by the mechanical HDD, I created a 
small replica 2 volume using /dev/shm/gluster for both bricks.

Testing using fio with 4k random write and fsync=1, I see the following:
- write to /dev/shm: almost 300K IOPs (expected, as I am writing in 
memory);
- write to /mnt/fuse (where the volume is mounted): 250 IOPs;
- as above, but something I get as low as 20-30 IOPs for minutes.

Writing to /mnt/fuse without fsync=1 I get about 5500 IOPs, without the 
strange (and very low) drops. In this case, I seems limited by CPU 
because both glusterd and glusterfsd are near/above 100% core 
utilization.

Further details and volume info can be found below [1]

So, I have some questions:
- why performance are so low with fsync?
- why do I have so low IOPs (20/30) for minutes?
- what is capping the non-fsync test?
- why both glusterd and glusterfd are so CPU intensive? I can understand 
glusterfd itself requiring more CPU, but glusterd should only manage the 
other processes and send the volume information to asking client, right?

Thanks.

[1] [root at blackhole gluster]# gluster volume info tv0
Volume Name: tv0
Type: Replicate
Volume ID: 96534afe-bfde-4d60-a94d-379278ab45c4
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: singularity:/dev/shm/gluster
Brick2: blackhole:/dev/shm/gluster
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

[root at blackhole tmp]# mount | grep glusterfs
localhost:tv0 on /mnt/fuse type fuse.glusterfs 
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)

[root at blackhole glusterfs]# fio --name=test 
--filename=/mnt/fuse/test.img --size=256M --rw=randwrite --fsync=1
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 
4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][11.6%][r=0KiB/s,w=976KiB/s][r=0,w=244 IOPS][eta 
04m:19s]

[root at blackhole tmp]# fio --name=test --filename=/mnt/fuse/test.img 
--size=256M --rw=randwrite --fsync=1
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 
4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][1.8%][r=0KiB/s,w=112KiB/s][r=0,w=28 IOPS][eta 
38m:16s]

[root at blackhole tmp]# fio --name=test --filename=/mnt/fuse/test.img 
--size=256M --rw=randwrite
test: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 
4096B-4096B, ioengine=psync, iodepth=1
fio-3.7
Starting 1 process
Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=21.3MiB/s][r=0,w=5462 IOPS][eta 
00m:00s]

[root at blackhole gluster]# gluster volume heal tv0 info
Brick singularity:/dev/shm/gluster
Status: Connected
Number of entries: 0
Brick blackhole:/dev/shm/gluster
Status: Connected
Number of entries: 0

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti at assyoma.it - info at assyoma.it
GPG public key ID: FF5F32A8