[Gluster-users] glusterfs performance issue with fio fdatasync

Thu Jan 16 19:43:52 UTC 2020

On January 16, 2020 11:10:14 AM GMT+02:00, venky evr <venky.evr at gmail.com> wrote:
>Description of problem:
>
>I am getting bad performance with  glusterfs 7 with fio.
>Any help is greatly appreicated.
>
>With glusterfs :
>
>1 node
>1 brick
>1 volume (ssd volume) formated with ext4
>
>volume created and mounted with default values.
>
>
>
>fio --rw=write --ioengine=sync --fdatasync=1 --directory=/mnt/1
>--size=64m --bs=1024 --name=mytest
>
>
>
>fio-3.16
>Starting 1 process
>mytest2: Laying out IO file (1 file / 64MiB)
>Jobs: 1 (f=1): [W(1)][100.0%][w=2759KiB/s][w=2759 IOPS][eta 00m:00s]
>mytest2: (groupid=0, jobs=1): err= 0: pid=11970: Thu Jan 16 13:06:50
>2020
>  write: IOPS=2632, BW=2633KiB/s (2696kB/s)(64.0MiB/24892msec)
>    clat (usec): min=50, max=1433, avg=61.43, stdev=11.78
>     lat (usec): min=50, max=1435, avg=61.68, stdev=11.82
>    clat percentiles (usec):
>  |  1.00th=[   54],  5.00th=[   55], 10.00th=[   55], 20.00th=[   56],
>  | 30.00th=[   56], 40.00th=[   57], 50.00th=[   58], 60.00th=[   59],
>  | 70.00th=[   64], 80.00th=[   69], 90.00th=[   72], 95.00th=[   79],
>  | 99.00th=[  103], 99.50th=[  121], 99.90th=[  133], 99.95th=[  137],
>     | 99.99th=[  155]
>   bw (  KiB/s): min= 1834, max= 2752, per=99.94%, avg=2630.37,
>stdev=146.81, samples=49
>iops        : min= 1834, max= 2752, avg=2630.37, stdev=146.81,
>samples=49
>  lat (usec)   : 100=98.72%, 250=1.28%, 750=0.01%
>  lat (msec)   : 2=0.01%
>  fsync/fdatasync/sync_file_range:
>    sync (usec): min=245, max=13622, avg=315.54, stdev=84.93
>    sync percentiles (usec):
>  |  1.00th=[  255],  5.00th=[  262], 10.00th=[  265], 20.00th=[  269],
>  | 30.00th=[  273], 40.00th=[  277], 50.00th=[  281], 60.00th=[  289],
>  | 70.00th=[  310], 80.00th=[  400], 90.00th=[  416], 95.00th=[  433],
>  | 99.00th=[  486], 99.50th=[  519], 99.90th=[  676], 99.95th=[  848],
>     | 99.99th=[  914]
>  cpu          : usr=1.41%, sys=3.50%, ctx=196761, majf=0, minf=13
>IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>=64=0.0%
>submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>=64=0.0%
>complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>=64=0.0%
>     issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0
>     latency   : target=0, window=0, percentile=100.00%, depth=1
>
>Run status group 0 (all jobs):
>  WRITE: bw=2633KiB/s (2696kB/s), 2633KiB/s-2633KiB/s
>(2696kB/s-2696kB/s), io=64.0MiB (67.1MB), run=24892-24892msec
>
>
>Without Glusterfs:
>
>
>Ext4 filesystem created and mounted locally, there is no gluster
>involved during this test
>
> mount /dev/sdb1  /mnt/1
>
>
>Starting 1 process
>mytest: Laying out IO file (1 file / 64MiB)
>Jobs: 1 (f=1): [W(1)][100.0%][w=10.2MiB/s][w=10.5k IOPS][eta 00m:00s]
>mytest: (groupid=0, jobs=1): err= 0: pid=9308: Thu Jan 16 13:00:20 2020
>  write: IOPS=10.2k, BW=9.99MiB/s (10.5MB/s)(64.0MiB/6407msec)
>    clat (nsec): min=1257, max=831943, avg=2674.65, stdev=10327.82
>     lat (nsec): min=1323, max=832038, avg=2783.73, stdev=10330.58
>    clat percentiles (nsec):
>|  1.00th=[  1352],  5.00th=[  1400], 10.00th=[  1448], 20.00th=[ 
>1496],
>| 30.00th=[  1560], 40.00th=[  1656], 50.00th=[  2320], 60.00th=[ 
>2448],
>| 70.00th=[  2864], 80.00th=[  2992], 90.00th=[  3280], 95.00th=[ 
>4384],
>| 99.00th=[  8896], 99.50th=[ 11328], 99.90th=[ 20608],
>99.95th=[222208],
>     | 99.99th=[464896]
>   bw (  KiB/s): min= 8946, max=10536, per=99.92%, avg=10219.75,
>stdev=441.53, samples=12
>iops        : min= 8946, max=10536, avg=10219.75, stdev=441.53,
>samples=12
>  lat (usec)   : 2=42.86%, 4=50.26%, 10=6.19%, 20=0.57%, 50=0.06%
>  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.04%, 750=0.01%, 1000=0.01%
>  fsync/fdatasync/sync_file_range:
>    sync (usec): min=50, max=1174, avg=93.61, stdev=62.99
>    sync percentiles (usec):
>  |  1.00th=[   52],  5.00th=[   53], 10.00th=[   54], 20.00th=[   55],
>  | 30.00th=[   55], 40.00th=[   56], 50.00th=[   57], 60.00th=[   61],
>  | 70.00th=[   82], 80.00th=[  178], 90.00th=[  194], 95.00th=[  215],
>  | 99.00th=[  253], 99.50th=[  265], 99.90th=[  334], 99.95th=[  627],
>     | 99.99th=[  758]
>  cpu          : usr=2.68%, sys=16.98%, ctx=81932, majf=0, minf=16
>IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>>=64=0.0%
>submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>=64=0.0%
>complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>>=64=0.0%
>     issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0
>     latency   : target=0, window=0, percentile=100.00%, depth=1
>
>Run status group 0 (all jobs):
>  WRITE: bw=9.99MiB/s (10.5MB/s), 9.99MiB/s-9.99MiB/s
>(10.5MB/s-10.5MB/s), io=64.0MiB (67.1MB), run=6407-6407msec
>
>Disk stats (read/write):
>sdb: ios=0/98184, merge=0/16369, ticks=0/4804, in_queue=4788,
>util=73.71%
>
>
>
>
>
>we tried following tuning parameters, but performance didnt increase.
>
>performance. Client-io-threads on
>performance.io-thread-count 64
>server. Event-threads 4 or 8
>client. Event-threads 4 or 8

Something that come to my mind:
1. Switch to XFS
2. Set I/O scheduler of the brick  to none/noop
3. Mount  options of the brick : 'noatime' and if there  is battery-backed cache  'nobarrier'
4. Set reasonable value  for 'vm.dirty_background_bytes' &  'vm.dirty_bytes' to a reasonable values
5. Disable gluster logs or raise them at ERR or higher.

Check if any of the described 'profiles' in /var/lib/gluster/groups' match your workload.

Best Regards,
Strahil Nikolov