[Gluster-users] glusterfs performance issue with fio fdatasync

[venky evr]

Description of problem:

I am getting bad performance with  glusterfs 7 with fio.
Any help is greatly appreicated.

With glusterfs :

1 node
1 brick
1 volume (ssd volume) formated with ext4

volume created and mounted with default values.



fio --rw=write --ioengine=sync --fdatasync=1 --directory=/mnt/1
--size=64m --bs=1024 --name=mytest



fio-3.16
Starting 1 process
mytest2: Laying out IO file (1 file / 64MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=2759KiB/s][w=2759 IOPS][eta 00m:00s]
mytest2: (groupid=0, jobs=1): err= 0: pid=11970: Thu Jan 16 13:06:50 2020
  write: IOPS=2632, BW=2633KiB/s (2696kB/s)(64.0MiB/24892msec)
    clat (usec): min=50, max=1433, avg=61.43, stdev=11.78
     lat (usec): min=50, max=1435, avg=61.68, stdev=11.82
    clat percentiles (usec):
     |  1.00th=[   54],  5.00th=[   55], 10.00th=[   55], 20.00th=[   56],
     | 30.00th=[   56], 40.00th=[   57], 50.00th=[   58], 60.00th=[   59],
     | 70.00th=[   64], 80.00th=[   69], 90.00th=[   72], 95.00th=[   79],
     | 99.00th=[  103], 99.50th=[  121], 99.90th=[  133], 99.95th=[  137],
     | 99.99th=[  155]
   bw (  KiB/s): min= 1834, max= 2752, per=99.94%, avg=2630.37,
stdev=146.81, samples=49
   iops        : min= 1834, max= 2752, avg=2630.37, stdev=146.81, samples=49
  lat (usec)   : 100=98.72%, 250=1.28%, 750=0.01%
  lat (msec)   : 2=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=245, max=13622, avg=315.54, stdev=84.93
    sync percentiles (usec):
     |  1.00th=[  255],  5.00th=[  262], 10.00th=[  265], 20.00th=[  269],
     | 30.00th=[  273], 40.00th=[  277], 50.00th=[  281], 60.00th=[  289],
     | 70.00th=[  310], 80.00th=[  400], 90.00th=[  416], 95.00th=[  433],
     | 99.00th=[  486], 99.50th=[  519], 99.90th=[  676], 99.95th=[  848],
     | 99.99th=[  914]
  cpu          : usr=1.41%, sys=3.50%, ctx=196761, majf=0, minf=13
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2633KiB/s (2696kB/s), 2633KiB/s-2633KiB/s
(2696kB/s-2696kB/s), io=64.0MiB (67.1MB), run=24892-24892msec


Without Glusterfs:


Ext4 filesystem created and mounted locally, there is no gluster
involved during this test

 mount /dev/sdb1  /mnt/1


Starting 1 process
mytest: Laying out IO file (1 file / 64MiB)
Jobs: 1 (f=1): [W(1)][100.0%][w=10.2MiB/s][w=10.5k IOPS][eta 00m:00s]
mytest: (groupid=0, jobs=1): err= 0: pid=9308: Thu Jan 16 13:00:20 2020
  write: IOPS=10.2k, BW=9.99MiB/s (10.5MB/s)(64.0MiB/6407msec)
    clat (nsec): min=1257, max=831943, avg=2674.65, stdev=10327.82
     lat (nsec): min=1323, max=832038, avg=2783.73, stdev=10330.58
    clat percentiles (nsec):
     |  1.00th=[  1352],  5.00th=[  1400], 10.00th=[  1448], 20.00th=[  1496],
     | 30.00th=[  1560], 40.00th=[  1656], 50.00th=[  2320], 60.00th=[  2448],
     | 70.00th=[  2864], 80.00th=[  2992], 90.00th=[  3280], 95.00th=[  4384],
     | 99.00th=[  8896], 99.50th=[ 11328], 99.90th=[ 20608], 99.95th=[222208],
     | 99.99th=[464896]
   bw (  KiB/s): min= 8946, max=10536, per=99.92%, avg=10219.75,
stdev=441.53, samples=12
   iops        : min= 8946, max=10536, avg=10219.75, stdev=441.53, samples=12
  lat (usec)   : 2=42.86%, 4=50.26%, 10=6.19%, 20=0.57%, 50=0.06%
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.04%, 750=0.01%, 1000=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=50, max=1174, avg=93.61, stdev=62.99
    sync percentiles (usec):
     |  1.00th=[   52],  5.00th=[   53], 10.00th=[   54], 20.00th=[   55],
     | 30.00th=[   55], 40.00th=[   56], 50.00th=[   57], 60.00th=[   61],
     | 70.00th=[   82], 80.00th=[  178], 90.00th=[  194], 95.00th=[  215],
     | 99.00th=[  253], 99.50th=[  265], 99.90th=[  334], 99.95th=[  627],
     | 99.99th=[  758]
  cpu          : usr=2.68%, sys=16.98%, ctx=81932, majf=0, minf=16
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,65536,0,0 short=65535,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=9.99MiB/s (10.5MB/s), 9.99MiB/s-9.99MiB/s
(10.5MB/s-10.5MB/s), io=64.0MiB (67.1MB), run=6407-6407msec

Disk stats (read/write):
  sdb: ios=0/98184, merge=0/16369, ticks=0/4804, in_queue=4788, util=73.71%





we tried following tuning parameters, but performance didnt increase.

performance. Client-io-threads on
performance.io-thread-count 64
server. Event-threads 4 or 8
client. Event-threads 4 or 8
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200116/69d8126a/attachment.html>