[Gluster-users] fio testing performance problem

Mon Jan 26 03:22:56 UTC 2015

Hi:

environment:
node A :client with fio running

node B :brick
network：1G

Problem is fio with libgfapi getting poor performance than fio whit 
fuse,what could be the reason?

[fuse]
/usr/local/bin/fio --filename=/home/gluster/file --direct=1 --rw=randwrite 
--bs=32k --size=20G --numjobs=60  --runtime=120 --group_reporting 
--name=test-read  --ioengine=libaio --iodepth=1
test-read: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, 
ioengine=libaio, iodepth=1
...
fio-2.2.5
Starting 60 processes
Jobs: 60 (f=60): [w(60)] [100.0% done] [0KB/103.8MB/0KB /s] [0/3318/0 
iops] [eta 00m:00s]
test-read: (groupid=0, jobs=60): err= 0: pid=46659: Mon Jan 26 04:40:28 
2015
  write: io=12804MB, bw=109247KB/s, iops=3413, runt=120016msec
    slat (usec): min=46, max=14914, avg=546.08, stdev=1596.17
    clat (usec): min=73, max=123085, avg=17017.70, stdev=4057.91
     lat (usec): min=208, max=123264, avg=17564.57, stdev=3443.93
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[   11], 10.00th=[   16], 20.00th=[   17],
     | 30.00th=[   17], 40.00th=[   17], 50.00th=[   18], 60.00th=[   18],
     | 70.00th=[   18], 80.00th=[   19], 90.00th=[   19], 95.00th=[   20],
     | 99.00th=[   35], 99.50th=[   38], 99.90th=[   48], 99.95th=[   56],
     | 99.99th=[  118]
    bw (KB  /s): min= 1418, max= 2091, per=1.67%, avg=1822.44, stdev=66.16
    lat (usec) : 100=0.18%, 250=0.11%, 500=0.04%, 750=0.03%, 1000=0.04%
    lat (msec) : 2=0.20%, 4=0.55%, 10=3.41%, 20=92.28%, 50=3.08%
    lat (msec) : 100=0.05%, 250=0.03%
  cpu          : usr=0.11%, sys=0.22%, ctx=955257, majf=0, minf=1696
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
     issued    : total=r=0/w=409732/d=0, short=r=0/w=0/d=0, 
drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: io=12804MB, aggrb=109247KB/s, minb=109247KB/s, maxb=109247KB/s, 
mint=120016msec, maxt=120016msec

[libgfapi]
/usr/local/bin/fio --direct=1 --rw=randwrite --bs=32k --size=20g 
--runtime=120 --numjobs=60 --group_reporting --ioengine=gfapi 
--volume=gluster_*** --brick=*** --name=/file
/file: (g=0): rw=randwrite, bs=32K-32K/32K-32K/32K-32K, ioengine=gfapi, 
iodepth=1
...
fio-2.2.5
Starting 60 processes
Jobs: 42 (f=0): 
[w(3),_(5),w(1),_(1),w(1),_(1),w(1),_(1),w(1),_(2),w(4),_(3),w(1),_(1),w(1),_(1),w(1),_(1),w(3),_(1),w(5),_(1),w(20)] 
[5.4% done] [0KB/3168KB/0KB /s] [0/99/0 iops] [eta 36m:17s]
/file: (groupid=0, jobs=60): err= 0: pid=46325: Mon Jan 26 04:37:13 2015
  write: io=11288MB, bw=96260KB/s, iops=3008, runt=120075msec
    clat (usec): min=11, max=488899, avg=19927.46, stdev=15351.15
     lat (usec): min=12, max=488901, avg=19928.92, stdev=15351.30
    clat percentiles (usec):
     |  1.00th=[  217],  5.00th=[ 1496], 10.00th=[ 2896], 20.00th=[ 6048],
     | 30.00th=[10048], 40.00th=[13888], 50.00th=[17792], 60.00th=[21888],
     | 70.00th=[27008], 80.00th=[31360], 90.00th=[36096], 95.00th=[46336],
     | 99.00th=[66048], 99.50th=[74240], 99.90th=[101888], 
99.95th=[126464],
     | 99.99th=[189440]
    bw (KB  /s): min=  101, max=27130, per=1.68%, avg=1615.36, 
stdev=900.74
    lat (usec) : 20=0.34%, 50=0.19%, 100=0.14%, 250=0.44%, 500=0.65%
    lat (usec) : 750=0.77%, 1000=0.87%
    lat (msec) : 2=3.49%, 4=6.76%, 10=16.36%, 20=25.76%, 50=40.16%
    lat (msec) : 100=3.95%, 250=0.12%, 500=0.01%
  cpu          : usr=0.39%, sys=0.17%, ctx=362268, majf=0, minf=3608
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, 
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, 
>=64=0.0%
     issued    : total=r=0/w=361200/d=0, short=r=0/w=0/d=0, 
drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: io=11288MB, aggrb=96259KB/s, minb=96259KB/s, maxb=96259KB/s, 
mint=120075msec, maxt=120075msec
--------------------------------------------------------
ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s).  If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited.  If you have received this mail in error, please delete it and notify us immediately.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150126/cfff8715/attachment.html>