[Bugs] [Bug 1250704] Random errors when reading multiple files in parallel on disperse volume

bugzilla at redhat.com bugzilla at redhat.com
Thu Aug 20 18:03:19 UTC 2015


https://bugzilla.redhat.com/show_bug.cgi?id=1250704



--- Comment #9 from Eivind Sarto <eivind at pacbell.net> ---
I am still getting read errors after turning performance parameters off.
Could there be a bug in CentOS 7.1 FUSE?
I do not see any read errors when I use fio-ioengine=gfapi.
I do not see any errors reading through FUSE with dispersed or
replicated-dispersed volumes.

I will attach the full log from the client.  It shows the read errors.

[root at science03 ~]# gluster volume info voltest
Volume Name: voltest
Type: Disperse
Volume ID: 4b04975b-b667-413a-a4d8-d39bbe5bbb3e
Status: Started
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: science03:/brick_ssd_1/gv
Brick2: science04:/brick_ssd_1/gv
Brick3: science05:/brick_ssd_1/gv
Brick4: science06:/brick_ssd_1/gv
Brick5: science07:/brick_ssd_1/gv
Brick6: science09:/brick_ssd_1/gv
Options Reconfigured:
performance.open-behind: off
performance.read-ahead: off
performance.stat-prefetch: off
performance.write-behind: off
performance.io-cache: off
performance.quick-read: off
performance.readdir-ahead: off

[root at science08 ~]# fio job.test
test: (g=0): rw=read, bs=64K-64K/64K-64K/64K-64K, ioengine=sync, iodepth=1
...
fio-2.2.8
Starting 8 threads
fio: io_u error on file /voltest/test.0: Input/output error: read
offset=47575040, buflen=4096
fio: pid=18783, err=5/file:io_u.c:1575, func=io_u error, error=Input/output
error
fio: io_u error on file /voltest/test.1: Input/output error: read
offset=987037696, buflen=65536
fio: pid=18790, err=5/file:io_u.c:1575, func=io_u error, error=Input/output
error
Jobs: 3 (f=3): [X(1),_(1),R(2),_(2),R(1),X(1)] [96.6% done] [81600KB/0KB/0KB
/s] [1275/0/0 iopsJobs: 3 (f=3): [X(1),_(1),R(2),_(2),R(1),X(1)] [98.9% done]
[66176KB/0KB/0KB /s] [1034/0/0 iopsJobs: 1 (f=1): [X(1),_(1),R(1),_(4),X(1)]
[100.0% done] [41280KB/0KB/0KB /s] [645/0/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=8): err= 5 (file:io_u.c:1575, func=io_u error,
error=Input/output error): pid=18783: Thu Aug 20 10:51:19 2015
  read : io=7130.6MB, bw=83996KB/s, iops=1312, runt= 86929msec
    clat (usec): min=6, max=892821, avg=5231.39, stdev=21733.34
     lat (usec): min=6, max=892821, avg=5231.54, stdev=21733.36
    clat percentiles (usec):
     |  1.00th=[    8],  5.00th=[   10], 10.00th=[   11], 20.00th=[   12],
     | 30.00th=[   13], 40.00th=[   15], 50.00th=[   16], 60.00th=[   22],
     | 70.00th=[   80], 80.00th=[ 8384], 90.00th=[12480], 95.00th=[22912],
     | 99.00th=[53504], 99.50th=[150528], 99.90th=[317440], 99.95th=[399360],
     | 99.99th=[561152]
    bw (KB  /s): min=  101, max=29401, per=14.96%, avg=12566.91, stdev=6000.23
    lat (usec) : 10=4.11%, 20=54.33%, 50=4.02%, 100=11.25%, 250=0.92%
    lat (usec) : 1000=0.01%
    lat (msec) : 2=0.01%, 4=0.11%, 10=9.86%, 20=9.73%, 50=4.50%
    lat (msec) : 100=0.59%, 250=0.38%, 500=0.16%, 750=0.02%, 1000=0.01%
  cpu          : usr=0.04%, sys=0.65%, ctx=35928, majf=0, minf=145
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued    : total=r=114102/w=0/d=0, short=r=11/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: io=7130.6MB, aggrb=83996KB/s, minb=83996KB/s, maxb=83996KB/s,
mint=86929msec, maxt=86929msec

[root at sccience08 ~]# cat job.test
[global]
#direct=1
ioengine=sync
size=1024m
rw=read
bs=64k
directory=/voltest
thread
group_reporting
numjobs=8
filename_format=test.$jobnum
[test]

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=VK7FhkZtsX&a=cc_unsubscribe


More information about the Bugs mailing list