[Bugs] [Bug 1220011] New: Force replace-brick lead to the persistent write(use dd) return Input/output error

bugzilla at redhat.com bugzilla at redhat.com
Sat May 9 05:08:29 UTC 2015


https://bugzilla.redhat.com/show_bug.cgi?id=1220011

            Bug ID: 1220011
           Summary: Force replace-brick lead to the persistent write(use
                    dd) return Input/output error
           Product: GlusterFS
           Version: 3.7.0
         Component: disperse
          Keywords: Triaged
          Assignee: bugs at gluster.org
          Reporter: pkarampu at redhat.com
                CC: bugs at gluster.org, gluster-bugs at redhat.com,
                    iesool at 163.com, lidi at perabytes.com,
                    xhernandez at datalab.es
        Depends On: 1176062
            Blocks: 1183716



+++ This bug was initially created as a clone of Bug #1176062 +++

Description of problem:
    I mkdir /mountpoint/a/b/c -p, after that exec dd if=/dev/zero
of=/mountpoint/a/b/c/test.bak bs=1M.  then I relace-brick commit force. 
replace-brick success, but the write return Input/output error.

Version-Release number of selected component (if applicable):
 glusterfs-master or glusterfs-3.6.2beta1

How reproducible:


Steps to Reproduce:
1.I create a disperse 3 redundancy 1 volume

Volume Name: test
Type: Disperse
Volume ID: bfdbfc8e-3dcc-4459-a1e4-9de17df03db5
Status: Started
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: node-1:/sda/
Brick2: node-1:/sdb/
Brick3: node-1:/sdc/
Options Reconfigured:
features.quota: on
performance.high-prio-threads: 64
performance.low-prio-threads: 64
performance.least-prio-threads: 64
performance.normal-prio-threads: 64
performance.io-thread-count: 64
server.allow-insecure: on
features.lock-heal: on
network.ping-timeout: 5
performance.client-io-threads: enable

2.mkdir -p /mountpoint/a/b/c

3.dd if=/dev/zero of=/mountpoint/a/b/c/test.bak bs=1M

4.gluster volume replace-brick node-1:/sda node-1:/sdd commit force

Actual results:

replace-brick success, but dd write return Input/output error.

Expected results:

replace-brick success and the persistent write all should be OK.

Additional info:

--- Additional comment from jiademing on 2014-12-19 05:30:26 EST ---

I test the the persistent read also has this problem.(glusterfs-master or
glusterfs-release-3.6.2beta1)

--- Additional comment from Anand Avati on 2015-01-07 06:51:33 EST ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#1) for review on master by Xavier Hernandez (xhernandez at datalab.es)

--- Additional comment from jiademing on 2015-01-09 03:08:45 EST ---

(In reply to Anand Avati from comment #2)
> REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
> posted (#1) for review on master by Xavier Hernandez (xhernandez at datalab.es)

I test this patch, after force relpace-brick,it can persistent write, but  I ls
/mountpoint,  return Input/output error Occasionally. then I stop the dd write,
ls /mountpoint is OK.

--- Additional comment from jiademing on 2015-01-09 04:36:54 EST ---

(In reply to jiademing from comment #3)
> (In reply to Anand Avati from comment #2)
> > REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
> > posted (#1) for review on master by Xavier Hernandez (xhernandez at datalab.es)
> 
> I test this patch, after force relpace-brick,it can persistent write, but  I
> ls /mountpoint,  return Input/output error Occasionally. then I stop the dd
> write, ls /mountpoint is OK.


Error logs:

[2015-01-09 17:30:04.058135] E [ec-helpers.c:410:ec_loc_setup_path]
3-test-disperse-0: Invalid path '<gfid:060bd8ef-6e58-4fcd-ac21-2c0e85b70e54>'
in loc
[2015-01-09 17:30:04.058165] I [dht-layout.c:663:dht_layout_normalize]
3-test-dht: Found anomalies in <gfid:060bd8ef-6e58-4fcd-ac21-2c0e85b70e54>
(gfid = 060bd8ef-6e58-4fcd-ac21-2c0e85b70e54). Holes=1 overlaps=0
[2015-01-09 17:30:04.058187] W [fuse-resolve.c:147:fuse_resolve_gfid_cbk]
0-fuse: 060bd8ef-6e58-4fcd-ac21-2c0e85b70e54: failed to resolve (Input/output
error)
[2015-01-09 17:30:04.058201] E [fuse-bridge.c:808:fuse_getattr_resume]
0-digioceanfs-fuse: 47449: GETATTR 6883340
(060bd8ef-6e58-4fcd-ac21-2c0e85b70e54) resolution failed

--- Additional comment from Xavier Hernandez on 2015-01-09 05:59:53 EST ---

(In reply to jiademing from comment #3)
> (In reply to Anand Avati from comment #2)
> > REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
> > posted (#1) for review on master by Xavier Hernandez (xhernandez at datalab.es)
> 
> I test this patch, after force relpace-brick,it can persistent write, but  I
> ls /mountpoint,  return Input/output error Occasionally. then I stop the dd
> write, ls /mountpoint is OK.

I've tried to do an ls of <mountpoint>, <mountpoint>/a, <mountpoint>/a/b and
<mountpoint>/a/b/c while the dd was running in background and replace brick had
completed. I haven't seen any Input/Output error. However I've seen that 'ls'
sometimes takes more time than expected to complete. I'll try to see why.

The error logs you show seem to come from a different version of ec (program
lines do not match with current code). I've tried it with current master with
this patch added. What version are you trying ?

--- Additional comment from jiademing on 2015-01-12 01:01:00 EST ---

(In reply to Xavier Hernandez from comment #5)
> (In reply to jiademing from comment #3)
> > (In reply to Anand Avati from comment #2)
> > > REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
> > > posted (#1) for review on master by Xavier Hernandez (xhernandez at datalab.es)
> > 
> > I test this patch, after force relpace-brick,it can persistent write, but  I
> > ls /mountpoint,  return Input/output error Occasionally. then I stop the dd
> > write, ls /mountpoint is OK.
> 
> I've tried to do an ls of <mountpoint>, <mountpoint>/a, <mountpoint>/a/b and
> <mountpoint>/a/b/c while the dd was running in background and replace brick
> had completed. I haven't seen any Input/Output error. However I've seen that
> 'ls' sometimes takes more time than expected to complete. I'll try to see
> why.
> 
> The error logs you show seem to come from a different version of ec (program
> lines do not match with current code). I've tried it with current master
> with this patch added. What version are you trying ?

Sorry, I merged this patch by manual.Then I try on master + this patch, that's
OK.

--- Additional comment from Anand Avati on 2015-05-03 07:38:51 EDT ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#2) for review on master by Pranith Kumar Karampuri
(pkarampu at redhat.com)

--- Additional comment from Anand Avati on 2015-05-03 22:58:27 EDT ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#3) for review on master by Pranith Kumar Karampuri
(pkarampu at redhat.com)

--- Additional comment from Anand Avati on 2015-05-04 00:24:07 EDT ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#4) for review on master by Pranith Kumar Karampuri
(pkarampu at redhat.com)

--- Additional comment from Anand Avati on 2015-05-06 10:30:42 EDT ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#5) for review on master by Xavier Hernandez (xhernandez at datalab.es)

--- Additional comment from Anand Avati on 2015-05-06 12:51:05 EDT ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#6) for review on master by Xavier Hernandez (xhernandez at datalab.es)

--- Additional comment from Anand Avati on 2015-05-07 03:34:33 EDT ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#7) for review on master by Xavier Hernandez (xhernandez at datalab.es)

--- Additional comment from Anand Avati on 2015-05-07 06:51:44 EDT ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#8) for review on master by Xavier Hernandez (xhernandez at datalab.es)

--- Additional comment from Anand Avati on 2015-05-08 02:51:22 EDT ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#9) for review on master by Vijay Bellur (vbellur at redhat.com)

--- Additional comment from Anand Avati on 2015-05-09 01:07:21 EDT ---

REVIEW: http://review.gluster.org/9407 (ec: Fix failures with missing files)
posted (#10) for review on master by Pranith Kumar Karampuri
(pkarampu at redhat.com)


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1176062
[Bug 1176062] Force replace-brick lead to the persistent write(use dd)
return Input/output error
https://bugzilla.redhat.com/show_bug.cgi?id=1183716
[Bug 1183716] Force replace-brick lead to the persistent write(use dd)
return Input/output error
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list