[Bugs] [Bug 1306131] New: Attach tier : Creates fail with invalid argument errors

Wed Feb 10 05:57:38 UTC 2016

https://bugzilla.redhat.com/show_bug.cgi?id=1306131

            Bug ID: 1306131
           Summary: Attach tier : Creates fail with invalid argument
                    errors
           Product: GlusterFS
           Version: 3.7.7
         Component: tiering
          Keywords: ZStream
          Assignee: bugs at gluster.org
          Reporter: rkavunga at redhat.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org, byarlaga at redhat.com,
                    dlambrig at redhat.com, nchilaka at redhat.com,
                    ndevos at redhat.com, nsathyan at redhat.com,
                    rcyriac at redhat.com, rjoseph at redhat.com,
                    skoduri at redhat.com
        Depends On: 1296048, 1297311

+++ This bug was initially created as a clone of Bug #1297311 +++

+++ This bug was initially created as a clone of Bug #1296048 +++

Description of problem:
=======================

Created a 2x(4+2) EC volume and nfs mounted on the client with quota and uss
enabled. Started IO (linux untar, mkdir's, dd (parallel - 1000's). Tried
attaching the tier (2x2 dist-rep) and seeing invalid argument errors
continuously. The same errors were seen during detach-tier but IO resumed after
some time. In this case, complete IO fails with the error messages. 

If quota and uss are turned off, below errors are seen for some time and then
the IO resumes.

tar:
linux-4.1.1/Documentation/devicetree/bindings/input/touchscreen/zforce_ts.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/input/tps65218-pwrbutton.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/input/tps65218-pwrbutton.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-keypad.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-keypad.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-pwrbutton.txt
tar: linux-4.1.1/Documentation/devicetree/bindings/input/twl4030-pwrbutton.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/abilis,tb10x-ictl.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/abilis,tb10x-ictl.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun4i-ic.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun4i-ic.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun67i-sc-nmi.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/allwinner,sun67i-sc-nmi.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/atmel,aic.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/atmel,aic.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm2835-armctrl-ic.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm2835-armctrl-ic.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm3380-l2-intc.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm3380-l2-intc.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7038-l1-intc.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7038-l1-intc.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7120-l2-intc.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,bcm7120-l2-intc.txt:
Cannot open: File exists
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,l2-intc.txt
tar:
linux-4.1.1/Documentation/devicetree/bindings/interrupt-controller/brcm,l2-intc.txt:
Cannot open: File exists

Version-Release number of selected component (if applicable):
===============================================================
3.7.5-14

How reproducible:
=================
100%

Steps to Reproduce:
1. Create a disperse volume 2x(4+2)
2. NFS mount on the client.
3. Start IO (linux untar - 2 instances, mkdir (1000 in parallel), dd (1000 in
parallel)
4. Attach tier (2x2 dist-rep) volume

Actual results:
===============
Invalid argument errors

Expected results:
=================
No errors to be seen and IO should be smooth.

Additional info:
================
sosrepots in rhsqe.

--- Additional comment from RHEL Product and Program Management on 2016-01-06
05:10:03 EST ---

This request has been proposed as a blocker, but a release flag has
not been requested. Please set a release flag to ? to ensure we may
track this bug against the appropriate upcoming release, and reset
the blocker flag to ?.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-01-06
09:28:10 EST ---

This bug is automatically being proposed for the current z-stream release of
Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change
the proposed release flag.

--- Additional comment from Mohammed Rafi KC on 2016-01-11 04:02:31 EST ---

RCA:

After add-brick, NFS server will be restarted to load new graph. ie NFS server
inode table will be fresh after restarting the process. So as part of the fop,
resolver will send a lookup on an entry if inode is not lookedup before. During
the lookup if healing requires for the entry from DHT (when directories are not
present on all of the subvol), we will initiate a healing to create the
directories on all of the subvolume. As part of the healing, we are doing a
series of named lookup on all the parents starting from root if the inodes are
not present, so for a successful lookup we will link the inode to inode table
also. This lookup will be initiated from dht, so inode ctx will be created only
for the xlators which are beneath of dht.

Since we already linked the inode, ie  resolver will not do a lookup for next
fop. So xlator which are above dht will not have inode ctx.

Here in this case, svc_access was complaining about missing inode_ctx.

Possible solutions:

1)  Move dht healing code to interface layer, if healing is required then dht
should let the interface layer about healing, and need to give a path to heal.
So that each interface layer should do a healing which include fuse, nfs,
gfapi.

2) Do not link the inode from any of the xlators other than master xlators, ie
do not link from dht. This will cause a huge performance degradation in healing
code path, and we might need to do some hack to heal without a linked inode.

3) During resolving of an entry, currently resolving will be successful if
there is an inode in the inode table. Make an extra check to see if the
inode_ctx is present or not, if inode_ctx is not present for a linked inode,
then resolver should consider as an invalid inode and need to do a lookup with
the same inode.

--- Additional comment from Vijay Bellur on 2016-01-12 01:50:20 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not
set) posted (#1) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-12 02:37:09 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not
set) posted (#1) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-12 04:36:11 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not
set) posted (#1) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-12 04:40:38 EST ---

REVIEW: http://review.gluster.org/13227 (snapview-client: remove check for
parent inode type) posted (#1) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-12 07:30:19 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not
set) posted (#2) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-12 07:30:28 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not
set) posted (#2) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-12 07:34:32 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not
set) posted (#2) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-12 08:52:19 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not
set) posted (#3) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-12 09:04:10 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not
set) posted (#3) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 01:07:04 EST ---

COMMIT: http://review.gluster.org/13227 committed in master by Rajesh Joseph
(rjoseph at redhat.com) 
------
commit e9d20f0e1a17e0ef2f148a9b49d1cf9596cbe609
Author: Mohammed Rafi KC <rkavunga at redhat.com>
Date:   Tue Jan 12 12:13:15 2016 +0530

    snapview-client: remove check for parent inode type

    In lookup call back path, we are checking for parent inode
    type to make sure parent inode was set properly, so that
    next op on the inode can be successfuly completed.

    More info can be found at bugid 1297311

    Change-Id: Ifb1b17f472f855f9b12088c3dd8328389d895e77
    BUG: 1297311
    Signed-off-by: Mohammed Rafi KC <rkavunga at redhat.com>
    Reviewed-on: http://review.gluster.org/13227
    Reviewed-by: Avra Sengupta <asengupt at redhat.com>
    Tested-by: NetBSD Build System <jenkins at build.gluster.org>
    Reviewed-by: Rajesh Joseph <rjoseph at redhat.com>

--- Additional comment from Vijay Bellur on 2016-01-13 01:52:14 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not
set) posted (#4) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 01:52:22 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not
set) posted (#3) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 01:59:13 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not
set) posted (#4) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 04:03:18 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not
set) posted (#5) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 04:03:23 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not
set) posted (#4) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 05:54:40 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not
set) posted (#5) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 05:58:55 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not
set) posted (#6) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 06:33:53 EST ---

REVIEW: http://review.gluster.org/13224 (nfs: send lookup if inode_ctx is not
set) posted (#7) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 08:14:27 EST ---

REVIEW: http://review.gluster.org/13226 (gfapi: send lookup if inode_ctx is not
set) posted (#6) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 08:14:32 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not
set) posted (#5) for review on master by mohammed rafi  kc
(rkavunga at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 20:35:26 EST ---

COMMIT: http://review.gluster.org/13224 committed in master by Dan Lambright
(dlambrig at redhat.com) 
------
commit 14f925f5262ecabb2faf8142267c37103413e189
Author: Mohammed Rafi KC <rkavunga at redhat.com>
Date:   Tue Jan 12 12:04:59 2016 +0530

    nfs: send lookup if inode_ctx is not set

    During resolving of an entry or inode, if inode ctx
    was not set, we will send a lookup.

    This patch also make sure that inode_ctx will be created
    after every inode_link.

    Change-Id: I137a7e2510635ff4ea6d007b671961341f89c949
    BUG: 1297311
    Signed-off-by: Mohammed Rafi KC <rkavunga at redhat.com>
    Reviewed-on: http://review.gluster.org/13224
    Reviewed-by: soumya k <skoduri at redhat.com>
    Tested-by: NetBSD Build System <jenkins at build.gluster.org>
    Reviewed-by: Dan Lambright <dlambrig at redhat.com>
    Tested-by: Dan Lambright <dlambrig at redhat.com>

--- Additional comment from Vijay Bellur on 2016-01-13 22:20:32 EST ---

REVIEW: http://review.gluster.org/13225 (fuse: send lookup if inode_ctx is not
set) posted (#6) for review on master by Dan Lambright (dlambrig at redhat.com)

--- Additional comment from Vijay Bellur on 2016-01-13 22:26:46 EST ---

COMMIT: http://review.gluster.org/13225 committed in master by Dan Lambright
(dlambrig at redhat.com) 
------
commit ad8c65612d6d7e32c06fd3aed11b7a9edd514607
Author: Mohammed Rafi KC <rkavunga at redhat.com>
Date:   Tue Jan 12 12:54:42 2016 +0530

    fuse: send lookup if inode_ctx is not set

    During resolving of an entry or inode, if inode ctx
    was not set, we will send a lookup.

    This patch also make sure that inode_ctx will be created
    after every inode_link

    Change-Id: I4211533ca96a51b89d9f010fc57133470e52dc11
    BUG: 1297311
    Signed-off-by: Mohammed Rafi KC <rkavunga at redhat.com>
    Reviewed-on: http://review.gluster.org/13225
    Reviewed-by: Dan Lambright <dlambrig at redhat.com>
    Tested-by: Dan Lambright <dlambrig at redhat.com>

--- Additional comment from Vijay Bellur on 2016-01-13 22:27:49 EST ---

COMMIT: http://review.gluster.org/13226 committed in master by Dan Lambright
(dlambrig at redhat.com) 
------
commit 252fe523e2932e83a98f203f71678f391eee6b22
Author: Mohammed Rafi KC <rkavunga at redhat.com>
Date:   Tue Jan 12 15:04:46 2016 +0530

    gfapi: send lookup if inode_ctx is not set

    During resolving of an entry or inode, if inode ctx
    was not set, we will send a lookup to pupulate inode
    ctx for every xlators

    This patch also make sure that inode_ctx will be created
    after every inode_link. We will store inode_ctx value as
    LOOKUP_NEEDED if the inode is liked via readdirp, in all
    other case we will store inode_ctx value as LOOKUP_NOT_NEEDED.

    Change-Id: I3a10c298944200fa3862127187ae8988e582d352
    BUG: 1297311
    Signed-off-by: Mohammed Rafi KC <rkavunga at redhat.com>
    Reviewed-on: http://review.gluster.org/13226
    Reviewed-by: Poornima G <pgurusid at redhat.com>
    Tested-by: NetBSD Build System <jenkins at build.gluster.org>
    Reviewed-by: Shyamsundar Ranganathan <srangana at redhat.com>
    Reviewed-by: Dan Lambright <dlambrig at redhat.com>
    Tested-by: Dan Lambright <dlambrig at redhat.com>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1296048
[Bug 1296048] Attach tier + nfs : Creates fail with invalid argument errors
https://bugzilla.redhat.com/show_bug.cgi?id=1297311
[Bug 1297311] Attach tier : Creates fail with invalid argument errors
-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.