[Bugs] [Bug 1467010] New: Fd based fops fail with EBADF on file migration

Sat Jul 1 15:39:14 UTC 2017

https://bugzilla.redhat.com/show_bug.cgi?id=1467010

            Bug ID: 1467010
           Summary: Fd based fops fail with EBADF on file migration
           Product: GlusterFS
           Version: 3.10
         Component: distribute
          Assignee: bugs at gluster.org
          Reporter: rtalur at redhat.com
                CC: bugs at gluster.org, nbalacha at redhat.com
        Depends On: 1465075
            Blocks: 1463907, 1465123

+++ This bug was initially created as a clone of Bug #1465075 +++

Description of problem:

In DHT, there exists a scenario where fd based fops may be sent on the dst
subvolume after the file has been migrated but before the fd has been opened on
it. This is because certain operations update the cached subvol in the dht
inode ctx without checking to see if an fd has been opened on it on the
original subvol. Dht fd based fops currently rely on a phase1/phase2 migration
checks to open fds on the dst subvol. However, no such check is made causing
the fop to fail with EBADF.

This is seen with dist-rep volumes.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:

1. Create a 2x2 volume
2. Create a file FILE1. Assume it is created on subvol1. Rename it to NFILE1 so
it hashes to subvol2.
3. Open an fd on NFILE1 (on subvol1).
2. Perform a rebalance so the file is migrated to subvol2.                      
3. On the same mount point, perform a lookup/readdirp so the cached subvol in
the inode_ctx of NFILE1 is updated to subvol2.          
4. Perform a write on the fd.

The write is sent to subvol2 on an fd which has been opened only on subvol1.   
     Since the migration phase checks don't kick in, the fd is not opened on
subvol2 and the fop fails with EBADF.

Actual results:

Expected results:

Additional info:

This is being fixed by having every fd based fop check if the fd has been
opened on the cached subvol before winding the fop down.

--- Additional comment from Worker Ant on 2017-06-26 21:21:07 IST ---

REVIEW: https://review.gluster.org/17630 (cluster/dht: Check if fd is opened on
dst subvol) posted (#1) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-06-27 16:56:10 IST ---

REVIEW: https://review.gluster.org/17630 (cluster/dht: Check if fd is opened on
dst subvol) posted (#2) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-06-27 21:17:07 IST ---

REVIEW: https://review.gluster.org/17630 (cluster/dht: Check if fd is opened on
dst subvol) posted (#3) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-06-27 22:21:29 IST ---

REVIEW: https://review.gluster.org/17630 (cluster/dht: Check if fd is opened on
dst subvol) posted (#4) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-06-28 09:55:55 IST ---

REVIEW: https://review.gluster.org/17630 (cluster/dht: Check if fd is opened on
dst subvol) posted (#5) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-06-28 12:39:48 IST ---

REVIEW: https://review.gluster.org/17630 (cluster/dht: Check if fd is opened on
dst subvol) posted (#6) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-06-28 17:12:25 IST ---

COMMIT: https://review.gluster.org/17630 committed in master by Raghavendra G
(rgowdapp at redhat.com) 
------
commit 91db0d47ca267aecfc6124a3f337a4e2f2c9f1e2
Author: N Balachandran <nbalacha at redhat.com>
Date:   Mon Jun 26 21:12:56 2017 +0530

    cluster/dht: Check if fd is opened on dst subvol

    If an fd is opened on a file, the file is migrated
    and the cached subvol is updated in the inode_ctx
    before an fd based fop is sent, the fop is sent to
    the dst subvol on which the fd is not opened.
    This causes the FOP to fail with EBADF.

    Now, every fd based fop will check to see that the fd
    has been opened on the dst subvol before winding it down.

    Change-Id: Id92ef5eb7a5b5226688e2d2868b15e383f5f240e
    BUG: 1465075
    Signed-off-by: N Balachandran <nbalacha at redhat.com>
    Reviewed-on: https://review.gluster.org/17630
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
    Reviewed-by: Susant Palai <spalai at redhat.com>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1463907
[Bug 1463907] Application VMs, wth the disk images on replica 3 volume,
paused post rebalance
https://bugzilla.redhat.com/show_bug.cgi?id=1465075
[Bug 1465075] Fd based fops fail with EBADF on file migration
https://bugzilla.redhat.com/show_bug.cgi?id=1465123
[Bug 1465123] Fd based fops fail with EBADF on file migration
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.