[Bugs] [Bug 1479303] New: [Perf] : Large file sequential reads are off target by ~38% on FUSE/Ganesha

Tue Aug 8 10:38:02 UTC 2017

https://bugzilla.redhat.com/show_bug.cgi?id=1479303

            Bug ID: 1479303
           Summary: [Perf] : Large file sequential reads are off target by
                    ~38% on FUSE/Ganesha
           Product: GlusterFS
           Version: 3.12
         Component: distribute
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: nbalacha at redhat.com
                CC: amukherj at redhat.com, asoman at redhat.com,
                    bturner at redhat.com, bugs at gluster.org, dang at redhat.com,
                    jthottan at redhat.com, kkeithle at redhat.com,
                    mbenjamin at redhat.com, nbalacha at redhat.com,
                    rcyriac at redhat.com, rhinduja at redhat.com,
                    rhs-bugs at redhat.com, sanandpa at redhat.com,
                    skoduri at redhat.com, storage-qa-internal at redhat.com
        Depends On: 1475136, 1476665

+++ This bug was initially created as a clone of Bug #1476665 +++

+++ This bug was initially created as a clone of Bug #1475136 +++

Description of problem:
-----------------------

A regression seems to have been introduced in recent bits on large file seq
reads :

3.2 : 2480044.05 kB/sec    
3.8.4-35 : 1538178.2 kB/sec

Regression : ~38%

This is on a vanilla volume,without PR,NL,mdcache.

Version-Release number of selected component (if applicable):
-------------------------------------------------------------

3.8.4-35

How reproducible:
-----------------

100%

Additional info:
----------------
Volume Name: testvol
Type: Distributed-Replicate
Volume ID: 4b52bfb8-28fd-4e0f-8ee0-eb8116a296c4
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: server1:/bricks/testvol_brick0
Brick2: server2:/bricks/testvol_brick1
Brick3: server3:/bricks/testvol_brick2
Brick4: server4:/bricks/testvol_brick3
Options Reconfigured:
server.allow-insecure: on
performance.stat-prefetch: off
transport.address-family: inet
nfs.disable: on

--- Additional comment from Nithya Balachandran on 2017-07-31 02:58:32 EDT ---

The changes made in https://review.gluster.org/#/c/17630/ have introduced a
perf regression. 

The code has been changed to check if the fd is open on the dst subvol only if
the FOP returns with EBADFD.

--- Additional comment from Worker Ant on 2017-07-31 03:06:04 EDT ---

REVIEW: https://review.gluster.org/17922 (cluster/dht: Check for open fd only
on EBADF) posted (#1) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-07-31 04:10:16 EDT ---

REVIEW: https://review.gluster.org/17922 (cluster/dht: Check for open fd only
on EBADF) posted (#2) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-07-31 10:00:25 EDT ---

COMMIT: https://review.gluster.org/17922 committed in master by Raghavendra G
(rgowdapp at redhat.com) 
------
commit 91c9f4a19fde4894576b398252c77f730832a26a
Author: N Balachandran <nbalacha at redhat.com>
Date:   Mon Jul 31 12:32:59 2017 +0530

    cluster/dht: Check for open fd only on EBADF

    DHT fd based fops will now check if the fd is
    open on the cached subvol only if the call fails
    with EBADF.

    This will improve performance for scenarios where
    a rebalance is not running which would be most of
    the time.

    Change-Id: Idfaeb8927af769c6110d07a165a0fe2307369239
    BUG: 1476665
    Signed-off-by: N Balachandran <nbalacha at redhat.com>
    Reviewed-on: https://review.gluster.org/17922
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Raghavendra G <rgowdapp at redhat.com>

--- Additional comment from Worker Ant on 2017-08-03 23:26:26 EDT ---

REVIEW: https://review.gluster.org/17970 (Revert "cluster/dht: Check for open
fd only on EBADF") posted (#1) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-08-04 04:56:25 EDT ---

COMMIT: https://review.gluster.org/17970 committed in master by N Balachandran
(nbalacha at redhat.com) 
------
commit 6d1068ddb35be19df36210c9fcaa7ce97e2a376a
Author: N Balachandran <nbalacha at redhat.com>
Date:   Fri Aug 4 08:52:31 2017 +0530

    Revert "cluster/dht: Check for open fd only on EBADF"

    This reverts commit 91c9f4a19fde4894576b398252c77f730832a26a.
    This patch needs to be reworked.

    Change-Id: I4c24f647c2b1abc68fc4e9fe6eb810418e2033aa
    BUG: 1476665
    Signed-off-by: N Balachandran <nbalacha at redhat.com>
    Reviewed-on: https://review.gluster.org/17970
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>

--- Additional comment from Worker Ant on 2017-08-04 05:21:40 EDT ---

REVIEW: https://review.gluster.org/17976 (cluster/dht: Check for open fd only
on EBADF) posted (#1) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-08-04 05:35:31 EDT ---

REVIEW: https://review.gluster.org/17976 (cluster/dht: Check for open fd only
on EBADF) posted (#2) for review on master by N Balachandran
(nbalacha at redhat.com)

--- Additional comment from Worker Ant on 2017-08-08 06:21:28 EDT ---

COMMIT: https://review.gluster.org/17976 committed in master by Raghavendra G
(rgowdapp at redhat.com) 
------
commit cdca1cb26a0aba390c6d8485c0d6d95e22ffc8bd
Author: N Balachandran <nbalacha at redhat.com>
Date:   Fri Aug 4 14:46:38 2017 +0530

    cluster/dht: Check for open fd only on EBADF

    DHT fd based fops used to check if the fd was open
    on the cached subvol before winding the call. However,
    this introduced a performance regression of about
    30% for reads.

    This check was introduced to handle cases where files
    were migrated while IOs were happening. As this is not
    the common case, dht will now check if the fd is
    open on the cached subvol only if the call fails
    with EBADF.

    This will prevent a performance hit where a rebalance
    is not running.

    Change-Id: I2035a858d63c3fcd22bb634055bbb0ad01686808
    BUG: 1476665
    Signed-off-by: N Balachandran <nbalacha at redhat.com>
    Reviewed-on: https://review.gluster.org/17976
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Amar Tumballi <amarts at redhat.com>
    Reviewed-by: Susant Palai <spalai at redhat.com>
    Reviewed-by: Raghavendra G <rgowdapp at redhat.com>

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1475136
[Bug 1475136] [Perf] : Large file sequential reads are off target by ~38%
on FUSE/Ganesha
https://bugzilla.redhat.com/show_bug.cgi?id=1476665
[Bug 1476665] [Perf] : Large file sequential reads are off target by ~38%
on FUSE/Ganesha
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.