[Bugs] [Bug 1391851] New: removal of file from nfs mount crashs ganesha server

Fri Nov 4 09:03:56 UTC 2016

https://bugzilla.redhat.com/show_bug.cgi?id=1391851

            Bug ID: 1391851
           Summary: removal of file from nfs mount crashs ganesha server
           Product: GlusterFS
           Version: 3.7.15
         Component: upcall
          Keywords: Triaged
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: skoduri at redhat.com
                CC: bugs at gluster.org, jthottan at redhat.com,
                    ndevos at redhat.com, skoduri at redhat.com
        Depends On: 1344714, 1347715
            Blocks: 1372171 (glusterfs-3.7.16)

+++ This bug was initially created as a clone of Bug #1347715 +++

+++ This bug was initially created as a clone of Bug #1344714 +++

Description of problem:

The struct callback_inode_arg is a variable used by upcall infra for populating
upcall entries. This variable is allocated using GF_CALLOC in
glfs_h_poll_cache_invalidation(), but it also freedin FSAL_GLUSTER using free.
This will result in a crash when we try to remove a file from mount

Version-Release number of selected component (if applicable):
mainline

How reproducible:

Steps to Reproduce:
1. enable nfs-ganesha and export a volume 
2. mount the volume
3. create a file and delete it from the mount

Actual results:
ganesha server crashes

Expected results:
should not crash

Additional info:

--- Additional comment from Vijay Bellur on 2016-06-10 09:17:53 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing glfs_free()
api) posted (#1) for review on master by jiffin tony Thottan
(jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-10 09:17:56 EDT ---

REVIEW: http://review.gluster.org/14702 (libgfapi/upcall : appending glfs to
callback_arg, callback_inode_arg) posted (#1) for review on master by jiffin
tony Thottan (jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-10 16:57:16 EDT ---

REVIEW: http://review.gluster.org/14702 (libgfapi/upcall : prepend "glfs_" to
callback_arg, callback_inode_arg) posted (#2) for review on master by Jeff
Darcy (jdarcy at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-10 17:09:37 EDT ---

COMMIT: http://review.gluster.org/14702 committed in master by Jeff Darcy
(jdarcy at redhat.com) 
------
commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6
Author: Jiffin Tony Thottan <jthottan at redhat.com>
Date:   Fri Jun 10 18:23:43 2016 +0530

    libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg

    Change-Id: I371525775db4f6a4d69beb94baaa53d17b16fb41
    BUG: 1344714
    Signed-off-by: Jiffin Tony Thottan <jthottan at redhat.com>
    Reviewed-on: http://review.gluster.org/14702
    CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    Reviewed-by: Jeff Darcy <jdarcy at redhat.com>
    Tested-by: Jeff Darcy <jdarcy at redhat.com>
    Smoke: Gluster Build System <jenkins at build.gluster.com>

--- Additional comment from Vijay Bellur on 2016-06-14 04:58:05 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing
glfs_free_callback_inode_arg() api) posted (#2) for review on master by jiffin
tony Thottan (jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-14 06:14:16 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing
glfs_free_callback_inode_arg() api) posted (#3) for review on master by jiffin
tony Thottan (jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-16 09:28:10 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing
glfs_free_callback_inode_arg() api) posted (#4) for review on master by jiffin
tony Thottan (jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-16 09:30:18 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : Introducing
glfs_free_callback_inode_arg() api) posted (#5) for review on master by jiffin
tony Thottan (jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-16 09:41:28 EDT ---

REVIEW: http://review.gluster.org/14701 (libgfapi : free
glfs_callback_inode_arg from application layer) posted (#6) for review on
master by jiffin tony Thottan (jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-17 08:46:07 EDT ---

REVIEW: http://review.gluster.org/14759 (libgfapi/upcall : use calloc and free
for callback_inode_arg) posted (#1) for review on master by jiffin tony Thottan
(jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-06-17 08:55:22 EDT ---

REVIEW: http://review.gluster.org/14760 (libgfapi/upcall : use calloc and free
for callback_inode_arg) posted (#1) for review on release-3.7 by jiffin tony
Thottan (jthottan at redhat.com)

--- Additional comment from Worker Ant on 2016-09-30 06:48:37 EDT ---

REVIEW: http://review.gluster.org/15600 (Upcall/cache-invalidation: Use parent
stbuf while updating parent entry) posted (#1) for review on release-3.7 by
Niels de Vos (ndevos at redhat.com)

--- Additional comment from Worker Ant on 2016-09-30 06:48:40 EDT ---

REVIEW: http://review.gluster.org/15601 (libglusterfs: add gf_get_mem_type())
posted (#1) for review on release-3.7 by Niels de Vos (ndevos at redhat.com)

--- Additional comment from Worker Ant on 2016-09-30 06:48:43 EDT ---

REVIEW: http://review.gluster.org/15602 (gfapi: redesign the public interface
for upcall consumers) posted (#1) for review on release-3.7 by Niels de Vos
(ndevos at redhat.com)

--- Additional comment from Worker Ant on 2016-09-30 08:11:17 EDT ---

REVIEW: http://review.gluster.org/15601 (libglusterfs: add gf_get_mem_type())
posted (#2) for review on release-3.7 by Niels de Vos (ndevos at redhat.com)

--- Additional comment from Worker Ant on 2016-09-30 08:12:08 EDT ---

REVIEW: http://review.gluster.org/15602 (gfapi: redesign the public interface
for upcall consumers) posted (#2) for review on release-3.7 by Niels de Vos
(ndevos at redhat.com)

--- Additional comment from Worker Ant on 2016-10-03 08:15:30 EDT ---

COMMIT: http://review.gluster.org/15600 committed in release-3.7 by Kaushal M
(kaushal at redhat.com) 
------
commit fee92f7c6abc651e88a8b52d5187286a4b8df2c3
Author: Soumya Koduri <skoduri at redhat.com>
Date:   Mon Dec 14 17:52:37 2015 +0530

    Upcall/cache-invalidation: Use parent stbuf while updating parent entry

    For *create*  fops (CREATE, MKDIR, MKNOD), we invalidate the parent
    entry. Hence send parent attributes in the stat field.

    Also "UP_PARENT_DENTRY_FLAGS" has to be set only for the fops which shall
    result in two invalidations requests - one for the inode on which fop is
    being performed and another on parent entry.

    In case of CREATE/MKDIR/MKNOD fops, there shall be only one invalidation
    request sent, that too on parent inode. We send invalidation directly on
    parent inode's gfid. So there is no necessity to set these flags which
    when set shall endup invalidating the parent's parent entry.

    Cherry picked from commit f4282bd927e2e0d826d62cf1192102382c5697b2:
    > Change-Id: I7514ee08382081e3e060818ede497dbca26987dc
    > BUG: 1291259
    > Signed-off-by: Soumya Koduri <skoduri at redhat.com>
    > Reviewed-on: http://review.gluster.org/12962
    > CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
    > NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    > Smoke: Gluster Build System <jenkins at build.gluster.com>
    > Reviewed-by: Niels de Vos <ndevos at redhat.com>

    Change-Id: I7514ee08382081e3e060818ede497dbca26987dc
    BUG: 1347715
    Signed-off-by: Niels de Vos <ndevos at redhat.com>
    Reviewed-on: http://review.gluster.org/15600
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Kaushal M <kaushal at redhat.com>

--- Additional comment from Worker Ant on 2016-10-03 08:16:48 EDT ---

COMMIT: http://review.gluster.org/15601 committed in release-3.7 by Kaushal M
(kaushal at redhat.com) 
------
commit 2cee4a3e47518aeb28ac2b611c6f01c0f9d00dab
Author: Niels de Vos <ndevos at redhat.com>
Date:   Sat Aug 6 16:04:29 2016 +0200

    libglusterfs: add gf_get_mem_type()

    gfapi needs to provide a function towards applications to free memory
    that it allocated. Depending on how the application is compiled/linked,
    it could use a different memory allocator than Gluster itself. Therefore
    it is not safe for gfapi to request applications to free memory with
    'standard' free().

    Examples for this are Gluster allocated structures with GF_CALLOC() when
    memory accounting is enabled (the default). Some gfapi functions use
    malloc() to allocate memory as a workaround, but the free() from the
    jemalloc implementation should not be combined with the malloc() from
    glibc.

    Cherry picked from commit db4e26ed71a01e5f760fbc3c7051962426f102c9:
    > Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093
    > BUG: 1344714
    > Signed-off-by: Niels de Vos <ndevos at redhat.com>
    > Reviewed-on: http://review.gluster.org/15108
    > Smoke: Gluster Build System <jenkins at build.gluster.org>
    > Reviewed-by: soumya k <skoduri at redhat.com>
    > Reviewed-by: jiffin tony Thottan <jthottan at redhat.com>
    > NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    > CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    > Reviewed-by: Shyamsundar Ranganathan <srangana at redhat.com>

    Change-Id: I626cd1a60abf8965f9263290f4045d1f69fc2093
    BUG: 1347715
    Signed-off-by: Niels de Vos <ndevos at redhat.com>
    Reviewed-on: http://review.gluster.org/15601
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Kaushal M <kaushal at redhat.com>

--- Additional comment from Worker Ant on 2016-10-03 08:17:24 EDT ---

REVIEW: http://review.gluster.org/15602 (gfapi: redesign the public interface
for upcall consumers) posted (#3) for review on release-3.7 by Kaushal M
(kaushal at redhat.com)

--- Additional comment from Worker Ant on 2016-10-03 08:20:47 EDT ---

COMMIT: http://review.gluster.org/15602 committed in release-3.7 by Kaushal M
(kaushal at redhat.com) 
------
commit 55c92db32ba7d88359f0562953a3a6d8874dd1a5
Author: Niels de Vos <ndevos at redhat.com>
Date:   Fri Jun 10 18:23:43 2016 +0530

    gfapi: redesign the public interface for upcall consumers

    The glfs_callback_arg and glfs_callback_inode_arg were allocated by
    gfapi, and expected to be free()'d by the application. However it is not
    reasonable to expect that applications use the same memory allocator to
    as the compiled libgfapi.so. For instance, it is possible that gfapi
    uses glibc malloc/free, and an application like NFS-Ganesha the versions
    from jemalloc. Mismatching of the malloc() and free() functions causes
    segmentation faults at best.

    In order to prevent problems like this in the future, the API for
    applications that consume upcalls has been remodeled. Any of the
    structures that gfapi allocates, should be free'd with glfs_free(). The
    members of the structures can not be accessed directly anymore, each
    has its own function to access now.

    Correcting the naming of the functions, structures and constants is a
    continuation of commit 2775dc64101ed37c8d9809bf9852dbf0746ee2b6. These
    new improvements not only have correct prefixes for the functions and
    structures, the naming also reflects more to the upcall framework and
    does not use "callback" anymore.

    Cherry picked from commit 4721188a154acd9a0a4c096d8d73e97f3bf1b2a9:
    > Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
    > BUG: 1344714
    > Signed-off-by: Niels de Vos <ndevos at redhat.com>
    > Reviewed-on: http://review.gluster.org/14701
    > Smoke: Gluster Build System <jenkins at build.gluster.org>
    > NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    > CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    > Reviewed-by: Kaleb KEITHLEY <kkeithle at redhat.com>
    > Reviewed-by: soumya k <skoduri at redhat.com>
    > Reviewed-by: jiffin tony Thottan <jthottan at redhat.com>

    Once difference with the version of this change in other branches is
    that leases are not included in glusterfs-3.7. Hence there is a little
    change that drops the handling of GF_UPCALL_RECALL_LEASE.

    In addition, this backport contains commit 2775dc6410:
    > libgfapi/upcall : prepend "glfs_" to callback_arg, callback_inode_arg
    > Reviewed-on: http://review.gluster.org/14702

    Change-Id: I2b8bd5a0a82036d2abea1a217f5e5975a1d4fe93
    BUG: 1347715
    Signed-off-by: Niels de Vos <ndevos at redhat.com>
    Reviewed-on: http://review.gluster.org/15602
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Kaushal M <kaushal at redhat.com>

--- Additional comment from Kaushal on 2016-10-17 05:04:48 EDT ---

This bug is getting closed because a release has been made available that
should address the reported issue. In case the problem is still not fixed with
glusterfs-3.7.16, please open a new bug report.

glusterfs-3.7.16 has been announced on the Gluster mailinglists [1], packages
for several distributions should become available in the near future. Keep an
eye on the Gluster Users mailinglist [2] and the update infrastructure for your
distribution.

[1] https://www.gluster.org/pipermail/gluster-devel/2016-October/051187.html
[2] https://www.gluster.org/pipermail/gluster-users/

Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1344714
[Bug 1344714] removal of file from nfs mount crashs ganesha server
https://bugzilla.redhat.com/show_bug.cgi?id=1347715
[Bug 1347715] removal of file from nfs mount crashs ganesha server
https://bugzilla.redhat.com/show_bug.cgi?id=1372171
[Bug 1372171] Tracker bug for GlusterFS-v3.7.16
-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.