[Bugs] [Bug 1336798] New: Unexporting a volume sometimes fails with " Dynamic export addition/deletion failed".

bugzilla at redhat.com bugzilla at redhat.com
Tue May 17 13:14:04 UTC 2016 

https://bugzilla.redhat.com/show_bug.cgi?id=1336798

            Bug ID: 1336798
           Summary: Unexporting a volume sometimes fails with "Dynamic
                    export addition/deletion failed".
           Product: GlusterFS
           Version: 3.8.0
         Component: ganesha-nfs
          Keywords: Triaged
          Severity: high
          Assignee: jthottan at redhat.com
          Reporter: jthottan at redhat.com
                CC: bugs at gluster.org, jthottan at redhat.com,
                    kkeithle at redhat.com, ndevos at redhat.com,
                    nlevinki at redhat.com, skoduri at redhat.com,
                    sraj at redhat.com, storage-qa-internal at redhat.com
        Depends On: 1330365, 1333319
            Blocks: 1333528, 1336331



+++ This bug was initially created as a clone of Bug #1333319 +++

+++ This bug was initially created as a clone of Bug #1330365 +++

Description of problem:

Unexporting a volume sometimes fails with "Dynamic export addition/deletion
failed".

Version-Release number of selected component (if applicable):

nfs-ganesha-2.3.1-4

How reproducible:

Intermittent

Steps to Reproduce:

Not sure about the exact steps but sometimes the unexport of volume fails with
"dynamic export addition/deletion failed"

Performed below steps when i hit this issue.

1.Created a 4 node ganesha cluster.
2.Created a tiered volume, enabled quota and mounted on 2 clients using vers=4
3.Performed IO's from both the mount points.
4.Umounted the volume from both clients.
5.Unexported the volume and it fails with "dynamic export addition/deletion
fails" with below messages in logs

[2016-04-25 16:20:35.129472] E [MSGID: 106472]
[glusterd-ganesha.c:145:glusterd_check_ganesha_cmd] 0-management: Handling
NFS-Ganesha op failed.
[2016-04-25 16:20:35.129582] E [MSGID: 106123]
[glusterd-syncop.c:1411:gd_commit_op_phase] 0-management: Commit of operation
'Volume Set' failed on localhost : Dynamic export addition/deletion failed.
Please see log file for details


It unexports the volume from the node where ganesha.enable off, was run but
from all other nodes, the volume doesnt get unexported 

[root at dhcp37-180 exports]# pwd
/etc/ganesha/exports
[root at dhcp37-180 exports]# ls
[root at dhcp37-180 exports]#

[root at dhcp37-158 exports]# pwd
/etc/ganesha/exports
[root at dhcp37-158 exports]# ls
export.tiervolume.conf
[root at dhcp37-158 exports]#

[root at dhcp37-127 exports]# pwd
/etc/ganesha/exports
[root at dhcp37-127 exports]# ls
export.tiervolume.conf
[root at dhcp37-127 exports]#

[root at dhcp37-174 exports]# pwd
/etc/ganesha/exports
[root at dhcp37-174 exports]# ls
export.tiervolume.conf
[root at dhcp37-174 exports]#

whereas showmount shows volume exported on all the nodes

[root at dhcp37-180 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

[root at dhcp37-158 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

[root at dhcp37-127 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

[root at dhcp37-174 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)

peer status is proper in this case:

[root at dhcp37-180 exports]# gluster peer status
Number of Peers: 3

Hostname: dhcp37-158.lab.eng.blr.redhat.com
Uuid: 18fa3cca-c714-4c70-b227-cef260fffa27
State: Peer in Cluster (Connected)

Hostname: dhcp37-127.lab.eng.blr.redhat.com
Uuid: 43649367-7f47-41cf-8d63-97896e3504d4
State: Peer in Cluster (Connected)

Hostname: dhcp37-174.lab.eng.blr.redhat.com
Uuid: 1a5a806a-ab58-462b-b939-0b8158a2d914
State: Peer in Cluster (Connected)

After restarting glusterd on all the nodes, showmount doesn't list the volume
as exported on localhost:

[root at dhcp37-180 exports]# showmount -e localhost
Export list for localhost:
[root at dhcp37-180 exports]# ls
[root at dhcp37-180 exports]# 

[root at dhcp37-158 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)
[root at dhcp37-158 exports]# ls
export.tiervolume.conf
[root at dhcp37-158 exports]#

[root at dhcp37-127 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)
[root at dhcp37-127 exports]# ls
export.tiervolume.conf
[root at dhcp37-127 exports]#

[root at dhcp37-174 exports]# showmount -e localhost
Export list for localhost:
/tiervolume (everyone)
[root at dhcp37-174 exports]# ls
export.tiervolume.conf
[root at dhcp37-174 exports]#


Actual results:

Unexporting a volume sometimes fails with "Dynamic export addition/deletion
failed".

Expected results:

The volume should get unexported without any issues.

Additional info:

No AVC's/SElinux related issue seen in the logs

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-04-25
23:32:51 EDT ---

This bug is automatically being proposed for the current z-stream release of
Red Hat Gluster Storage 3 by setting the release flag 'rhgs‑3.1.z' to '?'. 

If this bug should be proposed for a different release, please manually change
the proposed release flag.

--- Additional comment from Kaleb KEITHLEY on 2016-04-28 08:52:14 EDT ---

hard to reproduce. Document as known issue for 3.1.3 while we RCA

--- Additional comment from Shashank Raj on 2016-04-29 03:42:05 EDT ---

This issue seems to be hitting most of the time when we unexport the volume
after doing some IO's. However still not sure about of the exact steps:

[root at dhcp37-180 ~]# gluster volume set newvolume ganesha.enable off
volume set: failed: Dynamic export addition/deletion failed. Please see log
file for details

the volume gets unexported from the node from where we run the command and on
the other nodes it remains as it is.

Since, we are hitting this very frequently now, setting the severity to high

--- Additional comment from Vijay Bellur on 2016-05-05 05:35:37 EDT ---

REVIEW: http://review.gluster.org/14225 (NFS-Ganesha : Parse the Export_Id
correctly for unexporting volume) posted (#1) for review on master by jiffin
tony Thottan (jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-05-05 08:02:29 EDT ---

REVIEW: http://review.gluster.org/14225 (NFS-Ganesha : Parse the Export_Id
correctly for unexporting volume) posted (#2) for review on master by jiffin
tony Thottan (jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-05-05 14:13:49 EDT ---

COMMIT: http://review.gluster.org/14225 committed in master by Kaleb KEITHLEY
(kkeithle at redhat.com) 
------
commit 5440bad91026f27399df51856176aef85c1fcf4d
Author: Jiffin Tony Thottan <jthottan at redhat.com>
Date:   Thu May 5 14:52:32 2016 +0530

    NFS-Ganesha : Parse the Export_Id correctly for unexporting volume

    Currently export id parsed using "cut -d ' ' -f8" which might endup
    in giving wrong value. In case of multiple space chracter, output
    may differ. In this all those instance will replaced by awk call

    Change-Id: I60dea8ce116900da3c1fc9badf898e51183a2ca1
    BUG: 1333319
    Signed-off-by: Jiffin Tony Thottan <jthottan at redhat.com>
    Reviewed-on: http://review.gluster.org/14225
    Smoke: Gluster Build System <jenkins at build.gluster.com>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: soumya k <skoduri at redhat.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle at redhat.com>

--- Additional comment from Vijay Bellur on 2016-05-13 02:50:42 EDT ---

REVIEW: http://review.gluster.org/14325 (ganesha/scripts : Fixing refresh
config in ganesha-ha.sh) posted (#2) for review on master by jiffin tony
Thottan (jthottan at redhat.com)

--- Additional comment from Vijay Bellur on 2016-05-13 08:01:45 EDT ---

COMMIT: http://review.gluster.org/14325 committed in master by Niels de Vos
(ndevos at redhat.com) 
------
commit 8299ad374b2b93f123b29e0d411930733a372a04
Author: Jiffin Tony Thottan <jthottan at redhat.com>
Date:   Fri May 13 11:59:55 2016 +0530

    ganesha/scripts : Fixing refresh config in ganesha-ha.sh

    The change http://review.gluster.org/#/c/14225/ cause a regression
    for refresh config funtion in ganesha-ha.sh due to a invalid usage
    of awk arguement.

    Change-Id: Id5adfb12f99b29bdb3531773cd34bd67cfff8768
    BUG: 1333319
    Signed-off-by: Jiffin Tony Thottan <jthottan at redhat.com>
    Reviewed-on: http://review.gluster.org/14325
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: Kaleb KEITHLEY <kkeithle at redhat.com>
    Smoke: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: Niels de Vos <ndevos at redhat.com>


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1330365
[Bug 1330365] Unexporting a volume sometimes fails with "Dynamic export
addition/deletion failed".
https://bugzilla.redhat.com/show_bug.cgi?id=1333319
[Bug 1333319] Unexporting a volume sometimes fails with "Dynamic export
addition/deletion failed".
https://bugzilla.redhat.com/show_bug.cgi?id=1333528
[Bug 1333528] Unexporting a volume sometimes fails with "Dynamic export
addition/deletion failed".
https://bugzilla.redhat.com/show_bug.cgi?id=1336331
[Bug 1336331] Unexporting a volume sometimes fails with "Dynamic export
addition/deletion failed".
-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=CBiJERzCTD&a=cc_unsubscribe

More information about the Bugs mailing list