[Gluster-users] [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Meghana Madhusudhan
mmadhusu at redhat.com
Fri Jun 19 07:28:22 UTC 2015
I see that NFS-Ganesha is no longer running after the unexport. Can you move to the latest sources which has Soumya's fix and try once?
Even if the config file is still included in the ganesha.conf, dynamically removing that export should not affect the process itself.
Meghana
----- Original Message -----
From: "Alessandro De Salvo" <Alessandro.DeSalvo at roma1.infn.it>
To: "Meghana Madhusudhan" <mmadhusu at redhat.com>
Cc: gluster-users at gluster.org, nfs-ganesha-devel at lists.sourceforge.net, "Soumya Koduri" <skoduri at redhat.com>
Sent: Thursday, June 18, 2015 7:55:30 PM
Subject: Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Hi Meghana,
> Il giorno 18/giu/2015, alle ore 16:06, Meghana Madhusudhan <mmadhusu at redhat.com> ha scritto:
>
> Hi Allesandro,
>
> I need the following output from you,
>
> 1. After you execute ganesha.enable on command.
> ps aux | grep ganesha
root 6699 94.0 1.1 953488 193272 ? Ssl Jun17 1137:36 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -p /var/run/ganesha.nfsd.pid
root 22263 0.0 0.0 112644 960 pts/0 S+ 16:14 0:00 grep --color=auto ganesha
(both nodes of the cluster show the same, apart from the PIDs, of course)
> showmount -e localhost
Export list for node1:
/atlas-backup-01 (everyone)
The same output is shown on the other node.
> cat /etc/ganesha/ganesha.conf
# Ganesha options file
%include "/etc/ganesha/ganesha-opts.conf"
%include "/etc/ganesha/exports/export.atlas-backup-01.conf"
The first include just point to a different file with the following lines, to change the rquota port
NFS_Core_Param {
#Use a non-privileged port for RQuota
Rquota_Port = 4501;
}
> cat "/etc/ganesha/exports/export.VOLNAME.conf" or "/usr/..../export.VOLNAME.conf”
# WARNING : Using Gluster CLI will overwrite manual
# changes made to this file. To avoid it, edit the
# file, copy it over to all the NFS-Ganesha nodes
# and run ganesha-ha.sh --refresh-config.
EXPORT{
Export_Id= 2 ;
Path = "/atlas-backup-01";
FSAL {
name = GLUSTER;
hostname="localhost";
volume="atlas-backup-01";
}
Access_type = RW;
Disable_ACL = true;
Squash="No_root_squash";
Pseudo="/atlas-backup-01";
Protocols = "3", "4" ;
Transports = "UDP","TCP";
SecType = "sys";
}
> tail /var/log/ganesha.log or wherever the ganesha log is.
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
17/06/2015 20:02:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
18/06/2015 16:13:50 : epoch 5581b5da : node2 : ganesha.nfsd-26983[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/'
>
> 2. gluster vol set VOLNAME ganesha.enable off
>
> Output of the same as above.
# gluster vol set atlas-backup-01 ganesha.enable off
volume set: failed: Dynamic export addition/deletion failed. Please see log file for details
# ps aux | grep ganesha
root 32033 0.0 0.0 112640 960 pts/0 S+ 16:20 0:00 grep --color=auto ganesha
[This is due to the fact that the export file was deleted by the config still references it, so ganesha fails to resume]
# cat /etc/ganesha/ganesha.conf
# Ganesha options file
%include "/etc/ganesha/ganesha-opts.conf"
%include "/etc/ganesha/exports/export.atlas-backup-01.conf"
# cat /etc/ganesha/exports/export.atlas-backup-01.conf
cat: /etc/ganesha/exports/export.atlas-backup-01.conf: No such file or directory
# tail /var/log/ganesha.log
18/06/2015 16:19:20 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/'
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 60
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (2:2)
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :9P/TCP dispatcher thread was started successfully
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[_9p_disp] _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
18/06/2015 16:20:22 : epoch 5582d3a6 : node2 : ganesha.nfsd-31542[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31550[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31557[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31562[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31567[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31908[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31932[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31949[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31956[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31971[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
I have not enabled the FULL_DEBUG but if you need it I can do it.
>
> Thanks for your perseverance :)
I have to thank you for the help! :-)
Cheers,
Alessandro
>
> Meghana
>
> ----- Original Message -----
> From: "Alessandro De Salvo" <Alessandro.DeSalvo at roma1.infn.it>
> To: "Meghana Madhusudhan" <mmadhusu at redhat.com>
> Cc: gluster-users at gluster.org, nfs-ganesha-devel at lists.sourceforge.net, "Soumya Koduri" <skoduri at redhat.com>
> Sent: Thursday, June 18, 2015 7:24:55 PM
> Subject: Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
>
> Hi Meghana,
>
>> Il giorno 18/giu/2015, alle ore 07:04, Meghana Madhusudhan <mmadhusu at redhat.com> ha scritto:
>>
>>
>>
>>
>> On 06/17/2015 10:57 PM, Alessandro De Salvo wrote:
>>> Hi,
>>> when disabling exports from gluster 3.7.1, by using gluster vol set <volume> ganesha.enable off, I always get the following error:
>>>
>>> Error: Dynamic export addition/deletion failed. Please see log file for details
>>>
>>> This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see:
>> you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by
>> Gluster-CLI to unexport the volume "gluster volume set <volname>
>> ganesha.enable off" which rightly deletes the export file too while
>> un-exporting the volume.
>>
>>>
>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2
>>> Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus)
>>>
>>> So, there is a timeout and it fails completely.
>> Check if nfs-ganesha is still running. There was a bug in unexporting
>> the volume. Its been fixed recently in V2.3-dev, yet to be back-ported
>> to V2.2-stable branch.
>> https://review.gerrithub.io/#/c/236129/
>>
>> Thanks,
>> Soumya
>>
>>> In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration.
>>
>> Yes. I see that the "sed -i /$VOL.conf/d $CONF" is placed after the check_cmd_status. I shall send a fix upstream in a related bug. But dynamic export removal
>> will fail in three cases,
>> 1. nfs-ganesha is not running.
>
> no, it was running
>
>> 2. The export file that is particular to that volume is somehow deleted before you perform the removal. It does depend on that file to get the export ID.
>
> I tried to comment out the rm in check_cmd_status to avoid this race condition, but it did not solve the problems.
>
>> 3. The bug that Soumya pointed out.
>
> This might well be the real cause!
>
>>
>> If it is failing consistently, there could be something that you are missing. If you can send the exact sequence of sequence of steps that you have executed,
>> I can help you with it.
>
> Yes, it’s failing consistently, unless as I said I do a DisplayExport before the RemoveExport, in which case it always works.
>
>>
>> Ideally after exporting a particular volume, you'll see an entry in the /etc/ganesha/ganesha.conf file and the export file in "/etc/ganesha/exports" dir.
>
> And this works perfectly, I see them correctly.
>
>> If you have this in place and nfs-ganesha running, then dynamic export removal should work just fine.
>
> But this is not, at least in my case.
> The command I’m using are just the following:
>
> gluster vol set <volume> ganesha.enable on
> gluster vol set <volume> ganesha.enable off
>
> It normally wait a few seconds between the two commands, to give time to ganesha to actually export the volume.
> The export is always failing as described, unless I add the DisplayExport in dbus-send.sh before RemoveExport.
> Many thanks for the help,
>
> Alessandro
>
>
>>
>> Meghana
>>
>>
>>>
>>> What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works:
>>>
>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2
>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2
>>>
>>> So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why?
>>> I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3.
>>> Thanks,
>>>
>>> Alessandro
>>>
>>> PS: sorry for reporting so many things in a few days :-)
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>>
>>>
>>>
>>> _______________________________________________
>>> Nfs-ganesha-devel mailing list
>>> Nfs-ganesha-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Nfs-ganesha-devel mailing list
>> Nfs-ganesha-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>>
>
>
More information about the Gluster-users
mailing list