[Gluster-users] [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Alessandro De Salvo
Alessandro.DeSalvo at roma1.infn.it
Thu Jun 18 14:25:30 UTC 2015
Hi Meghana,
> Il giorno 18/giu/2015, alle ore 16:06, Meghana Madhusudhan <mmadhusu at redhat.com> ha scritto:
>
> Hi Allesandro,
>
> I need the following output from you,
>
> 1. After you execute ganesha.enable on command.
> ps aux | grep ganesha
root 6699 94.0 1.1 953488 193272 ? Ssl Jun17 1137:36 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -p /var/run/ganesha.nfsd.pid
root 22263 0.0 0.0 112644 960 pts/0 S+ 16:14 0:00 grep --color=auto ganesha
(both nodes of the cluster show the same, apart from the PIDs, of course)
> showmount -e localhost
Export list for node1:
/atlas-backup-01 (everyone)
The same output is shown on the other node.
> cat /etc/ganesha/ganesha.conf
# Ganesha options file
%include "/etc/ganesha/ganesha-opts.conf"
%include "/etc/ganesha/exports/export.atlas-backup-01.conf"
The first include just point to a different file with the following lines, to change the rquota port
NFS_Core_Param {
#Use a non-privileged port for RQuota
Rquota_Port = 4501;
}
> cat "/etc/ganesha/exports/export.VOLNAME.conf" or "/usr/..../export.VOLNAME.conf”
# WARNING : Using Gluster CLI will overwrite manual
# changes made to this file. To avoid it, edit the
# file, copy it over to all the NFS-Ganesha nodes
# and run ganesha-ha.sh --refresh-config.
EXPORT{
Export_Id= 2 ;
Path = "/atlas-backup-01";
FSAL {
name = GLUSTER;
hostname="localhost";
volume="atlas-backup-01";
}
Access_type = RW;
Disable_ACL = true;
Squash="No_root_squash";
Pseudo="/atlas-backup-01";
Protocols = "3", "4" ;
Transports = "UDP","TCP";
SecType = "sys";
}
> tail /var/log/ganesha.log or wherever the ganesha log is.
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
17/06/2015 20:02:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
18/06/2015 16:13:50 : epoch 5581b5da : node2 : ganesha.nfsd-26983[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/'
>
> 2. gluster vol set VOLNAME ganesha.enable off
>
> Output of the same as above.
# gluster vol set atlas-backup-01 ganesha.enable off
volume set: failed: Dynamic export addition/deletion failed. Please see log file for details
# ps aux | grep ganesha
root 32033 0.0 0.0 112640 960 pts/0 S+ 16:20 0:00 grep --color=auto ganesha
[This is due to the fact that the export file was deleted by the config still references it, so ganesha fails to resume]
# cat /etc/ganesha/ganesha.conf
# Ganesha options file
%include "/etc/ganesha/ganesha-opts.conf"
%include "/etc/ganesha/exports/export.atlas-backup-01.conf"
# cat /etc/ganesha/exports/export.atlas-backup-01.conf
cat: /etc/ganesha/exports/export.atlas-backup-01.conf: No such file or directory
# tail /var/log/ganesha.log
18/06/2015 16:19:20 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/'
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 60
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (2:2)
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :9P/TCP dispatcher thread was started successfully
18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[_9p_disp] _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
18/06/2015 16:20:22 : epoch 5582d3a6 : node2 : ganesha.nfsd-31542[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31550[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31557[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31562[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31567[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31908[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31932[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31949[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31956[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31971[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
I have not enabled the FULL_DEBUG but if you need it I can do it.
>
> Thanks for your perseverance :)
I have to thank you for the help! :-)
Cheers,
Alessandro
>
> Meghana
>
> ----- Original Message -----
> From: "Alessandro De Salvo" <Alessandro.DeSalvo at roma1.infn.it>
> To: "Meghana Madhusudhan" <mmadhusu at redhat.com>
> Cc: gluster-users at gluster.org, nfs-ganesha-devel at lists.sourceforge.net, "Soumya Koduri" <skoduri at redhat.com>
> Sent: Thursday, June 18, 2015 7:24:55 PM
> Subject: Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
>
> Hi Meghana,
>
>> Il giorno 18/giu/2015, alle ore 07:04, Meghana Madhusudhan <mmadhusu at redhat.com> ha scritto:
>>
>>
>>
>>
>> On 06/17/2015 10:57 PM, Alessandro De Salvo wrote:
>>> Hi,
>>> when disabling exports from gluster 3.7.1, by using gluster vol set <volume> ganesha.enable off, I always get the following error:
>>>
>>> Error: Dynamic export addition/deletion failed. Please see log file for details
>>>
>>> This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see:
>> you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by
>> Gluster-CLI to unexport the volume "gluster volume set <volname>
>> ganesha.enable off" which rightly deletes the export file too while
>> un-exporting the volume.
>>
>>>
>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2
>>> Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus)
>>>
>>> So, there is a timeout and it fails completely.
>> Check if nfs-ganesha is still running. There was a bug in unexporting
>> the volume. Its been fixed recently in V2.3-dev, yet to be back-ported
>> to V2.2-stable branch.
>> https://review.gerrithub.io/#/c/236129/
>>
>> Thanks,
>> Soumya
>>
>>> In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration.
>>
>> Yes. I see that the "sed -i /$VOL.conf/d $CONF" is placed after the check_cmd_status. I shall send a fix upstream in a related bug. But dynamic export removal
>> will fail in three cases,
>> 1. nfs-ganesha is not running.
>
> no, it was running
>
>> 2. The export file that is particular to that volume is somehow deleted before you perform the removal. It does depend on that file to get the export ID.
>
> I tried to comment out the rm in check_cmd_status to avoid this race condition, but it did not solve the problems.
>
>> 3. The bug that Soumya pointed out.
>
> This might well be the real cause!
>
>>
>> If it is failing consistently, there could be something that you are missing. If you can send the exact sequence of sequence of steps that you have executed,
>> I can help you with it.
>
> Yes, it’s failing consistently, unless as I said I do a DisplayExport before the RemoveExport, in which case it always works.
>
>>
>> Ideally after exporting a particular volume, you'll see an entry in the /etc/ganesha/ganesha.conf file and the export file in "/etc/ganesha/exports" dir.
>
> And this works perfectly, I see them correctly.
>
>> If you have this in place and nfs-ganesha running, then dynamic export removal should work just fine.
>
> But this is not, at least in my case.
> The command I’m using are just the following:
>
> gluster vol set <volume> ganesha.enable on
> gluster vol set <volume> ganesha.enable off
>
> It normally wait a few seconds between the two commands, to give time to ganesha to actually export the volume.
> The export is always failing as described, unless I add the DisplayExport in dbus-send.sh before RemoveExport.
> Many thanks for the help,
>
> Alessandro
>
>
>>
>> Meghana
>>
>>
>>>
>>> What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works:
>>>
>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2
>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2
>>>
>>> So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why?
>>> I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3.
>>> Thanks,
>>>
>>> Alessandro
>>>
>>> PS: sorry for reporting so many things in a few days :-)
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>>
>>>
>>>
>>> _______________________________________________
>>> Nfs-ganesha-devel mailing list
>>> Nfs-ganesha-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Nfs-ganesha-devel mailing list
>> Nfs-ganesha-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>>
>
>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1770 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150618/3de6ceb7/attachment.p7s>
More information about the Gluster-users
mailing list