[Gluster-users] [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
Soumya Koduri
skoduri at redhat.com
Thu Jun 18 16:07:33 UTC 2015
On 06/18/2015 07:55 PM, Alessandro De Salvo wrote:
> Hi Meghana,
>
>> Il giorno 18/giu/2015, alle ore 16:06, Meghana Madhusudhan <mmadhusu at redhat.com> ha scritto:
>>
>> Hi Allesandro,
>>
>> I need the following output from you,
>>
>> 1. After you execute ganesha.enable on command.
>> ps aux | grep ganesha
>
> root 6699 94.0 1.1 953488 193272 ? Ssl Jun17 1137:36 /usr/bin/ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT -p /var/run/ganesha.nfsd.pid
> root 22263 0.0 0.0 112644 960 pts/0 S+ 16:14 0:00 grep --color=auto ganesha
>
> (both nodes of the cluster show the same, apart from the PIDs, of course)
>
>> showmount -e localhost
>
> Export list for node1:
> /atlas-backup-01 (everyone)
>
> The same output is shown on the other node.
>
>> cat /etc/ganesha/ganesha.conf
>
> # Ganesha options file
> %include "/etc/ganesha/ganesha-opts.conf"
> %include "/etc/ganesha/exports/export.atlas-backup-01.conf"
>
>
> The first include just point to a different file with the following lines, to change the rquota port
>
> NFS_Core_Param {
> #Use a non-privileged port for RQuota
> Rquota_Port = 4501;
> }
>
>
>> cat "/etc/ganesha/exports/export.VOLNAME.conf" or "/usr/..../export.VOLNAME.conf”
>
> # WARNING : Using Gluster CLI will overwrite manual
> # changes made to this file. To avoid it, edit the
> # file, copy it over to all the NFS-Ganesha nodes
> # and run ganesha-ha.sh --refresh-config.
> EXPORT{
> Export_Id= 2 ;
> Path = "/atlas-backup-01";
> FSAL {
> name = GLUSTER;
> hostname="localhost";
> volume="atlas-backup-01";
> }
> Access_type = RW;
> Disable_ACL = true;
> Squash="No_root_squash";
> Pseudo="/atlas-backup-01";
> Protocols = "3", "4" ;
> Transports = "UDP","TCP";
> SecType = "sys";
> }
>
>
>> tail /var/log/ganesha.log or wherever the ganesha log is.
>
> 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
> 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
> 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
> 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
> 17/06/2015 20:01:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
> 17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
> 17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
> 17/06/2015 20:01:01 : epoch 5581b5da : node2 : ganesha.nfsd-26983[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
> 17/06/2015 20:02:00 : epoch 5581b5da : node2 : ganesha.nfsd-26983[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now NOT IN GRACE
> 18/06/2015 16:13:50 : epoch 5581b5da : node2 : ganesha.nfsd-26983[dbus_heartbeat] glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/'
>
>
>>
>> 2. gluster vol set VOLNAME ganesha.enable off
>>
>> Output of the same as above.
>
> # gluster vol set atlas-backup-01 ganesha.enable off
> volume set: failed: Dynamic export addition/deletion failed. Please see log file for details
>
> # ps aux | grep ganesha
> root 32033 0.0 0.0 112640 960 pts/0 S+ 16:20 0:00 grep --color=auto ganesha
>
> [This is due to the fact that the export file was deleted by the config still references it, so ganesha fails to resume]
>
That would mean NFS-Ganesha process has crashed due the bug which I had
mentioned earlier. The fix is merged into V2.3-dev branch and will get
back-ported to V2.2-stable branch this week.
> # cat /etc/ganesha/ganesha.conf
> # Ganesha options file
> %include "/etc/ganesha/ganesha-opts.conf"
> %include "/etc/ganesha/exports/export.atlas-backup-01.conf"
>
> # cat /etc/ganesha/exports/export.atlas-backup-01.conf
> cat: /etc/ganesha/exports/export.atlas-backup-01.conf: No such file or directory
>
>
> # tail /var/log/ganesha.log
> 18/06/2015 16:19:20 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] glusterfs_create_export :FSAL :EVENT :Volume atlas-backup-01 exported at : '/'
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] lower_my_caps :NFS STARTUP :EVENT :CAP_SYS_RESOURCE was successfully removed for proper quota management in FSAL
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] lower_my_caps :NFS STARTUP :EVENT :currenty set capabilities are: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap+ep
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Init_admin_thread :NFS CB :EVENT :Admin thread initialized
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs4_start_grace :STATE :EVENT :NFS Server Now IN GRACE, duration 60
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_rpc_cb_init_ccache :NFS STARTUP :EVENT :Callback creds directory (/var/run/ganesha) already exists
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (2:2)
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :Starting delayed executor.
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :9P/TCP dispatcher thread was started successfully
> 18/06/2015 16:19:22 : epoch 5582d368 : node2 : ganesha.nfsd-30835[_9p_disp] _9p_dispatcher_thread :9P DISP :EVENT :9P dispatcher started
> 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :gsh_dbusthread was started successfully
> 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :admin thread was started successfully
> 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :reaper thread was started successfully
> 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[reaper] nfs_in_grace :STATE :EVENT :NFS Server Now IN GRACE
> 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_Start_threads :THREAD :EVENT :General fridge was started successfully
> 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
> 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT : NFS SERVER INITIALIZED
> 18/06/2015 16:19:23 : epoch 5582d368 : node2 : ganesha.nfsd-30835[main] nfs_start :NFS STARTUP :EVENT :-------------------------------------------------
> 18/06/2015 16:20:22 : epoch 5582d3a6 : node2 : ganesha.nfsd-31542[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
> 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31550[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
> 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31557[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
> 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31562[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
> 18/06/2015 16:20:23 : epoch 5582d3a7 : node2 : ganesha.nfsd-31567[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
> 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31908[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
> 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31932[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
> 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31949[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
> 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31956[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
> 18/06/2015 16:20:49 : epoch 5582d3c1 : node2 : ganesha.nfsd-31971[main] main :MAIN :EVENT :ganesha.nfsd Starting: Ganesha Version /root/rpmbuild/BUILD/nfs-ganesha-2.2.0-3-0.1.1-Source, built at Jun 15 2015 22:13:18 on node2
>
>
>
> I have not enabled the FULL_DEBUG but if you need it I can do it.
>
>>
>> Thanks for your perseverance :)
>
> I have to thank you for the help! :-)
> Cheers,
>
> Alessandro
>
>>
>> Meghana
>>
>> ----- Original Message -----
>> From: "Alessandro De Salvo" <Alessandro.DeSalvo at roma1.infn.it>
>> To: "Meghana Madhusudhan" <mmadhusu at redhat.com>
>> Cc: gluster-users at gluster.org, nfs-ganesha-devel at lists.sourceforge.net, "Soumya Koduri" <skoduri at redhat.com>
>> Sent: Thursday, June 18, 2015 7:24:55 PM
>> Subject: Re: [Nfs-ganesha-devel] Problems in /usr/libexec/ganesha/dbus-send.sh and ganesha dbus interface when disabling exports from gluster
>>
>> Hi Meghana,
>>
>>> Il giorno 18/giu/2015, alle ore 07:04, Meghana Madhusudhan <mmadhusu at redhat.com> ha scritto:
>>>
>>>
>>>
>>>
>>> On 06/17/2015 10:57 PM, Alessandro De Salvo wrote:
>>>> Hi,
>>>> when disabling exports from gluster 3.7.1, by using gluster vol set <volume> ganesha.enable off, I always get the following error:
>>>>
>>>> Error: Dynamic export addition/deletion failed. Please see log file for details
>>>>
>>>> This message is produced by the failure of /usr/libexec/ganesha/dbus-send.sh, and in fact if I manually perform the command to remove the share I see:
>>> you got it wrong. '/usr/libexec/ganesha/dbus-send.sh' is used by
>>> Gluster-CLI to unexport the volume "gluster volume set <volname>
>>> ganesha.enable off" which rightly deletes the export file too while
>>> un-exporting the volume.
>>>
>>>>
>>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2
>>>> Error org.freedesktop.DBus.Error.NoReply: Message did not receive a reply (timeout by message bus)
>>>>
>>>> So, there is a timeout and it fails completely.
>>> Check if nfs-ganesha is still running. There was a bug in unexporting
>>> the volume. Its been fixed recently in V2.3-dev, yet to be back-ported
>>> to V2.2-stable branch.
>>> https://review.gerrithub.io/#/c/236129/
>>>
>>> Thanks,
>>> Soumya
>>>
>>>> In this case I think there is a bug in /usr/libexec/ganesha/dbus-send.sh, since it blindly deletes the share config if the RemoveExport fails (function check_cmd_status()), but leaves the %include inside ganesha.conf as in the check_cmd_status() there is a runaway condition and the other removal statements are then not executed. I believe the logic should be fixed here, otherwise even a restart of the service will fail due to the bad configuration.
>>>
>>> Yes. I see that the "sed -i /$VOL.conf/d $CONF" is placed after the check_cmd_status. I shall send a fix upstream in a related bug. But dynamic export removal
>>> will fail in three cases,
>>> 1. nfs-ganesha is not running.
>>
>> no, it was running
>>
>>> 2. The export file that is particular to that volume is somehow deleted before you perform the removal. It does depend on that file to get the export ID.
>>
>> I tried to comment out the rm in check_cmd_status to avoid this race condition, but it did not solve the problems.
>>
>>> 3. The bug that Soumya pointed out.
>>
>> This might well be the real cause!
>>
>>>
>>> If it is failing consistently, there could be something that you are missing. If you can send the exact sequence of sequence of steps that you have executed,
>>> I can help you with it.
>>
>> Yes, it’s failing consistently, unless as I said I do a DisplayExport before the RemoveExport, in which case it always works.
>>
>>>
>>> Ideally after exporting a particular volume, you'll see an entry in the /etc/ganesha/ganesha.conf file and the export file in "/etc/ganesha/exports" dir.
>>
>> And this works perfectly, I see them correctly.
>>
>>> If you have this in place and nfs-ganesha running, then dynamic export removal should work just fine.
>>
>> But this is not, at least in my case.
>> The command I’m using are just the following:
>>
>> gluster vol set <volume> ganesha.enable on
>> gluster vol set <volume> ganesha.enable off
>>
>> It normally wait a few seconds between the two commands, to give time to ganesha to actually export the volume.
>> The export is always failing as described, unless I add the DisplayExport in dbus-send.sh before RemoveExport.
>> Many thanks for the help,
>>
>> Alessandro
>>
>>
>>>
>>> Meghana
>>>
>>>
>>>>
>>>> What’s more worrying is the problem with the dbus. Issuing a DisplayExport before the RemoveExport apparently fixes the problem, so something like this always works:
>>>>
>>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.DisplayExport uint16:2
>>>> # dbus-send --print-reply --system --dest=org.ganesha.nfsd /org/ganesha/nfsd/ExportMgr org.ganesha.nfsd.exportmgr.RemoveExport uint16:2
>>>>
>>>> So, it’s like the DisplayExport is forcing someway a refresh that is needed by the RemoveExport. Any idea why?
>>>> I’m using the latest version of ganesha 2.2.0, i.e. 2.2.0-3.
>>>> Thanks,
>>>>
>>>> Alessandro
>>>>
>>>> PS: sorry for reporting so many things in a few days :-)
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Nfs-ganesha-devel mailing list
>>>> Nfs-ganesha-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Nfs-ganesha-devel mailing list
>>> Nfs-ganesha-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/nfs-ganesha-devel
>>>
>>
>>
>
More information about the Gluster-users
mailing list