[Gluster-users] gluster-3.7 cannot start volume & ganesha feature cannot turn on problem

Jiffin Tony Thottan jthottan at redhat.com
Fri Jun 5 09:11:19 UTC 2015 


On 05/06/15 13:22, 莊尚豪 wrote:
> Thank your support!
> I have already success pnfs on glusterfs & ganesha

Awesome.
> Then I have some questions.
> I have considered glusterfs with ganesha to achieve the pnfs goal.
> I read glusterfs function, and it seems to support nfs protocol by glusterfs self.
> And why glusterfs need ganesha for nfs protocol?

NFS server supported glusterfs know as gluster-nfs only supports nfs 
version 3.
To support newer versions of nfs(4, 4.1 , pnfs ), glusterfs need 
nfs-ganesha.

> It seems gluster-HA and pnfs are conflict so far.
> Does glusterfs-HA will combine with pnfs in the future?

Sure. pnfs will combine with HA in future release.
Right now , mulithead support and fail over is not guaranteed by the 
pnfs cluster.
That is why HA does not provide support for pNFS.

If you are using latest sources for nfs-ganesha(> V2.3-dev-4) , due to 
the new changes in the code,
following is required :

1.) turn on "features.cache-invalidation" for the volume
gluster v set <volume_name>   features.cache-invalidation on

2.) Requires modification in ganesha configuration file (by default it 
will be /etc/ganesha/ganesha.conf)
for the meta data server(M.D.S) ,add following block to it :

GLUSTER
{
pnfs_mds = true;
}

I suggest you use to latest sources, it has better performance 
improvement for the pnfs.

--
Jiffin.

> Perhaps I miss these information in glusterfs documents.
> Thank you replying my question.
>
> Ben
>
> -----Original Message-----
> From: Soumya Koduri [mailto:skoduri at redhat.com]
> Sent: Wednesday, June 03, 2015 9:06 PM
> To: 莊尚豪; 'Anoop C S'; gluster-users at gluster.org
> Cc: 'Meghana Madhusudhan'
> Subject: Re: [Gluster-users] gluster-3.7 cannot start volume & ganesha feature cannot turn on problem
>
> Hi Ben,
>
> Current NFS-Ganesha version (>V2.2.0) support pNFS with Gluster but not with HA. That means you need to disable HA to use pNFS. More details are in the doc - https://github.com/gluster/glusterfs/blob/master/doc/features/mount_gluster_volume_using_pnfs.md
>
> For HA, as mentioned in the demo video, there are certain pre-requisites to be followed. Noting down them below -
>
> 1. Populate /etc/hosts (or ensure DNS) on all machines.
> 2. Disable and stop NetworkManager service, enable and start network service on all machines 3. Create a gluster shared volume 'gluster_shared_storage' and mount it on '/var/run/gluster/shared_storage' on all the cluster nodes using glusterfs native mount.
> 4. Touch /etc/ganesha/ganesha.conf (empty file) 5.. `yum -y install pacemaker cman pcs ccs resource-agents corosync` on all machines 6. set cluster auth password on all machines
> - `echo hacluster | passwd --stdin hacluster`
> -  pcs cluster auth on all the nodes
> 7. Enable IPv6 on all the cluster nodes.
> 8. Enable and start pcsd on all machines
> - `chkconfig --add pcsd; chkconfig pcsd on; service pcsd start` 9. Populate /etc/ganesha/ganesha-ha.conf on all the cluster nodes.
>
> Please make sure if you have followed all of them. From the logs, looks like step3 and step4 are missed.
>
> Thanks,
> Soumya
>
> On 06/03/2015 03:29 PM, 莊尚豪 wrote:
>> Sorry I post wrong command.
>> I want to turn on ganesha feature like your command.
>> And these are the output.
>>
>> [root at n1 ~]# gluster nfs-ganesha enable Enabling NFS-Ganesha requires
>> Gluster-NFS to be disabled across the trusted pool. Do you still want
>> to continue? (y/n) y
>> nfs-ganesha: failed: Commit failed on localhost. Please check the log file for more details.
>>
>> And these are the detail log about turn on ganesha feature
>>
>> [2015-06-03 09:56:19.951124] W [common-utils.c:1676:gf_string2boolean]
>> (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x147)[0x7f9937dc0b77]
>> (--> /lib64/libglusterfs.so.0(gf_string2boolean+0x13f)[0x7f9937dd8ccf]
>> (-->
>> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_op_stage_s
>> et_ganesha+0xaa)[0x7f992cc0a32a] (-->
>> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_op_stage_v
>> alidate+0x9b)[0x7f992cbbf76b] (-->
>> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(gd_stage_op_phase+0
>> x1bd)[0x7f992cc369ed] ))))) 0-management: argument invalid
>> [2015-06-03 09:56:19.951312] I [MSGID: 106132]
>> [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already
>> stopped
>> [2015-06-03 09:56:19.952278] I
>> [glusterd-ganesha.c:364:check_host_list] 0-management: couldn't open
>> the file /etc/ganesha/ganesha-ha.conf
>> [2015-06-03 09:56:19.957777] E [glusterd-ganesha.c:670:pre_setup]
>> 0-THIS->name: mkdir() failed on path
>> /var/run/gluster/shared_storage/nfs-ganesha,errno: 沒有此一檔案或目錄
>> [2015-06-03 09:56:19.957874] E
>> [glusterd-ganesha.c:254:glusterd_op_set_ganesha] 0-management: Initial
>> NFS-Ganesha set up failed
>> [2015-06-03 09:56:19.957899] E
>> [glusterd-syncop.c:1396:gd_commit_op_phase] 0-management: Commit of
>> operation 'Volume (null)' failed on localhost
>>
>>
>>
>> By the way, does glusterfs HA for ganesha support pnfs?
>>
>> Many thanks,
>> Ben
>>
>> -----Original Message-----
>> From: Soumya Koduri [mailto:skoduri at redhat.com]
>> Sent: Wednesday, June 03, 2015 3:35 PM
>> To: 莊尚豪; 'Anoop C S'; gluster-users at gluster.org
>> Cc: 'Meghana Madhusudhan'
>> Subject: Re: [Gluster-users] gluster-3.7 cannot start volume & ganesha
>> feature cannot turn on problem
>>
>>
>>
>> On 06/03/2015 12:01 PM, 莊尚豪 wrote:
>>> Thank for your reply.
>>>
>>> I have already upgrade glusterfs 3.7.1 and I can start the volume!
>>> Therefore I am going to turn on ganesha feature by glusterfs.
>>>
>>> [root at n1 yum.repos.d]# gluster volume set ganesha ganesha.enable on
>>> volume set: failed: The option nfs-ganesha should be enabled before setting ganesha.enable.
>> This error means that 'nfs-ganesha' feature is not enabled. It needs to be enabled prior to exporting any volumes via ganesha.
>>
>> cmd to be used - 'gluster nfs-ganesha enable'.
>>
>> In case if you already have it enabled, check if it was successful
>> using below commands -
>>
>> 'showmount -e localhost' --> should display '/'
>> 'pcs status' --> Should display all the nodes' (which you have configured for HA ) pcs status with out any errors reported.
>>
>> Thanks,
>> Soumya
>>
>>
>>> There is the log in "/var/log/glusterfs/etc-glusterfs-glusterd.vol.log"
>>>
>>> [2015-06-03 06:12:29.976469] W
>>> [common-utils.c:1676:gf_string2boolean]
>>> (-->
>>> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x147)[0x7f9937dc0b77]
>>> (-->
>>> /lib64/libglusterfs.so.0(gf_string2boolean+0x13f)[0x7f9937dd8ccf]
>>> (-->
>>> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_op_stage_
>>> s et_ganesha+0xaa)[0x7f992cc0a32a] (-->
>>> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_op_stage_
>>> v
>>> alidate+0x9b)[0x7f992cbbf76b] (-->
>>> /usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(gd_stage_op_phase+
>>> 0 x1bd)[0x7f992cc369ed] ))))) 0-management: argument invalid
>>> [2015-06-03 06:12:29.976823] I [MSGID: 106132]
>>> [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs
>>> already stopped
>>> [2015-06-03 06:12:29.976938] I
>>> [glusterd-ganesha.c:364:check_host_list] 0-management: couldn't open
>>> the file /etc/ganesha/ganesha-ha.conf
>>> [2015-06-03 06:12:29.984693] E [glusterd-ganesha.c:670:pre_setup]
>>> 0-THIS->name: mkdir() failed on path
>>> /var/run/gluster/shared_storage/nfs-ganesha,errno: 沒有此一檔案或目錄
>>> [2015-06-03 06:12:29.984851] E
>>> [glusterd-ganesha.c:254:glusterd_op_set_ganesha] 0-management:
>>> Initial NFS-Ganesha set up failed
>>> [2015-06-03 06:12:29.984883] E
>>> [glusterd-syncop.c:1396:gd_commit_op_phase] 0-management: Commit of
>>> operation 'Volume (null)' failed on localhost
>>>
>>> The Chinese log means that the path is empty or no directory.
>>> It seems that glusterfs does not build ganesha configuration in initialization.
>>> How do I continue next step?
>>>
>>> Thanks,
>>> Ben
>>>
>>>
>>> -----Original Message-----
>>> From: Anoop C S [mailto:achiraya at redhat.com]
>>> Sent: Tuesday, June 02, 2015 11:18 PM
>>> To: benchuang at iii.org.tw; gluster-users at gluster.org
>>> Cc: Meghana Madhusudhan; Soumya Koduri
>>> Subject: Re: [Gluster-users] gluster-3.7 cannot start volume &
>>> ganesha feature cannot turn on problem
>>>
>>>
>>> Can you please attach the glusterd logs here? You are having trouble to even start the volume here right?
>>> And also HA configuration is mandatory to use NFS-Ganesha in this release. Once you have the volume started, I can help you with the remaining steps in detail.
>>>
>>> Thanks
>>> Meghana
>>>
>>> ----- Original Message -----
>>> From: "Anoop C S" <achiraya at redhat.com>
>>> To: gluster-users at gluster.org
>>> Cc: "Meghana Madhusudhan" <mmadhusu at redhat.com>, "Soumya Koduri"
>>> <skoduri at redhat.com>
>>> Sent: Tuesday, June 2, 2015 4:38:39 PM
>>> Subject: Re: [Gluster-users] gluster-3.7 cannot start volume &
>>> ganesha feature cannot turn on problem manadatroy
>>>
>>>
>>> On 06/02/2015 01:42 PM, 莊尚豪 wrote:
>>>> Hi all,
>>>>
>>>> I have two question for glusterfs-3.7 on fedora-22
>>>>
>>>> I used to have a glusterfs cluster version 3.6.2.
>>>>
>>>> The following configuration can be work in version-3.6.2, but not in
>>>> version-3.7
>>>>
>>>>
>>>>
>>>> There is 2 node for glusterfs.
>>>>
>>>> OS: fedora 22
>>>>
>>>> Gluster: 3.7 on
>>>> https://download.gluster.org/pub/gluster/glusterfs/3.7/3.7.0/
>>>>
>>>>
>>>>
>>>> #gluster peer probe n1
>>>>
>>>> #gluster volume create ganesha n1:/data/brick1/gv0
>>>> n2:/data/brick1/gv0
>>>>
>>>>
>>>>
>>>> Volume Name: ganesha
>>>>
>>>> Type: Distribute
>>>>
>>>> Volume ID: cbb8d360-0025-419c-a12b-b29e4b91d7f8
>>>>
>>>> Status: Created
>>>>
>>>> Number of Bricks: 2
>>>>
>>>> Transport-type: tcp
>>>>
>>>> Bricks:
>>>>
>>>> Brick1: n1:/data/brick1/gv0
>>>>
>>>> Brick2: n2:/data/brick1/gv0
>>>>
>>>> Options Reconfigured:
>>>>
>>>> performance.readdir-ahead: on
>>>>
>>>>
>>>>
>>>> The problem to start the volume "ganesha"
>>>>
>>>> #gluster volume start ganesha
>>>>
>>>>
>>>>
>>>> volume start: ganesha: failed: Commit failed on localhost. Please
>>>> check the log file for more details.
>>>>
>>>>
>>>>
>>>> LOG in /var/log/glusterfs/bricks/data-brick1-gv0.log
>>>>
>>>>
>>>>
>>>> [2015-06-02 08:02:55.232923] I [MSGID: 100030]
>>>> [glusterfsd.c:2294:main] 0-/usr/sbin/glusterfsd: Started running
>>>> /usr/sbin/glusterfsd version 3.7.0 (args: /usr/sbin/glusterfsd -s
>>>> n2 --volfile-id ganesha.n2.data-brick1-gv0 -p
>>>> /var/lib/glusterd/vols/ganesha/run/n2-data-brick1-gv0.pid -S
>>>> /var/run/gluster/73ea8a39514304f5ebd440321d784386.socket
>>>> --brick-name /data/brick1/gv0 -l
>>>> /var/log/glusterfs/bricks/data-brick1-gv0.log --xlator-option
>>>> *-posix.glusterd-uuid=35547067-d343-4fee-802a-0e911b5a07cd
>>>> --brick-port 49157 --xlator-option
>>>> ganesha-server.listen-port=49157)
>>>>
>>>> [2015-06-02 08:02:55.284206] I
>>>> [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started
>>>> thread with index 1
>>>>
>>>> [2015-06-02 08:02:55.397923] W [xlator.c:192:xlator_dynload]
>>>> 0-xlator: /usr/lib64/glusterfs/3.7.0/xlator/features/changelog.so:
>>>> undefined symbol: changelog_select_event
>>>>
>>> This particular error for "undefined symbol changelog_select_event"
>>> was identified recently and corresponding fix [
>>> http://review.gluster.org/#/c/11004/ ] is already in master and
>>> hopefully will be available with v3.7.1.
>>>
>>>> [2015-06-02 08:02:55.397963] E [graph.y:212:volume_type] 0-parser:
>>>> Volume 'ganesha-changelog', line 30: type 'features/changelog' is
>>>> not valid or not found on this machine
>>>>
>>>> [2015-06-02 08:02:55.397992] E [graph.y:321:volume_end] 0-parser:
>>>> "type" not specified for volume ganesha-changelog
>>>>
>>>> [2015-06-02 08:02:55.398214] E [MSGID: 100026]
>>>> [glusterfsd.c:2149:glusterfs_process_volfp] 0-: failed to construct
>>>> the graph
>>>>
>>>> [2015-06-02 08:02:55.398423] W
>>>> [glusterfsd.c:1219:cleanup_and_exit] (--> 0-: received signum (0),
>>>> shutting down
>>>>
>>>>
>>>>
>>>> I cannot google method to resolve it.
>>>>
>>>> Does anyone have across this problem?
>>>>
>>>>
>>>>
>>>> Another question is the feature in nfs-ganesha(version 2.2)
>>>>
>>>> The volume command I cannot turn on this feature.
>>>>
>>>> I try to copy the demo glusterfs-ganesha video but cannot work.
>>>>
>>>> Demo link:
>>>> https://plus.google.com/events/c9omal6366f2cfkcd0iuee5ta1o
>>>>
>>>>
>>>>
>>>> [root at n1 brick1]# gluster nfs-ganesha enable
>>>>
>>>> Enabling NFS-Ganesha requires Gluster-NFS to be disabled across the
>>>> trusted pool. Do you still want to continue? (y/n) y
>>>>
>>>> nfs-ganesha: failed: Commit failed on localhost. Please check the
>>>> log file for more details.
>>>>
>>>>
>>> Adding ganesha folks to the thread.
>>>
>>>> Does anyone have the detail configuration?
>>>>
>>>> THANKS for giving advice.
>>>>
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Ben
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________ Gluster-users
>>>> mailing list Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

More information about the Gluster-users mailing list