[Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot

[hvjunk]

> On 15 May 2017, at 12:56 PM, Soumya Koduri <skoduri at redhat.com> wrote:
> 
> 
> 
> On 05/12/2017 06:27 PM, Adam Ru wrote:
>> Hi Soumya,
>> 
>> Thank you very much for last response – very useful.
>> 
>> I apologize for delay, I had to find time for another testing.
>> 
>> I updated instructions that I provided in previous e-mail. *** means
>> that the step was added.
>> 
>> Instructions:
>> - Clean installation of CentOS 7.3 with all updates, 3x node,
>> resolvable IPs and VIPs
>> - Stopped firewalld (just for testing)
>> - *** SELinux in permissive mode (I had to, will explain bellow)
>> - Install “centos-release-gluster" to get "centos-gluster310" repo

should I also install the centos-gluster310, or will that be automagically chosen by the centos-release-gluster?

>> and install following (nothing else):
>> --- glusterfs-server
>> --- glusterfs-ganesha
>> - Passwordless SSH between all nodes
>> (/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes)
>> - systemctl enable and start glusterd
>> - gluster peer probe <other nodes>
>> - gluster volume set all cluster.enable-shared-storage enable

After this step, I’ll advise (given my experience in doing this by Ansible) to make sure that the shared filesystem have propagated to all the nodes, as well as the needed entries made in fstab… safety check, and I’ll also load my systemd service and helper script to assist in cluster cold-bootstrapping.

>> - systemctl enable and start pcsd.service
>> - systemctl enable pacemaker.service (cannot be started at this moment)
>> - Set password for hacluster user on all nodes
>> - pcs cluster auth <node 1> <node 2> <node 3> -u hacluster -p blabla
>> - mkdir /var/run/gluster/shared_storage/nfs-ganesha/
>> - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not
>> sure if needed)
>> - vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and
>> insert configuration
>> - Try list files on other nodes: ls
>> /var/run/gluster/shared_storage/nfs-ganesha/
>> - gluster nfs-ganesha enable
>> - *** systemctl enable pacemaker.service (again, since pacemaker was
>> disabled at this point)
>> - *** Check owner of "state", "statd", "sm" and "sm.bak" in
>> /var/lib/nfs/ (I had to: chown rpcuser:rpcuser
>> /var/lib/nfs/statd/state)
>> - Check on other nodes that nfs-ganesha.service is running and "pcs
>> status" shows started resources
>> - gluster volume create mynewshare replica 3 transport tcp
>> node1:/<dir> node2:/<dir> node3:/<dir>
>> - gluster volume start mynewshare
>> - gluster vol set mynewshare ganesha.enable on
>> 
>> At this moment, this is status of important (I think) services:
>> 
>> -- corosync.service             disabled
>> -- corosync-notifyd.service     disabled
>> -- glusterd.service             enabled
>> -- glusterfsd.service           disabled
>> -- pacemaker.service            enabled
>> -- pcsd.service                 enabled
>> -- nfs-ganesha.service          disabled
>> -- nfs-ganesha-config.service   static
>> -- nfs-ganesha-lock.service     static
>> 
>> -- corosync.service             active (running)
>> -- corosync-notifyd.service     inactive (dead)
>> -- glusterd.service             active (running)
>> -- glusterfsd.service           inactive (dead)
>> -- pacemaker.service            active (running)
>> -- pcsd.service                 active (running)
>> -- nfs-ganesha.service          active (running)
>> -- nfs-ganesha-config.service   inactive (dead)
>> -- nfs-ganesha-lock.service     active (running)
>> 
>> May I ask you a few questions please?
>> 
>> 1. Could you please confirm that services above has correct status/state?
> 
> Looks good to the best of my knowledge.
> 
>> 
>> 2. When I restart a node then nfs-ganesha is not running. Of course I
>> cannot enable it since it needs to be enabled after shared storage is
>> mounted. What is best practice to start it automatically so I don’t
>> have to worry about restarting node? Should I create a script that
>> will check whether shared storage was mounted and then start
>> nfs-ganesha? How do you do this in production?
> 
> That's right.. We have plans to address this in near future (probably by having a new .service which mounts shared_storage before starting nfs-ganesha). But until then ..yes having a custom defined script to do so is the only way to automate it.

Refer to my previous posting that has a script & systemd service that address this problematic bootstrapping issue w.r.t. locally mounted gluster directories, which the shared directory is.
That could be used (with my permission) as a basis to help fix this issue…