<div dir="ltr"><div>Hi Soumya,</div><div><br></div><div>Thank you for the answer.</div><div><br></div><div>Enabling Pacemaker? Yes, you’re completely right, I didn’t do it. Thank you.</div><div><br></div><div>I spent some time by testing and I have some results. This is what I did:</div><div><br></div><div> - Clean installation of CentOS 7.3 with all updates, 3x node, resolvable IPs and VIPs</div><div> - Stopped firewalld (just for testing)</div><div> - Install "centos-release-gluster" to get "centos-gluster310" repo and install following (nothing else):</div><div> --- glusterfs-server</div><div> --- glusterfs-ganesha</div><div> - Passwordless SSH between all nodes (/var/lib/glusterd/nfs/secret.pem and secret.pem.pub on all nodes)</div><div> - systemctl enable and start glusterd</div><div> - gluster peer probe <other nodes></div><div> - gluster volume set all cluster.enable-shared-storage enable</div><div> - systemctl enable and start pcsd.service</div><div> - systemctl enable pacemaker.service (cannot be started at this moment)</div><div> - Set password for hacluster user on all nodes</div><div> - pcs cluster auth <node 1> <node 2> <node 3> -u hacluster -p blabla</div><div> - mkdir /var/run/gluster/shared_storage/nfs-ganesha/</div><div> - touch /var/run/gluster/shared_storage/nfs-ganesha/ganesha.conf (not sure if needed)</div><div> - vi /var/run/gluster/shared_storage/nfs-ganesha/ganesha-ha.conf and insert configuration</div><div> - Try list files on other nodes: ls /var/run/gluster/shared_storage/nfs-ganesha/</div><div> - gluster nfs-ganesha enable</div><div> - Check on other nodes that nfs-ganesha.service is running and "pcs status" shows started resources</div><div> - gluster volume create mynewshare replica 3 transport tcp node1:/<dir> node2:/<dir> node3:/<dir></div><div> - gluster volume start mynewshare</div><div> - gluster vol set mynewshare ganesha.enable on</div><div><br></div><div>After these steps, all VIPs are pingable and I can mount node1:/mynewshare</div><div><br></div><div>Funny thing is that pacemaker.service is disabled again (something disabled it). This is status of important (I think) services:</div><div><br></div><div>systemctl list-units --all</div><div># corosync.service loaded active running </div><div># glusterd.service loaded active running</div><div># nfs-config.service loaded inactive dead</div><div># nfs-ganesha-config.service loaded inactive dead</div><div># nfs-ganesha-lock.service loaded active running</div><div># nfs-ganesha.service loaded active running</div><div># nfs-idmapd.service loaded inactive dead</div><div># nfs-mountd.service loaded inactive dead</div><div># nfs-server.service loaded inactive dead</div><div># nfs-utils.service loaded inactive dead</div><div># pacemaker.service loaded active running</div><div># pcsd.service loaded active running</div><div><br></div><div>systemctl list-unit-files --all</div><div># corosync-notifyd.service disabled</div><div># corosync.service disabled</div><div># glusterd.service enabled</div><div># glusterfsd.service disabled</div><div># nfs-blkmap.service disabled</div><div># nfs-config.service static </div><div># nfs-ganesha-config.service static </div><div># nfs-ganesha-lock.service static </div><div># nfs-ganesha.service disabled</div><div># nfs-idmap.service static </div><div># nfs-idmapd.service static </div><div># nfs-lock.service static </div><div># nfs-mountd.service static </div><div># nfs-rquotad.service disabled</div><div># nfs-secure-server.service static </div><div># nfs-secure.service static </div><div># nfs-server.service disabled</div><div># nfs-utils.service static </div><div># nfs.service disabled</div><div># nfslock.service static </div><div># pacemaker.service disabled</div><div># pcsd.service enabled</div><div><br></div><div>I enabled pacemaker again on all nodes and restart all nodes one by one.</div><div><br></div><div>After reboot all VIPs are gone and I can see that nfs-ganesha.service isn’t running. When I start it on at least two nodes then VIPs are pingable again and I can mount NFS again. But there is still some issue in the setup because when I check nfs-ganesha-lock.service I get:</div><div> </div><div>systemctl -l status nfs-ganesha-lock.service </div><div>● nfs-ganesha-lock.service - NFS status monitor for NFSv2/3 locking.</div><div> Loaded: loaded (/usr/lib/systemd/system/nfs-ganesha-lock.service; static; vendor preset: disabled)</div><div> Active: failed (Result: exit-code) since Fri 2017-05-05 13:43:37 UTC; 31min ago</div><div> Process: 6203 ExecStart=/usr/sbin/rpc.statd --no-notify $STATDARGS (code=exited, status=1/FAILURE)</div><div><br></div><div>May 05 13:43:37 node0.localdomain systemd[1]: Starting NFS status monitor for NFSv2/3 locking....</div><div>May 05 13:43:37 node0.localdomain rpc.statd[6205]: Version 1.3.0 starting</div><div>May 05 13:43:37 node0.localdomain rpc.statd[6205]: Flags: TI-RPC</div><div>May 05 13:43:37 node0.localdomain rpc.statd[6205]: Failed to open directory sm: Permission denied</div><div>May 05 13:43:37 node0.localdomain rpc.statd[6205]: Failed to open /var/lib/nfs/statd/state: Permission denied</div><div>May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-lock.service: control process exited, code=exited status=1</div><div>May 05 13:43:37 node0.localdomain systemd[1]: Failed to start NFS status monitor for NFSv2/3 locking..</div><div>May 05 13:43:37 node0.localdomain systemd[1]: Unit nfs-ganesha-lock.service entered failed state.</div><div>May 05 13:43:37 node0.localdomain systemd[1]: nfs-ganesha-lock.service failed.</div><div><br></div><div>Thank you,</div><div><br></div><div>Kind regards,</div><div><br></div><div>Adam</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 3, 2017 at 10:32 AM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_4202432446914188956divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>Same here, when i reboot the node i have to manually execute "<span>pcs cluster start gluster01" and pcsd already enabled and started.</span></p>
<p><span>Gluster 3.8.11</span></p>
<p><span>Centos 7.3 latest</span></p>
<p><span>Installed using <span>CentOS Storage SIG repository</span></span></p><span class="HOEnZb"><font color="#888888">
<p><br>
</p>
<div id="m_4202432446914188956Signature"><br>
<div class="m_4202432446914188956ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</font></span></div><span class="HOEnZb"><font color="#888888">
<hr style="display:inline-block;width:98%">
<div id="m_4202432446914188956divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> <a href="mailto:gluster-users-bounces@gluster.org" target="_blank">gluster-users-bounces@gluster.<wbr>org</a> <<a href="mailto:gluster-users-bounces@gluster.org" target="_blank">gluster-users-bounces@<wbr>gluster.org</a>> on behalf of Adam Ru <<a href="mailto:ad.ruckel@gmail.com" target="_blank">ad.ruckel@gmail.com</a>><br>
<b>Sent:</b> Wednesday, May 3, 2017 12:09:58 PM<br>
<b>To:</b> Soumya Koduri<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster and NFS-Ganesha - cluster is down after reboot</font>
<div> </div>
</div></font></span><div><div class="h5">
<div>
<div dir="ltr">
<div>Hi Soumya,</div>
<div><br>
</div>
<div>thank you very much for your reply.</div>
<div><br>
</div>
<div>I enabled pcsd during setup and after reboot during troubleshooting I manually started it and checked resources (pcs status). They were not running. I didn’t find what was wrong but I’m going to try it again.</div>
<div><br>
</div>
<div>I’ve thoroughly checked</div>
<div><a href="http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/" target="_blank">http://gluster.readthedocs.io/<wbr>en/latest/Administrator%<wbr>20Guide/NFS-Ganesha%<wbr>20GlusterFS%20Integration/</a></div>
<div>and I can confirm that I followed all steps with one exception. I installed following RPMs:</div>
<div>glusterfs-server</div>
<div>glusterfs-fuse</div>
<div>glusterfs-cli</div>
<div>glusterfs-ganesha</div>
<div>nfs-ganesha-xfs</div>
<div><br>
</div>
<div>and the guide referenced above specifies:</div>
<div>glusterfs-server</div>
<div>glusterfs-api</div>
<div>glusterfs-ganesha</div>
<div><br>
</div>
<div>glusterfs-api is a dependency of one of RPMs that I installed so this is not a problem. But I cannot find any mention to install nfs-ganesha-xfs.</div>
<div><br>
</div>
<div>I’ll try to setup the whole environment again without installing nfs-ganesha-xfs (I assume glusterfs-ganesha has all required binaries).</div>
<div><br>
</div>
<div>Again, thank you for you time to answer my previous message.</div>
<div><br>
</div>
<div>Kind regards,</div>
<div>Adam</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, May 2, 2017 at 8:49 AM, Soumya Koduri <span dir="ltr">
<<a href="mailto:skoduri@redhat.com" target="_blank">skoduri@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi,<span class="m_4202432446914188956gmail-"><br>
<br>
On 05/02/2017 01:34 AM, Rudolf wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Hi Gluster users,<br>
<br>
First, I'd like to thank you all for this amazing open-source! Thank you!<br>
<br>
I'm working on home project – three servers with Gluster and<br>
NFS-Ganesha. My goal is to create HA NFS share with three copies of each<br>
file on each server.<br>
<br>
My systems are CentOS 7.3 Minimal install with the latest updates and<br>
the most current RPMs from "centos-gluster310" repository.<br>
<br>
I followed this tutorial:<br>
<a href="http://blog.gluster.org/2015/10/linux-scale-out-nfsv4-using-nfs-ganesha-and-glusterfs-one-step-at-a-time/" rel="noreferrer" target="_blank">http://blog.gluster.org/2015/1<wbr>0/linux-scale-out-nfsv4-using-<wbr>nfs-ganesha-and-glusterfs-one-<wbr>step-at-a-time/</a><br>
(second half that describes multi-node HA setup)<br>
<br>
with a few exceptions:<br>
<br>
1. All RPMs are from "centos-gluster310" repo that is installed by "yum<br>
-y install centos-release-gluster"<br>
2. I have three nodes (not four) with "replica 3" volume.<br>
3. I created empty ganesha.conf and not empty ganesha-ha.conf in<br>
"/var/run/gluster/shared_stora<wbr>ge/nfs-ganesha/" (referenced blog post is<br>
outdated, this is now requirement)<br>
4. ganesha-ha.conf doesn't have "HA_VOL_SERVER" since this isn't needed<br>
anymore.<br>
<br>
</blockquote>
<br>
</span>Please refer to <a href="http://gluster.readthedocs.io/en/latest/Administrator%20Guide/NFS-Ganesha%20GlusterFS%20Integration/" rel="noreferrer" target="_blank">
http://gluster.readthedocs.io/<wbr>en/latest/Administrator%20Guid<wbr>e/NFS-Ganesha%20GlusterFS%20In<wbr>tegration/</a><br>
<br>
It is being updated with latest changes happened wrt setup.<span class="m_4202432446914188956gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
When I finish configuration, all is good. nfs-ganesha.service is active<br>
and running and from client I can ping all three VIPs and I can mount<br>
NFS. Copied files are replicated to all nodes.<br>
<br>
But when I restart nodes (one by one, with 5 min. delay between) then I<br>
cannot ping or mount (I assume that all VIPs are down). So my setup<br>
definitely isn't HA.<br>
<br>
I found that:<br>
# pcs status<br>
Error: cluster is not currently running on this node<br>
</blockquote>
<br>
</span>This means pcsd service is not up. Did you enable (systemctl enable pcsd) pcsd service so that is comes up post reboot automatically. If not please start it manually.<span class="m_4202432446914188956gmail-"><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
and nfs-ganesha.service is in inactive state. Btw. I didn't enable<br>
"systemctl enable nfs-ganesha" since I assume that this is something<br>
that Gluster does.<br>
</blockquote>
<br>
</span>Please check /var/log/ganesha.log for any errors/warnings.<br>
<br>
We recommend not to enable nfs-ganesha.service (by default), as the shared storage (where the ganesha.conf file resides now) should be up and running before nfs-ganesha gets started.<br>
So if enabled by default it could happen that shared_storage mount point is not yet up and it resulted in nfs-ganesha service failure. If you would like to address this, you could have a cron job which keeps checking the mount point health and then start nfs-ganesha
service.<br>
<br>
Thanks,<br>
Soumya<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="m_4202432446914188956gmail-"><br>
I assume that my issue is that I followed instructions in blog post from<br>
2015/10 that are outdated. Unfortunately I cannot find anything better –<br>
I spent whole day by googling.<br>
<br>
Would you be so kind and check the instructions in blog post and let me<br>
know what steps are wrong / outdated? Or please do you have more current<br>
instructions for Gluster+Ganesha setup?<br>
<br>
Thank you.<br>
<br>
Kind regards,<br>
Adam<br>
<br>
<br>
<br>
</span>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
<br>
</blockquote>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="m_4202432446914188956gmail_signature">
<div dir="ltr">Adam</div>
</div>
</div>
</div>
</div>
</div></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Adam</div></div>
</div>