[Gluster-users] Fwd: nfs-ganesha HA with arbiter volume

Kaleb S. KEITHLEY kkeithle at redhat.com
Wed Sep 23 12:17:04 UTC 2015


On 09/23/2015 06:14 AM, Tiemen Ruiten wrote:
> 
> Does the ganesha-ha.sh script handle multiple network interfaces? 

No. There's a parameter for the IPaddr resource agent you can use to
specify the NIC. You will have to modify ganesha-ha.sh to add it.

> There
> are two interfaces on both servers and corosync/pacemaker should use
> only one of them.


> 
> On 22 September 2015 at 21:44, Tiemen Ruiten <t.ruiten at rdmedia.com
> <mailto:t.ruiten at rdmedia.com>> wrote:
> 
>     Allright, thank you Soumya. I actually did do the cleanup every time
>     (gluster nfs-ganesha disable), but it didn't always finish
>     succesfully. Sometimes it would just time out. I'll try with the
>     second command tomorrow.
> 
>     Good to know that it should work with two nodes as well.
> 
>     On 22 September 2015 at 19:26, Soumya Koduri <skoduri at redhat.com
>     <mailto:skoduri at redhat.com>> wrote:
> 
> 
> 
>         On 09/22/2015 05:06 PM, Tiemen Ruiten wrote:
> 
>             That's correct and my original question was actually if a
>             two node +
>             arbiter setup is possible. The documentation provided by
>             Soumya only
>             mentions two servers in the example ganesha-ha.sh script.
>             Perhaps that
>             could be updated as well then, to not give the wrong impression.
> 
>         It does work with 2-node as well. In the script, there is
>         already a check to verify if the number of servers < 3, it
>         automatically disables quorum.
>         Quorum cannot be enabled for a 2-node setup for obvious reasons.
>         If one node fails, other node just takes over the IP.
> 
>         Thanks,
>         Soumya
> 
>             I could try to change the script to disable quorum, but
>             wouldn't that
>             defeat the purpose? What will happen in case one node goes down
>             unexpectedly?
> 
>             On 22 September 2015 at 12:47, Kaleb Keithley
>             <kkeithle at redhat.com <mailto:kkeithle at redhat.com>
>             <mailto:kkeithle at redhat.com <mailto:kkeithle at redhat.com>>>
>             wrote:
> 
> 
>                 Hi,
> 
>                 IIRC, the setup is two nodes gluster+ganesha nodes plus
>             the arbiter
>                 node for gluster quorum.
> 
>                 Have I remembered that correctly?
> 
>                 The Ganesha HA in 3.7 requires a minimum of three
>             servers running
>                 ganesha and pacemaker. Two might work if you change the
>                 ganesha-ha.sh to not enable pacemaker quorum, but I
>             haven't tried
>                 that myself. I'll try and find time in the next couple
>             of days to
>                 update the documentation or write a blog post.
> 
> 
> 
>                 ----- Original Message ----
>                  >
>                  >
>                  >
>                  > On 21/09/15 21:21, Tiemen Ruiten wrote:
>                  > > Whoops, replied off-list.
>                  > >
>                  > > Additionally I noticed that the generated corosync
>             config is not
>                  > > valid, as there is no interface section:
>                  > >
>                  > > /etc/corosync/corosync.conf
>                  > >
>                  > > totem {
>                  > > version: 2
>                  > > secauth: off
>                  > > cluster_name: rd-ganesha-ha
>                  > > transport: udpu
>                  > > }
>                  > >
>                  > > nodelist {
>                  > >   node {
>                  > >         ring0_addr: cobalt
>                  > >         nodeid: 1
>                  > >        }
>                  > >   node {
>                  > >         ring0_addr: iron
>                  > >         nodeid: 2
>                  > >        }
>                  > > }
>                  > >
>                  > > quorum {
>                  > > provider: corosync_votequorum
>                  > > two_node: 1
>                  > > }
>                  > >
>                  > > logging {
>                  > > to_syslog: yes
>                  > > }
>                  > >
>                  > >
>                  > >
>                  >
>                  > May be Kaleb can help you out.
>                  > >
>                  > > ---------- Forwarded message ----------
>                 > > From: *Tiemen Ruiten* <t.ruiten at rdmedia.com
>             <mailto:t.ruiten at rdmedia.com> <mailto:t.ruiten at rdmedia.com
>             <mailto:t.ruiten at rdmedia.com>>
>                 <mailto:t.ruiten at rdmedia.com
>             <mailto:t.ruiten at rdmedia.com> <mailto:t.ruiten at rdmedia.com
>             <mailto:t.ruiten at rdmedia.com>>>>
>                 > > Date: 21 September 2015 at 17:16
>                 > > Subject: Re: [Gluster-users] nfs-ganesha HA with
>             arbiter volume
>                  > > To: Jiffin Tony Thottan <jthottan at redhat.com
>             <mailto:jthottan at redhat.com>
>                 <mailto:jthottan at redhat.com
>             <mailto:jthottan at redhat.com>> <mailto:jthottan at redhat.com
>             <mailto:jthottan at redhat.com>
>                 <mailto:jthottan at redhat.com <mailto:jthottan at redhat.com>>>>
>                  > >
>                  > >
>                  > > Could you point me to the latest documentation?
>             I've been
>                 struggling
>                  > > to find something up-to-date. I believe I have all the
>                 prerequisites:
>                  > >
>                  > > - shared storage volume exists and is mounted
>                  > > - all nodes in hosts files
>                  > > - Gluster-NFS disabled
>                  > > - corosync, pacemaker and nfs-ganesha rpm's installed
>                  > >
>                  > > Anything I missed?
>                  > >
>                  > > Everything has been installed by RPM so is in the
>             default
>                 locations:
>                  > > /usr/libexec/ganesha/ganesha-ha.sh
>                  > > /etc/ganesha/ganesha.conf (empty)
>                  > > /etc/ganesha/ganesha-ha.conf
>                  > >
>                  >
>                  > Looks fine for me.
>                  >
>                  > > After I started the pcsd service manually,
>             nfs-ganesha could be
>                  > > enabled successfully, but there was no virtual IP
>             present on the
>                  > > interfaces and looking at the system log, I noticed
>             corosync
>                 failed to
>                  > > start:
>                  > >
>                  > > - on the host where I issued the gluster
>             nfs-ganesha enable
>                 command:
>                  > >
>                  > > Sep 21 17:07:18 iron systemd: Starting NFS-Ganesha
>             file server...
>                  > > Sep 21 17:07:19 iron systemd: Started NFS-Ganesha
>             file server.
>                  > > Sep 21 17:07:19 iron rpc.statd[2409]: Received
>             SM_UNMON_ALL request
>                  > > from iron.int.rdmedia.com
>             <http://iron.int.rdmedia.com> <http://iron.int.rdmedia.com>
>                 <http://iron.int.rdmedia.com> while not
>                  > > monitoring any hosts
>                  > > Sep 21 17:07:20 iron systemd: Starting Corosync
>             Cluster Engine...
>                  > > Sep 21 17:07:20 iron corosync[3426]: [MAIN  ]
>             Corosync Cluster
>                 Engine
>                  > > ('2.3.4'): started and ready to provide service.
>                  > > Sep 21 17:07:20 iron corosync[3426]: [MAIN  ]
>             Corosync built-in
>                  > > features: dbus systemd xmlconf snmp pie relro bindnow
>                  > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ]
>             Initializing
>                 transport
>                  > > (UDP/IP Unicast).
>                  > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ]
>             Initializing
>                  > > transmit/receive security (NSS) crypto: none hash: none
>                  > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] The
>             network interface
>                  > > [10.100.30.38] is now up.
>                  > > Sep 21 17:07:20 iron corosync[3427]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync configuration map access [0]
>                  > > Sep 21 17:07:20 iron corosync[3427]: [QB    ]
>             server name: cmap
>                  > > Sep 21 17:07:20 iron corosync[3427]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync configuration service [1]
>                  > > Sep 21 17:07:20 iron corosync[3427]: [QB    ]
>             server name: cfg
>                  > > Sep 21 17:07:20 iron corosync[3427]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync cluster closed process group service v1.01 [2]
>                  > > Sep 21 17:07:20 iron corosync[3427]: [QB    ]
>             server name: cpg
>                  > > Sep 21 17:07:20 iron corosync[3427]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync profile loading service [4]
>                  > > Sep 21 17:07:20 iron corosync[3427]: [QUORUM] Using
>             quorum provider
>                  > > corosync_votequorum
>                  > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ]
>             Waiting for all
>                 cluster
>                  > > members. Current votes: 1 expected_votes: 2
>                  > > Sep 21 17:07:20 iron corosync[3427]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync vote quorum service v1.0 [5]
>                  > > Sep 21 17:07:20 iron corosync[3427]: [QB    ]
>             server name:
>                 votequorum
>                  > > Sep 21 17:07:20 iron corosync[3427]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync cluster quorum service v0.1 [3]
>                  > > Sep 21 17:07:20 iron corosync[3427]: [QB    ]
>             server name: quorum
>                  > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ]
>             adding new UDPU
>                 member
>                  > > {10.100.30.38}
>                  > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ]
>             adding new UDPU
>                 member
>                  > > {10.100.30.37}
>                  > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] A new
>             membership
>                  > > (10.100.30.38:104 <http://10.100.30.38:104>
>             <http://10.100.30.38:104>
>                 <http://10.100.30.38:104>) was formed. Members joined: 1
>                 > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ]
>             Waiting for all cluster
>                 > > members. Current votes: 1 expected_votes: 2
>                 > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ]
>             Waiting for all cluster
>                 > > members. Current votes: 1 expected_votes: 2
>                 > > Sep 21 17:07:20 iron corosync[3427]: [VOTEQ ]
>             Waiting for all cluster
>                 > > members. Current votes: 1 expected_votes: 2
>                 > > Sep 21 17:07:20 iron corosync[3427]: [QUORUM]
>             Members[1]: 1
>                 > > Sep 21 17:07:20 iron corosync[3427]: [MAIN  ]
>             Completed service
>                 > > synchronization, ready to provide service.
>                 > > Sep 21 17:07:20 iron corosync[3427]: [TOTEM ] A new
>             membership
>                  > > (10.100.30.37:108 <http://10.100.30.37:108>
>             <http://10.100.30.37:108>
>                 <http://10.100.30.37:108>) was formed. Members joined: 1
>                  > > Sep 21 17:08:21 iron corosync: Starting Corosync
>             Cluster Engine
>                  > > (corosync): [FAILED]
>                  > > Sep 21 17:08:21 iron systemd: corosync.service:
>             control process
>                  > > exited, code=exited status=1
>                  > > Sep 21 17:08:21 iron systemd: Failed to start
>             Corosync Cluster
>                 Engine.
>                  > > Sep 21 17:08:21 iron systemd: Unit corosync.service
>             entered
>                 failed state.
>                  > >
>                  > >
>                  > > - on the other host:
>                  > >
>                  > > Sep 21 17:07:19 cobalt systemd: Starting Preprocess NFS
>                 configuration...
>                  > > Sep 21 17:07:19 cobalt systemd: Starting RPC Port
>             Mapper.
>                  > > Sep 21 17:07:19 cobalt systemd: Reached target RPC
>             Port Mapper.
>                  > > Sep 21 17:07:19 cobalt systemd: Starting Host and
>             Network Name
>                 Lookups.
>                  > > Sep 21 17:07:19 cobalt systemd: Reached target Host
>             and Network
>                 Name
>                  > > Lookups.
>                  > > Sep 21 17:07:19 cobalt systemd: Starting RPC bind
>             service...
>                  > > Sep 21 17:07:19 cobalt systemd: Started Preprocess NFS
>                 configuration.
>                  > > Sep 21 17:07:19 cobalt systemd: Started RPC bind
>             service.
>                  > > Sep 21 17:07:19 cobalt systemd: Starting NFS status
>             monitor for
>                  > > NFSv2/3 locking....
>                  > > Sep 21 17:07:19 cobalt rpc.statd[2662]: Version
>             1.3.0 starting
>                  > > Sep 21 17:07:19 cobalt rpc.statd[2662]: Flags: TI-RPC
>                  > > Sep 21 17:07:19 cobalt systemd: Started NFS status
>             monitor for
>                 NFSv2/3
>                  > > locking..
>                  > > Sep 21 17:07:19 cobalt systemd: Starting
>             NFS-Ganesha file server...
>                  > > Sep 21 17:07:19 cobalt systemd: Started NFS-Ganesha
>             file server.
>                  > > Sep 21 17:07:19 cobalt kernel: warning:
>             `ganesha.nfsd' uses 32-bit
>                  > > capabilities (legacy support in use)
>                  > > Sep 21 17:07:19 cobalt logger: setting up rd-ganesha-ha
>                  > > Sep 21 17:07:19 cobalt rpc.statd[2662]: Received
>             SM_UNMON_ALL
>                 request
>                  > > from cobalt.int.rdmedia.com
>             <http://cobalt.int.rdmedia.com> <http://cobalt.int.rdmedia.com>
>                 <http://cobalt.int.rdmedia.com> while not
>                  > > monitoring any hosts
>                  > > Sep 21 17:07:19 cobalt logger: setting up cluster
>             rd-ganesha-ha
>                 with
>                  > > the following cobalt iron
>                  > > Sep 21 17:07:20 cobalt systemd: Stopped Pacemaker
>             High Availability
>                  > > Cluster Manager.
>                  > > Sep 21 17:07:20 cobalt systemd: Stopped Corosync
>             Cluster Engine.
>                  > > Sep 21 17:07:20 cobalt systemd: Reloading.
>                  > > Sep 21 17:07:20 cobalt systemd:
>                  > > [/usr/lib/systemd/system/dm-event.socket:10]
>             Unknown lvalue
>                  > > 'RemoveOnStop' in section 'Socket'
>                  > > Sep 21 17:07:20 cobalt systemd:
>                  > > [/usr/lib/systemd/system/lvm2-lvmetad.socket:9]
>             Unknown lvalue
>                  > > 'RemoveOnStop' in section 'Socket'
>                  > > Sep 21 17:07:20 cobalt systemd: Reloading.
>                  > > Sep 21 17:07:20 cobalt systemd:
>                  > > [/usr/lib/systemd/system/dm-event.socket:10]
>             Unknown lvalue
>                  > > 'RemoveOnStop' in section 'Socket'
>                  > > Sep 21 17:07:20 cobalt systemd:
>                  > > [/usr/lib/systemd/system/lvm2-lvmetad.socket:9]
>             Unknown lvalue
>                  > > 'RemoveOnStop' in section 'Socket'
>                  > > Sep 21 17:07:20 cobalt systemd: Starting Corosync
>             Cluster Engine...
>                  > > Sep 21 17:07:20 cobalt corosync[2816]: [MAIN  ]
>             Corosync Cluster
>                  > > Engine ('2.3.4'): started and ready to provide service.
>                  > > Sep 21 17:07:20 cobalt corosync[2816]: [MAIN  ]
>             Corosync built-in
>                  > > features: dbus systemd xmlconf snmp pie relro bindnow
>                  > > Sep 21 17:07:20 cobalt corosync[2817]: [TOTEM ]
>             Initializing
>                 transport
>                  > > (UDP/IP Unicast).
>                  > > Sep 21 17:07:20 cobalt corosync[2817]: [TOTEM ]
>             Initializing
>                  > > transmit/receive security (NSS) crypto: none hash: none
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] The
>             network
>                 interface
>                  > > [10.100.30.37] is now up.
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync configuration map access [0]
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [QB    ]
>             server name: cmap
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync configuration service [1]
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [QB    ]
>             server name: cfg
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync cluster closed process group service v1.01 [2]
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [QB    ]
>             server name: cpg
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync profile loading service [4]
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM]
>             Using quorum
>                 provider
>                  > > corosync_votequorum
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ]
>             Waiting for all
>                  > > cluster members. Current votes: 1 expected_votes: 2
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync vote quorum service v1.0 [5]
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [QB    ]
>             server name:
>                 votequorum
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [SERV  ]
>             Service engine
>                 loaded:
>                  > > corosync cluster quorum service v0.1 [3]
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [QB    ]
>             server name: quorum
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ]
>             adding new UDPU
>                 member
>                  > > {10.100.30.37}
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ]
>             adding new UDPU
>                 member
>                  > > {10.100.30.38}
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] A
>             new membership
>                  > > (10.100.30.37:100 <http://10.100.30.37:100>
>             <http://10.100.30.37:100>
>                 <http://10.100.30.37:100>) was formed. Members joined: 1
>                 > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ]
>             Waiting for all
>                 > > cluster members. Current votes: 1 expected_votes: 2
>                 > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ]
>             Waiting for all
>                 > > cluster members. Current votes: 1 expected_votes: 2
>                 > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ]
>             Waiting for all
>                 > > cluster members. Current votes: 1 expected_votes: 2
>                 > > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM]
>             Members[1]: 1
>                 > > Sep 21 17:07:21 cobalt corosync[2817]: [MAIN  ]
>             Completed service
>                 > > synchronization, ready to provide service.
>                 > > Sep 21 17:07:21 cobalt corosync[2817]: [TOTEM ] A
>             new membership
>                  > > (10.100.30.37:108 <http://10.100.30.37:108>
>             <http://10.100.30.37:108>
>                 <http://10.100.30.37:108>) was formed. Members joined: 1
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [VOTEQ ]
>             Waiting for all
>                  > > cluster members. Current votes: 1 expected_votes: 2
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [QUORUM]
>             Members[1]: 1
>                  > > Sep 21 17:07:21 cobalt corosync[2817]: [MAIN  ]
>             Completed service
>                  > > synchronization, ready to provide service.
>                  > > Sep 21 17:08:50 cobalt systemd: corosync.service
>             operation
>                 timed out.
>                  > > Terminating.
>                  > > Sep 21 17:08:50 cobalt corosync: Starting Corosync
>             Cluster Engine
>                  > > (corosync):
>                  > > Sep 21 17:08:50 cobalt systemd: Failed to start
>             Corosync
>                 Cluster Engine.
>                  > > Sep 21 17:08:50 cobalt systemd: Unit
>             corosync.service entered
>                 failed
>                  > > state.
>                  > > Sep 21 17:08:55 cobalt logger: warning: pcs
>             property set
>                  > > no-quorum-policy=ignore failed
>                  > > Sep 21 17:08:55 cobalt logger: warning: pcs
>             property set
>                  > > stonith-enabled=false failed
>                  > > Sep 21 17:08:55 cobalt logger: warning: pcs
>             resource create
>                 nfs_start
>                  > > ganesha_nfsd
>             ha_vol_mnt=/var/run/gluster/shared_storage --clone
>                 failed
>                  > > Sep 21 17:08:56 cobalt logger: warning: pcs
>             resource delete
>                  > > nfs_start-clone failed
>                  > > Sep 21 17:08:56 cobalt logger: warning: pcs
>             resource create nfs-mon
>                  > > ganesha_mon --clone failed
>                  > > Sep 21 17:08:56 cobalt logger: warning: pcs
>             resource create
>                 nfs-grace
>                  > > ganesha_grace --clone failed
>                  > > Sep 21 17:08:57 cobalt logger: warning pcs resource
>             create
>                  > > cobalt-cluster_ip-1 ocf:heartbeat:IPaddr ip=
>             cidr_netmask=32 op
>                  > > monitor interval=15s failed
>                  > > Sep 21 17:08:57 cobalt logger: warning: pcs
>             resource create
>                  > > cobalt-trigger_ip-1 ocf:heartbeat:Dummy failed
>                  > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint
>                 colocation add
>                  > > cobalt-cluster_ip-1 with cobalt-trigger_ip-1 failed
>                  > > Sep 21 17:08:57 cobalt logger: warning: pcs
>             constraint order
>                  > > cobalt-trigger_ip-1 then nfs-grace-clone failed
>                  > > Sep 21 17:08:57 cobalt logger: warning: pcs
>             constraint order
>                  > > nfs-grace-clone then cobalt-cluster_ip-1 failed
>                  > > Sep 21 17:08:57 cobalt logger: warning pcs resource
>             create
>                  > > iron-cluster_ip-1 ocf:heartbeat:IPaddr ip=
>             cidr_netmask=32 op
>                 monitor
>                  > > interval=15s failed
>                  > > Sep 21 17:08:57 cobalt logger: warning: pcs
>             resource create
>                  > > iron-trigger_ip-1 ocf:heartbeat:Dummy failed
>                  > > Sep 21 17:08:57 cobalt logger: warning: pcs constraint
>                 colocation add
>                  > > iron-cluster_ip-1 with iron-trigger_ip-1 failed
>                  > > Sep 21 17:08:57 cobalt logger: warning: pcs
>             constraint order
>                  > > iron-trigger_ip-1 then nfs-grace-clone failed
>                  > > Sep 21 17:08:58 cobalt logger: warning: pcs
>             constraint order
>                  > > nfs-grace-clone then iron-cluster_ip-1 failed
>                  > > Sep 21 17:08:58 cobalt logger: warning: pcs
>             constraint location
>                  > > cobalt-cluster_ip-1 rule score=-INFINITY
>             ganesha-active ne 1 failed
>                  > > Sep 21 17:08:58 cobalt logger: warning: pcs
>             constraint location
>                  > > cobalt-cluster_ip-1 prefers iron=1000 failed
>                  > > Sep 21 17:08:58 cobalt logger: warning: pcs
>             constraint location
>                  > > cobalt-cluster_ip-1 prefers cobalt=2000 failed
>                  > > Sep 21 17:08:58 cobalt logger: warning: pcs
>             constraint location
>                  > > iron-cluster_ip-1 rule score=-INFINITY
>             ganesha-active ne 1 failed
>                  > > Sep 21 17:08:58 cobalt logger: warning: pcs
>             constraint location
>                  > > iron-cluster_ip-1 prefers cobalt=1000 failed
>                  > > Sep 21 17:08:58 cobalt logger: warning: pcs
>             constraint location
>                  > > iron-cluster_ip-1 prefers iron=2000 failed
>                  > > Sep 21 17:08:58 cobalt logger: warning pcs cluster
>             cib-push
>                  > > /tmp/tmp.nXTfyA1GMR failed
>                  > > Sep 21 17:08:58 cobalt logger: warning: scp
>             ganesha-ha.conf to
>                 cobalt
>                  > > failed
>                  > >
>                  > > BTW, I'm using CentOS 7. There are multiple network
>             interfaces
>                 on the
>                  > > servers, could that be a problem?
>                  > >
>                  > >
>                  > >
>                  > >
>                  > > On 21 September 2015 at 11:48, Jiffin Tony Thottan
>                  > > <jthottan at redhat.com <mailto:jthottan at redhat.com>
>             <mailto:jthottan at redhat.com <mailto:jthottan at redhat.com>>
>                 <mailto:jthottan at redhat.com <mailto:jthottan at redhat.com>
>             <mailto:jthottan at redhat.com <mailto:jthottan at redhat.com>>>>
>             wrote:
>                  > >
>                  > >
>                  > >
>                  > >     On 21/09/15 13:56, Tiemen Ruiten wrote:
>                  > >>     Hello Soumya, Kaleb, list,
>                  > >>
>                  > >>     This Friday I created the
>             gluster_shared_storage volume
>                 manually,
>                  > >>     I just tried it with the command you supplied,
>             but both
>                 have the
>                  > >>     same result:
>                  > >>
>                  > >>     from etc-glusterfs-glusterd.vol.log on the
>             node where I issued
>                  > >>     the command:
>                  > >>
>                  > >>     [2015-09-21 07:59:47.756845] I [MSGID: 106474]
>                  > >>     [glusterd-ganesha.c:403:check_host_list]
>             0-management: ganesha
>                  > >>     host found Hostname is cobalt
>                  > >>     [2015-09-21 07:59:48.071755] I [MSGID: 106474]
>                  > >>     [glusterd-ganesha.c:349:is_ganesha_host]
>             0-management: ganesha
>                  > >>     host found Hostname is cobalt
>                  > >>     [2015-09-21 07:59:48.653879] E [MSGID: 106470]
>                  > >>   
>              [glusterd-ganesha.c:264:glusterd_op_set_ganesha] 0-management:
>                  > >>     Initial NFS-Ganesha set up failed
>                  > >
>                  > >     As far as what I understand from the logs, it
>             called
>                  > >     setup_cluser()[calls `ganesha-ha.sh` script ]
>             but script
>                 failed.
>                  > >     Can u please provide following details :
>                  > >     -Location of ganesha.sh file??
>                  > >     -Location of ganesha-ha.conf, ganesha.conf files ?
>                  > >
>                  > >
>                  > >     And also can u cross check whether all the
>             prerequisites
>                 before HA
>                  > >     setup satisfied ?
>                  > >
>                  > >     --
>                  > >     With Regards,
>                  > >     Jiffin
>                  > >
>                  > >
>                  > >>     [2015-09-21 07:59:48.653912] E [MSGID: 106123]
>                  > >>     [glusterd-syncop.c:1404:gd_commit_op_phase]
>             0-management:
>                 Commit
>                  > >>     of operation 'Volume (null)' failed on
>             localhost : Failed
>                 to set
>                  > >>     up HA config for NFS-Ganesha. Please check the
>             log file
>                 for details
>                  > >>     [2015-09-21 07:59:45.402458] I [MSGID: 106006]
>                  > >>   
>              [glusterd-svc-mgmt.c:323:glusterd_svc_common_rpc_notify]
>                  > >>     0-management: nfs has disconnected from glusterd.
>                  > >>     [2015-09-21 07:59:48.071578] I [MSGID: 106474]
>                  > >>     [glusterd-ganesha.c:403:check_host_list]
>             0-management: ganesha
>                  > >>     host found Hostname is cobalt
>                  > >>
>                  > >>     from etc-glusterfs-glusterd.vol.log on the
>             other node:
>                  > >>
>                  > >>     [2015-09-21 08:12:50.111877] E [MSGID: 106062]
>                  > >>     [glusterd-op-sm.c:3698:glusterd_op_ac_unlock]
>             0-management:
>                  > >>     Unable to acquire volname
>                  > >>     [2015-09-21 08:14:50.548087] E [MSGID: 106062]
>                  > >>     [glusterd-op-sm.c:3635:glusterd_op_ac_lock]
>             0-management:
>                 Unable
>                  > >>     to acquire volname
>                  > >>     [2015-09-21 08:14:50.654746] I [MSGID: 106132]
>                  > >>     [glusterd-proc-mgmt.c:83:glusterd_proc_stop]
>             0-management: nfs
>                  > >>     already stopped
>                  > >>     [2015-09-21 08:14:50.655095] I [MSGID: 106474]
>                  > >>     [glusterd-ganesha.c:403:check_host_list]
>             0-management: ganesha
>                  > >>     host found Hostname is cobalt
>                  > >>     [2015-09-21 08:14:51.287156] E [MSGID: 106062]
>                  > >>     [glusterd-op-sm.c:3698:glusterd_op_ac_unlock]
>             0-management:
>                  > >>     Unable to acquire volname
>                  > >>
>                  > >>
>                  > >>     from etc-glusterfs-glusterd.vol.log on the
>             arbiter node:
>                  > >>
>                  > >>     [2015-09-21 08:18:50.934713] E [MSGID: 101075]
>                  > >>     [common-utils.c:3127:gf_is_local_addr]
>             0-management: error in
>                  > >>     getaddrinfo: Name or service not known
>                  > >>     [2015-09-21 08:18:51.504694] E [MSGID: 106062]
>                  > >>     [glusterd-op-sm.c:3698:glusterd_op_ac_unlock]
>             0-management:
>                  > >>     Unable to acquire volname
>                  > >>
>                  > >>     I have put the hostnames of all servers in my
>             /etc/hosts file,
>                  > >>     including the arbiter node.
>                  > >>
>                  > >>
>                  > >>     On 18 September 2015 at 16:52, Soumya Koduri
>                 <skoduri at redhat.com <mailto:skoduri at redhat.com>
>             <mailto:skoduri at redhat.com <mailto:skoduri at redhat.com>>
>                  > >>     <mailto:skoduri at redhat.com
>             <mailto:skoduri at redhat.com> <mailto:skoduri at redhat.com
>             <mailto:skoduri at redhat.com>>>>
> 
>                 wrote:
>                  > >>
>                  > >>         Hi Tiemen,
>                  > >>
>                  > >>         One of the pre-requisites before setting up
>                 nfs-ganesha HA is
>                  > >>         to create and mount shared_storage volume.
>             Use below
>                 CLI for that
>                  > >>
>                  > >>         "gluster volume set all
>             cluster.enable-shared-storage
>                 enable"
>                  > >>
>                  > >>         It shall create the volume and mount in
>             all the nodes
>                  > >>         (including the arbiter node). Note this
>             volume shall be
>                  > >>         mounted on all the nodes of the gluster
>             storage pool
>                 (though
>                  > >>         in this case it may not be part of
>             nfs-ganesha cluster).
>                  > >>
>                  > >>         So instead of manually creating those
>             directory paths,
>                 please
>                  > >>         use above CLI and try re-configuring the
>             setup.
>                  > >>
>                  > >>         Thanks,
>                  > >>         Soumya
>                  > >>
>                  > >>         On 09/18/2015 07:29 PM, Tiemen Ruiten wrote:
>                  > >>
>                  > >>             Hello Kaleb,
>                  > >>
>                  > >>             I don't:
>                  > >>
>                  > >>             # Name of the HA cluster created.
>                  > >>             # must be unique within the subnet
>                  > >>             HA_NAME="rd-ganesha-ha"
>                  > >>             #
>                  > >>             # The gluster server from which to
>             mount the
>                 shared data
>                  > >>             volume.
>                  > >>             HA_VOL_SERVER="iron"
>                  > >>             #
>                  > >>             # N.B. you may use short names or long
>             names; you
>                 may not
>                  > >>             use IP addrs.
>                  > >>             # Once you select one, stay with it as
>             it will be
>                 mildly
>                  > >>             unpleasant to
>                  > >>             # clean up if you switch later on.
>             Ensure that all
>                 names
>                  > >>             - short and/or
>                  > >>             # long - are in DNS or /etc/hosts on
>             all machines
>                 in the
>                  > >>             cluster.
>                  > >>             #
>                  > >>             # The subset of nodes of the Gluster
>             Trusted Pool that
>                  > >>             form the ganesha
>                  > >>             # HA cluster. Hostname is specified.
>                  > >>             HA_CLUSTER_NODES="cobalt,iron"
>                  > >>           
>              #HA_CLUSTER_NODES="server1.lab.redhat.com
>             <http://server1.lab.redhat.com>
>                 <http://server1.lab.redhat.com>
>                  > >>             <http://server1.lab.redhat.com>
>                  > >>
>                   <http://server1.lab.redhat.com>,server2.lab.redhat.com
>             <http://server2.lab.redhat.com>
>                 <http://server2.lab.redhat.com>
>                  > >>             <http://server2.lab.redhat.com>
>                  > >>             <http://server2.lab.redhat.com>,..."
>                  > >>             #
>                  > >>             # Virtual IPs for each of the nodes
>             specified above.
>                  > >>             VIP_server1="10.100.30.101"
>                  > >>             VIP_server2="10.100.30.102"
>                  > >>             #VIP_server1_lab_redhat_com="10.0.2.1"
>                  > >>             #VIP_server2_lab_redhat_com="10.0.2.2"
>                  > >>
>                  > >>             hosts cobalt & iron are the data
>             nodes, the arbiter
>                  > >>             ip/hostname (neon)
>                  > >>             isn't mentioned anywhere in this
>             config file.
>                  > >>
>                  > >>
>                  > >>             On 18 September 2015 at 15:56, Kaleb
>             S. KEITHLEY
>                  > >>             <kkeithle at redhat.com
>             <mailto:kkeithle at redhat.com> <mailto:kkeithle at redhat.com
>             <mailto:kkeithle at redhat.com>>
>                 <mailto:kkeithle at redhat.com <mailto:kkeithle at redhat.com>
>             <mailto:kkeithle at redhat.com <mailto:kkeithle at redhat.com>>>
>                  > >>             <mailto:kkeithle at redhat.com
>             <mailto:kkeithle at redhat.com>
>                 <mailto:kkeithle at redhat.com <mailto:kkeithle at redhat.com>>
>                  > >>             <mailto:kkeithle at redhat.com
>             <mailto:kkeithle at redhat.com>
>                 <mailto:kkeithle at redhat.com
>             <mailto:kkeithle at redhat.com>>>>> wrote:
>                  > >>
>                  > >>                 On 09/18/2015 09:46 AM, Tiemen
>             Ruiten wrote:
>                  > >>                 > Hello,
>                  > >>                 >
>                  > >>                 > I have a Gluster cluster with a
>             single
>                 replica 3,
>                  > >>             arbiter 1 volume (so
>                  > >>                 > two nodes with actual data, one
>             arbiter node). I
>                  > >>             would like to setup
>                  > >>                 > NFS-Ganesha HA for this volume
>             but I'm
>                 having some
>                  > >>             difficulties.
>                  > >>                 >
>                  > >>                 > - I needed to create a directory
>                  > >>             /var/run/gluster/shared_storage
>                  > >>                 > manually on all nodes, or the
>             command 'gluster
>                  > >>             nfs-ganesha enable would
>                  > >>                 > fail with the following error:
>                  > >>                 > [2015-09-18 13:13:34.690416] E
>             [MSGID: 106032]
>                  > >>                 >
>             [glusterd-ganesha.c:708:pre_setup] 0-THIS->name:
>                  > >>             mkdir() failed on path
>                  > >>                 >
>             /var/run/gluster/shared_storage/nfs-ganesha, [No
>                  > >>             such file or directory]
>                  > >>                 >
>                  > >>                 > - Then I found out that the
>             command connects
>                 to the
>                  > >>             arbiter node as
>                  > >>                 > well, but obviously I don't want
>             to set up
>                  > >>             NFS-Ganesha there. Is it
>                  > >>                 > actually possible to setup
>             NFS-Ganesha HA
>                 with an
>                  > >>             arbiter node? If it's
>                  > >>                 > possible, is there any
>             documentation on how
>                 to do that?
>                  > >>                 >
>                  > >>
>                  > >>                 Please send the
>             /etc/ganesha/ganesha-ha.conf file
>                  > >>             you're using.
>                  > >>
>                  > >>                 Probably you have included the
>             arbiter in your HA
>                  > >>             config; that would be
>                  > >>                 a mistake.
>                  > >>
>                  > >>                 --
>                  > >>
>                  > >>                 Kaleb
>                  > >>
>                  > >>
>                  > >>
>                  > >>
>                  > >>             --
>                  > >>             Tiemen Ruiten
>                  > >>             Systems Engineer
>                  > >>             R&D Media
>                  > >>
>                  > >>
>                  > >>           
>              _______________________________________________
>                  > >>             Gluster-users mailing list
>                  > >> Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>
>             <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>>
>                 <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>
>             <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>>>
>                 > >>http://www.gluster.org/mailman/listinfo/gluster-users
>                 > >>
>                 > >>
>                 > >>
>                 > >>
>                 > >>     --
>                 > >>     Tiemen Ruiten
>                 > >>     Systems Engineer
>                 > >>     R&D Media
>                 > >>
>                 > >>
>                 > >>     _______________________________________________
>                 > >>     Gluster-users mailing list
>                  > >> Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>
>             <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>>
>                 <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>
>             <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>>>
>                 > >>http://www.gluster.org/mailman/listinfo/gluster-users
>                 > >
>                 > >
>                 > >     _______________________________________________
>                 > >     Gluster-users mailing list
>                  > > Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>
>             <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>>
>                 <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>
>             <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>>>
> 
>                 > >http://www.gluster.org/mailman/listinfo/gluster-users
>                 > >
>                 > >
>                 > >
>                 > >
>                 > > --
>                 > > Tiemen Ruiten
>                 > > Systems Engineer
>                 > > R&D Media
>                 > >
>                 > >
>                 > >
>                 > > --
>                 > > Tiemen Ruiten
>                 > > Systems Engineer
>                 > > R&D Media
>                 > >
>                 > >
>                 > > _______________________________________________
>                 > > Gluster-users mailing list
>                 > >Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>
>             <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>>
>                 > >http://www.gluster.org/mailman/listinfo/gluster-users
>                 >
>                 >
>                 _______________________________________________
>                 Gluster-users mailing list
>                 Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>
>             <mailto:Gluster-users at gluster.org
>             <mailto:Gluster-users at gluster.org>>
>                 http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> 
>             --
>             Tiemen Ruiten
>             Systems Engineer
>             R&D Media
> 
> 
>             _______________________________________________
>             Gluster-users mailing list
>             Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>             http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 
> 
> 
>     -- 
>     Tiemen Ruiten
>     Systems Engineer
>     R&D Media
> 
> 
> 
> 
> -- 
> Tiemen Ruiten
> Systems Engineer
> R&D Media

-- 

Kaleb


More information about the Gluster-users mailing list