[Gluster-users] Monitor replicated Gluster setup ?

[Brian Candler]

On Wed, Jul 11, 2012 at 03:06:19PM +0000, Stefan Schloesser wrote:
>    My Idea is to use GlusterFS as a replicated filesystem (for apache) and
>    built-in mysql replication for the database. In the event of a failure
>    I need to run a script to implement an ip switch.

To switch IP for what? glusterfs in replicated mode, using the native (FUSE)
client, doesn't need this.  The client talks to both backends, and if either
backend fails, it continues to work.

>    How should I monitor such a system?

I'm not sure if there's a proper API. As a starting point, try running
'gluster volume status' as root and parsing the results. e.g. here the bricks
on one server are unavailable:

$ sudo gluster volume status
Status of volume: safe
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick dev-storage1:/disk/storage1/safe			24009	N	N/A
Brick dev-storage2:/disk/storage2/safe			24009	Y	1710
NFS Server on localhost					38467	Y	2034
Self-heal Daemon on localhost				N/A	Y	1736
NFS Server on 10.0.1.1					38467	Y	1631
Self-heal Daemon on 10.0.1.1				N/A	Y	1637
 
Status of volume: fast
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick dev-storage1:/disk/storage1/fast			24010	N	N/A
Brick dev-storage2:/disk/storage2/fast			24010	Y	1720
NFS Server on localhost					38467	Y	2034
NFS Server on 10.0.1.1					38467	Y	1631
 
Status of volume: single1
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick dev-storage1:/disk/storage1/single1		24011	N	N/A
NFS Server on localhost					38467	Y	2034
NFS Server on 10.0.1.1					38467	Y	1631
 
Status of volume: single2
Gluster process						Port	Online	Pid
------------------------------------------------------------------------------
Brick dev-storage2:/disk/storage2/single2		24011	Y	2028
NFS Server on localhost					38467	Y	2034
NFS Server on 10.0.1.1					38467	Y	1631

It would be neat if this could be integrated into SNMP. There are some good
tools I found which provide a framework to integrate mdraid and smartctl
status:

http://www.mad-hacking.net/software/index.xml
http://downloads.mad-hacking.net/software/

It should be relatively straightforward to add gluster into this.

>    Does Gluster provide an integration
>    with corosync as a resource?

Not that I'm aware of, as far as I can see it's not needed. Volume status
information is synchronised between the peers in the cluster using some
internal protocol, and I'm not exactly sure how it deals with split brain
scenarios.

Another project which *does* use corosync is sheepdog. This is completely
different to glusterfs though - it's a distributed block level store for
KVM.

>    I guess it is not a good idea to keep the mysql data files on the
>    glustered fs, or is that feasible?

I'd say that's definitely a bad idea, especially if you had two different
mysqld's talking to the same storage.

Regards,

Brian.