[Gluster-users] Subject: Help needed in improving monitoring in Gluster

[Maarten van Baarsel]

On 23/07/18 16:03, Pranith Kumar Karampuri wrote:

> We want gluster's monitoring/observability to be as easy as possible 
> going forward. As part of reaching this goal we are starting this 
> initiative to add improvements to existing apis/commands and create new 
> apis/commands to gluster so that the admin can integrate it with 
> whichever monitoring tool he/she likes. The gluster-prometheus project 
> hosted at https://github.com/gluster/gluster-prometheus is the direction 
> in which we feel metrics can be collected and distributed from a Gluster 
> cluster enabling analysis and visualization.
> 
> 
> As a first step we want to hear from you what you feel needs to be 
> addressed.

Regarding monitoring; I would love to see in my monitoring that 
geo-replication is working as intended; at the moment I'm faking georep 
monitoring by having a process touch a file (every server involved in 
gluster touches another file) on every volume and checking mtime on the 
slave.

However, I discovered that this is not foolproof: if the georep run 
stops for whatever reason the mtime of the monitored file is being kept
updated, probably because it's updated to often, but the georep is not 
complete.

I've also seen that a crashed glusterd escapes this monitoring.



What would also be fun is some kind of monitoring where you can find out 
why gluster is running at X MB/sec where Y MB/sec is expected (bit large 
target, that)



I've once tried monitoring 'gluster volume status all' output but that 
only works if everything is OK; with some network problems you can wait 
for hours for output which then causes more problems.



Also, I've checked the example output at 
https://github.com/gluster/gluster-prometheus:

would JSON or something like that be more friendy to parse instead of 
the "[parameter] { [details] } [number]" format?

thanks,
Maarten.