[Gluster-devel] Bootstrapping glusterfsiostat

Tue Jun 17 22:56:04 UTC 2014

Hello,

Basing my first patch to Gluster as a stepping stone, I've written a small utility glusterfsiostat, in python which you can find attached with this email. Currently, the modifications done by my patch to io-stats which is under review as of now, dumps private information from the xlator object to the proper file for private info in the meta directory. This includes total bytes read/written along with read/write speed in the previous 10 seconds. The speed at every 1 second is identified by it's respective unix timestamp and hence given out in bytes/second. These values at discrete points of time can be used to generate a graph. 

The python tool first identifies all gluster mounts in the system, identifies the mount path and parses the meta xlator output in order to generate output similar to the iostat tool. Passing '-j' option gives you extra information in a consumable json format. By default, the tool pretty prints the basic stats which are human readable. This tool is supposed to be a framework on which other applications can be built upon. I'm putting this out for community feedback so as to improve it further.

Do you think the stats and the way they're generated as of now is something that might be usable by someone? Any other implementation suggestions or addition of some more stats that I can/should possibly provide with the utility?

Note: In order to test this, you need to apply my patch(http://review.gluster.org/#/c/8030/) in your repo first, build and then mount a volume. Preferably perform a big read/write operation with a file on your Gluster mount before executing the python script. Then run it as 'python stat.py' or 'python stat.py -j'

Regards
Vipul Nayyar 

On Sunday, 8 June 2014 8:51 PM, Vipul Nayyar <nayyar_vipul at yahoo.com> wrote:

Hi,

If I have a say in this decision, then I'd like to go with the way of keeping the io-stats xlator in it's place with the addition of dumpops. Or atleast postpone duplicating the functionality of io-stats in latency.c. I think focusing on my framework without the worry of a major upheaval in the xlator world would help me better in achieving my goals.

Regards
Vipul Nayyar 

On Saturday, 7 June 2014 1:05 AM, Anand Avati <avati at gluster.org> wrote:

On Fri, Jun 6, 2014 at 10:13 AM, Vipul Nayyar <nayyar_vipul at yahoo.com> wrote:

Hello,
>
>
>I'm Vipul and I'll be working on a tool called glusterfsiostat under GSOC this summer with KP as my mentor. Based on our discussion, the plan for the future is to build an initial working version of the tool in python and improve it later based on feedback. This tool will display i/o information about every glusterfs mount in the system.  The primary source for getting stats would be the .meta folder accessible in every mount. As of now, the meta xlator is a good resource to get the hierarchy and basic information about every translator being used in a mount. But in terms of statistics, it only shows latency of every FOP for each xlator. 
>
>
>Since, our aim is to provide information similar to the tool nfsiostat, we're hungry for more information, which is available in the private data structures maintained by io-stats. One way that I see to achieve this is to define a dumpops structure in io-stats and a priv function in it, which can be used to dump custom info into the private file in the .meta folder. If you feel that there's a better way for doing this, please do guide me.

You could do that, or enhance the latency capturing functionality with more stats, essentially making io-stats xlator redundant on the client side.

On a second note, the tool would provide stats in properly formatted manner by default for human consumption and also in json format if it is needed by any other application.

That sounds good.

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140618/7d78b46a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: stat.py
Type: text/x-python
Size: 3227 bytes
Desc: not available
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140618/7d78b46a/attachment-0001.py>