[Gluster-devel] New project on the Forge - gstatus

[Kaushal M]

Avati that's awesome! I didn't know that the meta xlator could do that
already.


On Sat, May 17, 2014 at 9:30 AM, Anand Avati <avati at gluster.org> wrote:

> KP, Vipul,
>
> It will be awesome to get io-stats like instrumentation on the client
> side. Here are some further thoughts on how to implement that. If you have
> a recent git HEAD build, I would suggest that you explore the latency stats
> on the client side exposed through meta at
> $MNT/.meta/graphs/active/$xlator/profile. You can enable latency
> measurement with "echo 1 > $MNT/.meta/measure_latency". I would suggest
> extending these stats with the extra ones io-stats has, and make
> glusterfsiostats expose these stats.
>
> If you can compare libglusterfs/src/latency.c:gf_latency_begin(),
> gf_latency_end() and gf_latency_udpate() with the macros in io-stats.c UPDATE_PROFILE_STATS()
> and START_FOP_LATENCY(), you will quickly realize how a lot of logic is
> duplicated between io-stats and latency.c. If you can enhance latency.c and
> make it capture the remaining stats what io-stats is capturing, the
> benefits of this approach would be:
>
> - stats are already getting captured at all xlator levels, and not just at
> the position where io-stats is inserted
> - file like interface makes the stats more easily inspectable and
> consumable, and updated on the fly
> - conforms with the way rest of the internals are exposed through
> $MNT/.meta
>
> In order to this, you might want to look into:
>
> - latency.c as of today captures fop count, mean latency, total time,
> whereas io-stats measures these along with min-time, max-time and
> block-size histogram.
> - extend gf_proc_dump_latency_info() to dump the new stats
> - either prettify that output like 'volume profile info' output, or
> JSONify it like xlators/meta/src/frames-file.c
> - add support for cumulative vs interval stats (store an extra copy of
> this->latencies[])
>
> etc..
>
> Thanks!
>
>
> On Fri, Apr 25, 2014 at 9:09 PM, Krishnan Parthasarathi <
> kparthas at redhat.com> wrote:
>
>> [Resending due to gluster-devel mailing list issue]
>>
>> Apologies for the late reply.
>>
>> glusterd uses its socket connection with brick processes (where io-stats
>> xlator is loaded) to
>> gather information from io-stats via an RPC request. This facility is
>> restricted to brick processes
>> as it stands today.
>>
>> Some background ...
>> io-stats xlator is loaded, both in GlusterFS mounts and brick processes.
>> So, we have the capabilities
>> to monitor I/O statistics on both sides. To collect I/O statistics at the
>> server side, we have
>>
>> # gluster volume profile VOLNAME [start | info | stop]
>> AND
>> #gluster volume top VOLNAME info [and other options]
>>
>> We don't have a usable way of gathering I/O statistics (not monitoring,
>> though the counters could be enhanced)
>> at the client-side, ie. for a given mount point. This is the gap
>> glusterfsiostat aims to fill. We need to remember
>> that the machines hosting GlusterFS mounts may not have glusterd
>> installed on them.
>>
>> We are considering rrdtool as a possible statistics database because it
>> seems like a natural choice for storing time-series
>> data. rrdtool is capable of answering high-level statistical queries on
>> statistics that were logged in it by io-stats xlator
>> over and above printing running counters periodically.
>>
>> Hope this gives some more clarity on what we are thinking.
>>
>> thanks,
>> Krish
>> ----- Original Message -----
>>
>> > Probably me not understanding.
>>
>> > the comment "iostats making data available to glusterd over RPC" - is
>> what I
>> > latched on to. I wondered whether this meant that a socket could be
>> opened
>> > that way to get at the iostats data flow.
>>
>> > Cheers,
>>
>> > PC
>>
>> > ----- Original Message -----
>>
>> > > From: "Vipul Nayyar" <nayyar_vipul at yahoo.com>
>> >
>> > > To: "Paul Cuzner" <pcuzner at redhat.com>, "Krishnan Parthasarathi"
>> > > <kparthas at redhat.com>
>> >
>> > > Cc: "Vijay Bellur" <vbellur at redhat.com>, "gluster-devel"
>> > > <gluster-devel at nongnu.org>
>> >
>> > > Sent: Thursday, 20 February, 2014 5:06:27 AM
>> >
>> > > Subject: Re: [Gluster-devel] New project on the Forge - gstatus
>> >
>>
>> > > Hi Paul,
>> >
>>
>> > > I'm really not sure, if this can be done in python(at least
>> comfortably).
>> > > Maybe we can tread on the same path as Justin's glusterflow in
>> python. But
>> > > I
>> > > don't think, all the io-stats counters will be available with the way
>> how
>> > > Justin's used Jeff Darcy's previous work to build his tool. I can be
>> wrong.
>> > > My knowledge is a bit incomplete and based on very less experience as
>> a
>> > > user
>> > > and an amateur Gluster developer. Please do correct me, if I can be.
>> >
>>
>> > > Regards
>> >
>> > > Vipul Nayyar
>> >
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140517/9e17301d/attachment-0002.html>