[Gluster-devel] Volgen support for loading trace and io-stats translators at specific points in the graph

Shyam srangana at redhat.com
Fri May 26 15:49:20 UTC 2017

On 05/26/2017 05:44 AM, Krutika Dhananjay wrote:
> Hi,
> debug/io-stats and debug/trace are immensely useful for isolating
> translators that are performance bottlenecks and those that are causing
> iatt inconsistencies, respectively.
> There are other translators too under xlators/debug such as error-gen,
> which are useful for debugging/testing our code.
> The trick is to load these above and below one or more suspect
> translators, run the test and analyse the output they dump and debug
> your problem.
> Unfortunately, there is no way to load these at specific points in the
> graph using the volume-set CLI as of today. Our only option is to
> manually edit the volfile and restart the process and be super-careful
> not to perform *any* volume-{reset,set,profile} operation and graph
> switch operations in general that could rewrite the volfile, wiping out
> all previous edits to it.
> I propose the following CLI for achieving the same:
> # gluster volume set <VOL> {debug.trace, debug.io-stats,
> debug.error-gen} <xl-name>
> where <xl-name> represents the name of the translator above which you
> want this translator loaded (as parent).
> For example, if i have a 2x2 dis-rep volume named testvol and I want to
> load trace above and below first child of DHT, I execute the following
> commands:
> # gluster volume set <VOL> debug.trace testvol-replicate-0
> # gluster volume set <VOL> debug.trace testvol-client-0
> # gluster volume set <VOL> debug.trace testvol-client-1
> The corresponding debug/trace translators will be named
> testvol-replicate-0-trace-parent, testvol-client-0-trace-parent,
> testvol-client-1-trace-parent and so on.
> To revert the change, the user simply uses volume-reset CLI:
> # gluster volume reset <VOL> testvol-replicate-0-trace-parent
> # gluster volume reset <VOL> testvol-client-0-trace-parent
> # gluster volume reset <VOL> testvol-client-1-trace-parent
> What should happen when the translator with a trace/io-stat/error-gen
> parent gets disabled?
> Well glusterd should be made to take care to remove the trace xl too
> from the graph.
> Comments and suggestions welcome.

+1, dynamic placement of io-stats was something that I added to this 
spec [1] as well. So I am all for the change.

I have one problem though that bothered me when I wrote the spec, 
currently brick vol files are static, and do not undergo a graph change 
(or code is not yet ready to do that). So when we want to do this on the 
bricks, what happens? Do you have solutions for the same? I am 
interested, hence asking!

[1] Initial feature description for improved io-stats: 

More information about the Gluster-devel mailing list