[Gluster-devel] Volgen support for loading trace and io-stats translators at specific points in the graph

[Shyam]

On 05/26/2017 05:44 AM, Krutika Dhananjay wrote:
> Hi,
>
> debug/io-stats and debug/trace are immensely useful for isolating
> translators that are performance bottlenecks and those that are causing
> iatt inconsistencies, respectively.
>
> There are other translators too under xlators/debug such as error-gen,
> which are useful for debugging/testing our code.
>
> The trick is to load these above and below one or more suspect
> translators, run the test and analyse the output they dump and debug
> your problem.
>
> Unfortunately, there is no way to load these at specific points in the
> graph using the volume-set CLI as of today. Our only option is to
> manually edit the volfile and restart the process and be super-careful
> not to perform *any* volume-{reset,set,profile} operation and graph
> switch operations in general that could rewrite the volfile, wiping out
> all previous edits to it.
>
> I propose the following CLI for achieving the same:
>
> # gluster volume set <VOL> {debug.trace, debug.io-stats,
> debug.error-gen} <xl-name>
>
> where <xl-name> represents the name of the translator above which you
> want this translator loaded (as parent).
>
> For example, if i have a 2x2 dis-rep volume named testvol and I want to
> load trace above and below first child of DHT, I execute the following
> commands:
>
> # gluster volume set <VOL> debug.trace testvol-replicate-0
> # gluster volume set <VOL> debug.trace testvol-client-0
> # gluster volume set <VOL> debug.trace testvol-client-1
>
> The corresponding debug/trace translators will be named
> testvol-replicate-0-trace-parent, testvol-client-0-trace-parent,
> testvol-client-1-trace-parent and so on.
>
> To revert the change, the user simply uses volume-reset CLI:
>
> # gluster volume reset <VOL> testvol-replicate-0-trace-parent
> # gluster volume reset <VOL> testvol-client-0-trace-parent
> # gluster volume reset <VOL> testvol-client-1-trace-parent
>
> What should happen when the translator with a trace/io-stat/error-gen
> parent gets disabled?
> Well glusterd should be made to take care to remove the trace xl too
> from the graph.
>
>
>
> Comments and suggestions welcome.

+1, dynamic placement of io-stats was something that I added to this 
spec [1] as well. So I am all for the change.

I have one problem though that bothered me when I wrote the spec, 
currently brick vol files are static, and do not undergo a graph change 
(or code is not yet ready to do that). So when we want to do this on the 
bricks, what happens? Do you have solutions for the same? I am 
interested, hence asking!

[1] Initial feature description for improved io-stats: 
https://review.gluster.org/#/c/16558/1/under_review/Performance_monitoring_and_debugging.md