[Gluster-devel] Volgen support for loading trace and io-stats translators at specific points in the graph

[Krutika Dhananjay]

You're right. With brick graphs, this will be a problem.

Couple of options:

1. To begin with we identify points where we think it would be useful to
load io-stats in the brick graph and unconditionally have glusterd-volgen
load them in the volfile only at these places (not very useful if we want
to load trace xl though. Plus, this again makes io-stats placement static).

2. Embed the trace/io-stats functionality within xlator_t object itself,
and keep the accounting disabled by default. Only when required, the user
can perhaps enable the accounting options with volume-set or through
volume-profile start command for the brief period where they want to
capture the stats and disable it as soon as they're done.

Let me know what you think.

-Krutika

On Fri, May 26, 2017 at 9:19 PM, Shyam <srangana at redhat.com> wrote:

> On 05/26/2017 05:44 AM, Krutika Dhananjay wrote:
>
>> Hi,
>>
>> debug/io-stats and debug/trace are immensely useful for isolating
>> translators that are performance bottlenecks and those that are causing
>> iatt inconsistencies, respectively.
>>
>> There are other translators too under xlators/debug such as error-gen,
>> which are useful for debugging/testing our code.
>>
>> The trick is to load these above and below one or more suspect
>> translators, run the test and analyse the output they dump and debug
>> your problem.
>>
>> Unfortunately, there is no way to load these at specific points in the
>> graph using the volume-set CLI as of today. Our only option is to
>> manually edit the volfile and restart the process and be super-careful
>> not to perform *any* volume-{reset,set,profile} operation and graph
>> switch operations in general that could rewrite the volfile, wiping out
>> all previous edits to it.
>>
>> I propose the following CLI for achieving the same:
>>
>> # gluster volume set <VOL> {debug.trace, debug.io-stats,
>> debug.error-gen} <xl-name>
>>
>> where <xl-name> represents the name of the translator above which you
>> want this translator loaded (as parent).
>>
>> For example, if i have a 2x2 dis-rep volume named testvol and I want to
>> load trace above and below first child of DHT, I execute the following
>> commands:
>>
>> # gluster volume set <VOL> debug.trace testvol-replicate-0
>> # gluster volume set <VOL> debug.trace testvol-client-0
>> # gluster volume set <VOL> debug.trace testvol-client-1
>>
>> The corresponding debug/trace translators will be named
>> testvol-replicate-0-trace-parent, testvol-client-0-trace-parent,
>> testvol-client-1-trace-parent and so on.
>>
>> To revert the change, the user simply uses volume-reset CLI:
>>
>> # gluster volume reset <VOL> testvol-replicate-0-trace-parent
>> # gluster volume reset <VOL> testvol-client-0-trace-parent
>> # gluster volume reset <VOL> testvol-client-1-trace-parent
>>
>> What should happen when the translator with a trace/io-stat/error-gen
>> parent gets disabled?
>> Well glusterd should be made to take care to remove the trace xl too
>> from the graph.
>>
>>
>>
>> Comments and suggestions welcome.
>>
>
> +1, dynamic placement of io-stats was something that I added to this spec
> [1] as well. So I am all for the change.
>
> I have one problem though that bothered me when I wrote the spec,
> currently brick vol files are static, and do not undergo a graph change (or
> code is not yet ready to do that). So when we want to do this on the
> bricks, what happens? Do you have solutions for the same? I am interested,
> hence asking!
>
> [1] Initial feature description for improved io-stats:
> https://review.gluster.org/#/c/16558/1/under_review/Performa
> nce_monitoring_and_debugging.md
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20170530/0d2ab479/attachment-0001.html>