<div dir="ltr"><div><div><div><div><div>You're right. With brick graphs, this will be a problem.<br><br></div>Couple of options:<br><br></div>1. To begin with we identify points where we think it would be useful to load io-stats in the brick graph and unconditionally have glusterd-volgen load them in the volfile only at these places (not very useful if we want to load trace xl though. Plus, this again makes io-stats placement static).<br><br></div>2. Embed the trace/io-stats functionality within xlator_t object itself, and keep the accounting disabled by default. Only when required, the user can perhaps enable the accounting options with volume-set or through volume-profile start command for the brief period where they want to capture the stats and disable it as soon as they're done.<br><br></div>Let me know what you think.<br><br></div>-Krutika<br><div><div><div><div><div><div><div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 26, 2017 at 9:19 PM, Shyam <span dir="ltr"><<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On 05/26/2017 05:44 AM, Krutika Dhananjay wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
debug/io-stats and debug/trace are immensely useful for isolating<br>
translators that are performance bottlenecks and those that are causing<br>
iatt inconsistencies, respectively.<br>
<br>
There are other translators too under xlators/debug such as error-gen,<br>
which are useful for debugging/testing our code.<br>
<br>
The trick is to load these above and below one or more suspect<br>
translators, run the test and analyse the output they dump and debug<br>
your problem.<br>
<br>
Unfortunately, there is no way to load these at specific points in the<br>
graph using the volume-set CLI as of today. Our only option is to<br>
manually edit the volfile and restart the process and be super-careful<br>
not to perform *any* volume-{reset,set,profile} operation and graph<br>
switch operations in general that could rewrite the volfile, wiping out<br>
all previous edits to it.<br>
<br>
I propose the following CLI for achieving the same:<br>
<br>
# gluster volume set <VOL> {debug.trace, debug.io-stats,<br>
debug.error-gen} <xl-name><br>
<br>
where <xl-name> represents the name of the translator above which you<br>
want this translator loaded (as parent).<br>
<br>
For example, if i have a 2x2 dis-rep volume named testvol and I want to<br>
load trace above and below first child of DHT, I execute the following<br>
commands:<br>
<br>
# gluster volume set <VOL> debug.trace testvol-replicate-0<br>
# gluster volume set <VOL> debug.trace testvol-client-0<br>
# gluster volume set <VOL> debug.trace testvol-client-1<br>
<br>
The corresponding debug/trace translators will be named<br>
testvol-replicate-0-trace-pare<wbr>nt, testvol-client-0-trace-parent,<br>
testvol-client-1-trace-parent and so on.<br>
<br>
To revert the change, the user simply uses volume-reset CLI:<br>
<br>
# gluster volume reset <VOL> testvol-replicate-0-trace-pare<wbr>nt<br>
# gluster volume reset <VOL> testvol-client-0-trace-parent<br>
# gluster volume reset <VOL> testvol-client-1-trace-parent<br>
<br>
What should happen when the translator with a trace/io-stat/error-gen<br>
parent gets disabled?<br>
Well glusterd should be made to take care to remove the trace xl too<br>
from the graph.<br>
<br>
<br>
<br>
Comments and suggestions welcome.<br>
</blockquote>
<br></div></div>
+1, dynamic placement of io-stats was something that I added to this spec [1] as well. So I am all for the change.<br>
<br>
I have one problem though that bothered me when I wrote the spec, currently brick vol files are static, and do not undergo a graph change (or code is not yet ready to do that). So when we want to do this on the bricks, what happens? Do you have solutions for the same? I am interested, hence asking!<br>
<br>
[1] Initial feature description for improved io-stats: <a href="https://review.gluster.org/#/c/16558/1/under_review/Performance_monitoring_and_debugging.md" rel="noreferrer" target="_blank">https://review.gluster.org/#/c<wbr>/16558/1/under_review/Performa<wbr>nce_monitoring_and_debugging.<wbr>md</a><br>
</blockquote></div><br></div></div></div></div></div></div></div></div></div></div></div>