<div dir="ltr"><div><div><div><div><div>I just realised there was already some effort in 2010 to have per-xl latency measurement.<br></div>See commit d90c2f86ca7a8f6660e98da1e48c4798539b7d51.<br><br></div>Turns out you need to send SIGUSR2 to the process to enable it.<br></div>And the stats are collected through statedump.<br><br></div>Haven't yet tested it. Will try and get back.<br></div><div><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 31, 2017 at 3:08 PM, Krutika Dhananjay <span dir="ltr"><<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Tue, May 30, 2017 at 6:42 PM, Shyam <span dir="ltr"><<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a>></span> wrote:<br></span><span class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On 05/30/2017 05:28 AM, Krutika Dhananjay wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
You're right. With brick graphs, this will be a problem.<br>
<br>
Couple of options:<br>
<br>
1. To begin with we identify points where we think it would be useful to<br>
load io-stats in the brick graph and unconditionally have<br>
glusterd-volgen load them in the volfile only at these places (not very<br>
useful if we want to load trace xl though. Plus, this again makes<br>
io-stats placement static).<br>
</blockquote>
<br></span>
I think this is needed (easier to get in), so +1 for this.<br>
<br>
Additionally, if this is chosen, we may need specific triggers for each instance, to target measuring the io-stats. IOW, generic io-stats can measure below</blockquote><div><br></div></span><div>I tried this recently with existing code by configuring the stats-dump-interval and fop-sample-interval options, and each instance dumps its stats into a file under /var/lib/glusterd/stats. There's one file per io-stats xl. The downside is that with each interval, the stats from the prev interval gets overwritten. I'm planning to change this by adding a timestamp + pid suffix to the dump file name.<br><br></div><span class=""><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> FUSE (as an example) and below server-protocol. Then, we may want to enable io-threads (assuming this is one instance on the brick that is a static placement), or POSIX (or both/all) specifically, than have them enabled by default when io-stats is turned on (which is the current behaviour).<br>
<br></blockquote><div><br></div></span><div>I didn't follow the io-threads and posix part. Could you rephrase?<br></div><div>I'm thinking of changing code that would make io-stats be loaded above and below io-threads and also above posix. Not sure if you meant the same thing in your statement above ;) Would that be fine? Or is there value in loading it elsewhere?<span class="HOEnZb"><font color="#888888"><br><br></font></span></div><span class="HOEnZb"><font color="#888888"><div>-Krutika<br> <br></div></font></span><div><div class="h5"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Does this make sense?<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
2. Embed the trace/io-stats functionality within xlator_t object itself,<br>
and keep the accounting disabled by default. Only when required, the<br>
user can perhaps enable the accounting options with volume-set or<br>
through volume-profile start command for the brief period where they<br>
want to capture the stats and disable it as soon as they're done.<br>
</blockquote>
<br></span>
This is a better longer term solution IMO. This way there is no further injection of io-stats xltor, and we get a lot more control on this better.<br>
<br>
Depending on time to completion, I would choose 1/2 as presented above. This is because, I see a lot of value in this and in answering user queries on what is slowing down their systems, so sooner we have this the better (say 3.12), if (2) is possible by then, more power to it.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
<br>
Let me know what you think.<br>
<br>
-Krutika<br>
<br>
On Fri, May 26, 2017 at 9:19 PM, Shyam <<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a><br></span><div><div class="m_6631966888303058142h5">
<mailto:<a href="mailto:srangana@redhat.com" target="_blank">srangana@redhat.com</a>>> wrote:<br>
<br>
On 05/26/2017 05:44 AM, Krutika Dhananjay wrote:<br>
<br>
Hi,<br>
<br>
debug/io-stats and debug/trace are immensely useful for isolating<br>
translators that are performance bottlenecks and those that are<br>
causing<br>
iatt inconsistencies, respectively.<br>
<br>
There are other translators too under xlators/debug such as<br>
error-gen,<br>
which are useful for debugging/testing our code.<br>
<br>
The trick is to load these above and below one or more suspect<br>
translators, run the test and analyse the output they dump and debug<br>
your problem.<br>
<br>
Unfortunately, there is no way to load these at specific points<br>
in the<br>
graph using the volume-set CLI as of today. Our only option is to<br>
manually edit the volfile and restart the process and be<br>
super-careful<br>
not to perform *any* volume-{reset,set,profile} operation and graph<br>
switch operations in general that could rewrite the volfile,<br>
wiping out<br>
all previous edits to it.<br>
<br>
I propose the following CLI for achieving the same:<br>
<br>
# gluster volume set <VOL> {debug.trace, debug.io-stats,<br>
debug.error-gen} <xl-name><br>
<br>
where <xl-name> represents the name of the translator above<br>
which you<br>
want this translator loaded (as parent).<br>
<br>
For example, if i have a 2x2 dis-rep volume named testvol and I<br>
want to<br>
load trace above and below first child of DHT, I execute the<br>
following<br>
commands:<br>
<br>
# gluster volume set <VOL> debug.trace testvol-replicate-0<br>
# gluster volume set <VOL> debug.trace testvol-client-0<br>
# gluster volume set <VOL> debug.trace testvol-client-1<br>
<br>
The corresponding debug/trace translators will be named<br>
testvol-replicate-0-trace-pare<wbr>nt, testvol-client-0-trace-parent,<br>
testvol-client-1-trace-parent and so on.<br>
<br>
To revert the change, the user simply uses volume-reset CLI:<br>
<br>
# gluster volume reset <VOL> testvol-replicate-0-trace-pare<wbr>nt<br>
# gluster volume reset <VOL> testvol-client-0-trace-parent<br>
# gluster volume reset <VOL> testvol-client-1-trace-parent<br>
<br>
What should happen when the translator with a<br>
trace/io-stat/error-gen<br>
parent gets disabled?<br>
Well glusterd should be made to take care to remove the trace xl too<br>
from the graph.<br>
<br>
<br>
<br>
Comments and suggestions welcome.<br>
<br>
<br>
+1, dynamic placement of io-stats was something that I added to this<br>
spec [1] as well. So I am all for the change.<br>
<br>
I have one problem though that bothered me when I wrote the spec,<br>
currently brick vol files are static, and do not undergo a graph<br>
change (or code is not yet ready to do that). So when we want to do<br>
this on the bricks, what happens? Do you have solutions for the<br>
same? I am interested, hence asking!<br>
<br>
[1] Initial feature description for improved io-stats:<br>
<a href="https://review.gluster.org/#/c/16558/1/under_review/Performance_monitoring_and_debugging.md" rel="noreferrer" target="_blank">https://review.gluster.org/#/c<wbr>/16558/1/under_review/Performa<wbr>nce_monitoring_and_debugging.m<wbr>d</a><br></div></div>
<<a href="https://review.gluster.org/#/c/16558/1/under_review/Performance_monitoring_and_debugging.md" rel="noreferrer" target="_blank">https://review.gluster.org/#/<wbr>c/16558/1/under_review/Perform<wbr>ance_monitoring_and_debugging.<wbr>md</a>><br>
<br>
<br>
</blockquote>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br></div>