<div dir="ltr"><div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr">On Tue, Feb 5, 2019, 10:53 PM Xavi Hernandez <<a href="mailto:xhernandez@redhat.com" rel="noreferrer" target="_blank">xhernandez@redhat.com</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Feb 1, 2019 at 1:51 PM Xavi Hernandez <<a href="mailto:xhernandez@redhat.com" rel="noreferrer noreferrer" target="_blank">xhernandez@redhat.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">On Fri, Feb 1, 2019 at 1:25 PM Poornima Gurusiddaiah <<a href="mailto:pgurusid@redhat.com" rel="noreferrer noreferrer" target="_blank">pgurusid@redhat.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Can the threads be categorised to do certain kinds of fops?</div></blockquote><div><br></div><div>Could be, but creating multiple thread groups for different tasks is generally bad because many times you end up with lots of idle threads which waste resources and could increase contention. I think we should only differentiate threads if it's absolutely necessary.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"> Read/write affinitise to certain set of threads, the other metadata fops to other set of threads. So we limit the read/write threads and not the metadata threads? Also if aio is enabled in the backend the threads will not be blocked on disk IO right? </div></blockquote><div><br></div><div>If we don't block the thread but we don't prevent more requests to go to the disk, then we'll probably have the same problem. Anyway, I'll try to run some tests with AIO to see if anything changes.</div></div></div></blockquote><div><br></div><div>I've run some simple tests with AIO enabled and results are not good. A simple dd takes >25% more time. Multiple parallel dd take 35% more time to complete.</div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto"><br></div><div>Thank you. That is strange! Had few questions, what tests are you running for measuring the io-threads performance(not particularly aoi)? is it dd from multiple clients?</div><div><br></div><div>Regards,</div><div>Poornima<br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><div>Xavi</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto">All this is based on the assumption that large number of parallel read writes make the disk perf bad but not the large number of dentry and metadata ops. Is that true?</div></div></blockquote><div><br></div><div>It depends. If metadata is not cached, it's as bad as a read or write since it requires a disk access (a clear example of this is the bad performance of 'ls' in cold cache, which is basically metadata reads). In fact, cached data reads are also very fast, and data writes could go to the cache and be updated later in background, so I think the important point is if things are cached or not, instead of if they are data or metadata. Since we don't have this information from the user side, it's hard to tell what's better. My opinion is that we shouldn't differentiate requests of data/metadata. If metadata requests happen to be faster, then that thread will be able to handle other requests immediately, which seems good enough.</div><div><br></div><div>However there's one thing that I would do. I would differentiate reads (data or metadata) from writes. Normally writes come from cached information that is flushed to disk at some point, so this normally happens in the background. But reads tend to be in foreground, meaning that someone (user or application) is waiting for it. So I would give preference to reads over writes. To do so effectively, we need to not saturate the backend, otherwise when we need to send a read, it will still need to wait for all pending requests to complete. If disks are not saturated, we can have the answer to the read quite fast, and then continue processing the remaining writes.</div><div><br></div><div>Anyway, I may be wrong, since all these things depend on too many factors. I haven't done any specific tests about this. It's more like a brainstorming. As soon as I can I would like to experiment with this and get some empirical data.</div><div><br></div><div>Xavi</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div dir="auto"><br></div><div dir="auto">Thanks,</div><div dir="auto">Poornima</div><br><br><div class="gmail_quote" dir="auto"><div dir="ltr">On Fri, Feb 1, 2019, 5:34 PM Emmanuel Dreyfus <<a href="mailto:manu@netbsd.org" rel="noreferrer noreferrer" target="_blank">manu@netbsd.org</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, Jan 31, 2019 at 10:53:48PM -0800, Vijay Bellur wrote:<br>
> Perhaps we could throttle both aspects - number of I/O requests per disk<br>
<br>
While there it would be nice to detect and report a disk with lower than<br>
peer performance: that happen sometimes when a disk is dying, and last<br>
time I was hit by that performance problem, I had a hard time finding<br>
the culprit.<br>
<br>
-- <br>
Emmanuel Dreyfus<br>
<a href="mailto:manu@netbsd.org" rel="noreferrer noreferrer noreferrer" target="_blank">manu@netbsd.org</a><br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" rel="noreferrer noreferrer noreferrer" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer noreferrer noreferrer noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div></div>
</blockquote></div></div>
</blockquote></div></div>
</blockquote></div></div></div>
</div>