<div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr">On Fri 28 Dec, 2018, 12:44 Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com">rgowdapp@redhat.com</a> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Dec 24, 2018 at 6:05 PM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank" rel="noreferrer">rgowdapp@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Dec 24, 2018 at 3:40 PM Sankarshan Mukhopadhyay <<a href="mailto:sankarshan.mukhopadhyay@gmail.com" target="_blank" rel="noreferrer">sankarshan.mukhopadhyay@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[pulling the conclusions up to enable better in-line]<br>
<br>
> Conclusions:<br>
><br>
> We should never have a volume with caching-related xlators disabled. The price we pay for it is too high. We need to make them work consistently and aggressively to avoid as many requests as we can.<br>
<br>
Are there current issues in terms of behavior which are known/observed<br>
when these are enabled?<br></blockquote><div><br></div><div>We did have issues with pgbench in past. But they've have been fixed. Please refer to bz [1] for details. On 5.1, it runs successfully with all caching related xlators enabled. Having said that the only performance xlators which gave improved performance were open-behind and write-behind [2] (write-behind had some issues, which will be fixed by [3] and we'll have to measure performance again with fix to [3]). </div></div></div></div></div></div></div></blockquote><div><br></div><div>One quick update. Enabling write-behind and md-cache with fix for [3] reduced the total time taken for pgbench init phase roughly by 20%-25% (from 12.5 min to 9.75 min for a scale of 100). Though this is still a huge time (around 12hrs for a db of scale 8000). I'll follow up with a detailed report once my experiments are complete. Currently trying to optimize the read path.<br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div>For some reason, read-side caching didn't improve transactions per second. I am working on this problem currently. Note that these bugs measure transaction phase of pgbench, but what xavi measured in his mail is init phase. Nevertheless, evaluation of read caching (metadata/data) will still be relevant for init phase too.<br><br>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1512691" target="_blank" rel="noreferrer">https://bugzilla.redhat.com/show_bug.cgi?id=1512691</a><br>[2] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1629589#c4" target="_blank" rel="noreferrer">https://bugzilla.redhat.com/show_bug.cgi?id=1629589#c4</a><br>[3] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1648781" target="_blank" rel="noreferrer">https://bugzilla.redhat.com/show_bug.cgi?id=1648781</a></div></div></div></div></div></div></div></blockquote></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">I think that what I am looking forward to is a well defined set of next steps and potential update to this list windows that eventually result in a formal and recorded procedure to ensure that Gluster performs best for these application workloads.</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div class="gmail_quote"><div><br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> We need to analyze client/server xlators deeper to see if we can avoid some delays. However optimizing something that is already at the microsecond level can be very hard.<br>
<br>
That is true - are there any significant gains which can be accrued by<br>
putting efforts here or, should this be a lower priority?<br></blockquote><div><br></div><div>The problem identified by xavi is also the one we (Manoj, Krutika, me and Milind) had encountered in the past [4]. The solution we used was to have multiple rpc connections between single brick and client. The solution indeed fixed the bottleneck. So, there is definitely work involved here - either to fix the single connection model or go with multiple connection model. Its preferred to improve single connection and resort to multiple connections only if bottlenecks in single connection are not fixable. Personally I think this is high priority along with having appropriate client side caching.<br><br>[4] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1467614#c52" target="_blank" rel="noreferrer">https://bugzilla.redhat.com/show_bug.cgi?id=1467614#c52</a><br><br> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> We need to determine what causes the fluctuations in brick side and avoid them.<br>
> This scenario is very similar to a smallfile/metadata workload, so this is probably one important cause of its bad performance.<br>
<br>
What kind of instrumentation is required to enable the determination?<br>
<br>
On Fri, Dec 21, 2018 at 1:48 PM Xavi Hernandez <<a href="mailto:xhernandez@redhat.com" target="_blank" rel="noreferrer">xhernandez@redhat.com</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> I've done some tracing of the latency that network layer introduces in gluster. I've made the analysis as part of the pgbench performance issue (in particulat the initialization and scaling phase), so I decided to look at READV for this particular workload, but I think the results can be extrapolated to other operations that also have small latency (cached data from FS for example).<br>
><br>
> Note that measuring latencies introduces some latency. It consists in a call to clock_get_time() for each probe point, so the real latency will be a bit lower, but still proportional to these numbers.<br>
><br>
<br>
[snip]<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank" rel="noreferrer">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div></div></div>