<div dir="ltr"><div dir="ltr">On Mon, Dec 24, 2018 at 11:30 AM Sankarshan Mukhopadhyay <<a href="mailto:sankarshan.mukhopadhyay@gmail.com">sankarshan.mukhopadhyay@gmail.com</a>> wrote:<br></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[pulling the conclusions up to enable better in-line]<br>
<br>
> Conclusions:<br>
><br>
> We should never have a volume with caching-related xlators disabled. The price we pay for it is too high. We need to make them work consistently and aggressively to avoid as many requests as we can.<br>
<br>
Are there current issues in terms of behavior which are known/observed<br>
when these are enabled?<br>
<br>
> We need to analyze client/server xlators deeper to see if we can avoid some delays. However optimizing something that is already at the microsecond level can be very hard.<br>
<br>
That is true - are there any significant gains which can be accrued by<br>
putting efforts here or, should this be a lower priority?<br></blockquote><div><br></div><div>I would say that for volumes based on spinning disks this is not a high priority, but if we want to provide good performance for NVME storage, this is something that needs to be done. On NVME, reads and writes can be served in few tens of microseconds, so adding 100 us in the network layer could easily mean a performance reduction of 70% or more.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> We need to determine what causes the fluctuations in brick side and avoid them.<br>
> This scenario is very similar to a smallfile/metadata workload, so this is probably one important cause of its bad performance.<br>
<br>
What kind of instrumentation is required to enable the determination?<br>
<br>
On Fri, Dec 21, 2018 at 1:48 PM Xavi Hernandez <<a href="mailto:xhernandez@redhat.com" target="_blank">xhernandez@redhat.com</a>> wrote:<br>
><br>
> Hi,<br>
><br>
> I've done some tracing of the latency that network layer introduces in gluster. I've made the analysis as part of the pgbench performance issue (in particulat the initialization and scaling phase), so I decided to look at READV for this particular workload, but I think the results can be extrapolated to other operations that also have small latency (cached data from FS for example).<br>
><br>
> Note that measuring latencies introduces some latency. It consists in a call to clock_get_time() for each probe point, so the real latency will be a bit lower, but still proportional to these numbers.<br>
><br>
<br>
[snip]<br>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a><br>
</blockquote></div></div>