[Gluster-devel] Performance Translators' Stability and Usefulness

[Shehjar Tikoo]

Gordan Bobic wrote:
> Just reading through the wiki on this and a few things are unclear, 
> so I'm hoping someone can clarify.
> 
> 1) readahead
> 
> - Is there any point in using this on systems where the interconnect 
> <= 1Gb/s? The wiki implies there is no point in this, but doesn't 
> quite state it explicitly.
I am pretty sure it helps. The question of using read-ahead is more of a
question related to the workload rather than the interconnect, for eg.
it'll be useful for sequential reading, without any doubts.
Of course, there can be cases where excessive read-ahead chokes the
100 Mib/s link, but then read-ahead can be configured to reduce its
utilization of the network by reducing the page-count option.

> 
> - Is there any point in using this on a server that is also it's own 
> client when use with replicate/afr? I'm guessing there isn't since 
> the local fs will be doing it's own read-ahead but I'd like some 
> confirmation on that.
No. Generally, read-ahead will be most beneficial only on the client
side since it helps avoid the need to go to the network when an
application does need the data already read-ahead. Yes, on the server
side, on-disk file systems read-ahead already does it best.

In your setup above, in case the system has more than a few CPUs/cores,
it might be possible to get a little better performance while using
io-threads on the client. That'll make it possible to offload the
read-ahead to an io-thread without blocking the main glusterfs thread.
Then, the benefit of read-ahead + io-threads might show up when the data
is actually needed, and could be served without a kernel entry/exit for
file system call.

> 
> 2) io-threads
> 
> Is this (usefully) applicable on the client side?
> 

It is. Using io-threads on the client side helps offload the processing
of individual file operations onto a separate thread, freeing up
the main thread to perform other tasks. This is especially applicable
when using io-threads under a write-behind and/or read-ahead translators
where the write-behind and read-ahead requests, i.e. background or
asynchronous requests essentially, can be offloaded to the threads while
freeing up the main glusterfs thread to handle sync requests, i.e.
requests that could make the application block on a syscall.

Also, using io-threads on client side could help in performing network
IO in a separate thread, again freeing up the main thread for other
in-band tasks.

Then again, if the workload is not concurrent in terms of number of
processes or number of files/dirs, then io-threads might not help much.

> 3) io-cache
> 
> The wiki page has the same paragraph pasted for both io-threads and 
> io-cache. Are they the same thing, or is this a documentation bug?
No, they're not the same. The documentation is still in a flux. Hope
this version will help:
http://www.gluster.org/docs/index.php/Translators_options

> What does io-cache do?
io-cache is a translator that caches data from files so that future
references do not lead to network requests. It is generally used along
with read-ahead so that the data that gets read ahead or any data that
gets read, for that matter, will be available from the local client
cache. We're also working on incorporating support for write buffering
in io-cache so that write operations can also benefit from local
buffering until a point in time suitable for actual transmission to the
server.

> 
> Finally - which translators are deemed stable (no know issues - 
> memory leaks/bloat, crashes, corruption, etc.)?

We can definitely vouch for a higher degree of stability of the
releases. Otherwise, I dont think there is any performance translator we
can call completely stable/mature because of the roadmap we have for
constantly upgrading algorithms, functionality, etc.

> 
> Any particular suggestions on which performance translator 
> combination would be good to apply for a shared root AFR over a WAN? 
> I already have read-subvolume set to the local mirror, but any 
> improvement is welcome when latencies soar to 100ms and b/w gets 
> hammered down to 1-2.5 Mb/s.


WANs are generally characterised as having a large bandwidth-delay
product. That basically means, for good throughput, we should be
pipelining as much data as possible over the link, so that the long
latency overhead can be mitigated or amortised by sending larger amount
of data for the same fixed overhead.

That said, what particular workload is it that gives you a throughput of
1-2.5 Mb/s?

When you say "latencies soar to 100ms", does that mean, these are just
unusual spikes or is that the normal latency observed?

It'd help to see your volfiles and how the performance translators are
arranged.


> 
> Another thing - when a node works standalone in AFR, performance is 
> pretty good, but as soon as a peer node joins, even though the 
> original node is the primary, performance degrades on the primary 
> node quite significantly, even though the interconnect is direct 
> gigabit, which shouldn't be adding any particular latency (< 0.1ms) 
> or overheads, especially on the primary node. Is there any particular
>  reason for this degradation? It's OK in normal usage, but some 
> operations (e.g. building an big bootstrapping initrd (50MB 
> compressed, including all the gernel drivers) takes nearly 10x longer
>  when the peers join than when the node is standalone. I expected 
> some degradation, but only on the order of added network latency, and
>  this is way, way more. I tried with and without direct-io=off, and 
> that didn't make a great amount of difference. Which performance 
> translators are likely to help with this use case?
> 

I think Vikas will be able to answer that better.

-Shehjar

> Gordan
> 
> 
> _______________________________________________ Gluster-devel mailing
>  list Gluster-devel at nongnu.org 
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>