[Gluster-users] Gluster performance updates
xhernandez at redhat.com
Mon Oct 1 11:45:28 UTC 2018
this is an update containing some work done regarding performance and
consistency during latest weeks. We'll try to build a complete list of all
known issues and track them through this email thread. Please, let me know
of any performance issue not included in this email so that we can build
and track all of them.
While testing performance on Red Hat products, we have identified a problem
in the way eager-locking was working on replicate volumes for some
scenarios (virtualization and database workloads were affected). It caused
an unnecessary amount of finodelk and fxattrop requests, that was
increasing latency of write operations.
This has already been fixed with patches  and .
We have also identified some additional settings that provide better
performance for database workloads. A patch  to update the default
database profile with the new settings has been merged.
Combining all these changes (AFR fix and settings), pgbench performance has
improved ~300% on bare metal using NVME, and a random I/O fio test running
on VM has also improved more than 300%.
We have identified two issues in fuse mounts:
- Becasue of selinux in client machine, a getxattr request is sent by
fuse before each write request. Though it adds some latency, currently this
request is directly answered by fuse xlator when selinux is not enabled in
gluster (default setting).
- When *fopen-keep-cache* is enabled (default setting), kernel fuse
sends stat requests before each read. Even disabling fopen-keep-cache, fuse
still sends half of the stat requests. This has been tracked down to the
atime update, however mounting a volume with noatime doesn't help to solve
the issue because kernel fuse doesn't correctly handle noatime setting.
Some other issues are detected:
- Bad performance of write-behind when stat and writes to the same file
are mixed. Right now, when a stat is received, all previous cached writes
are flushed before processing the new request. The same happens for reads
when it overlaps with a cached previous write. This makes write-behind
useless in this scenario.
*Note*: fuse is currently sending stat requests before reads (see previous
known issue), making reads almost as problematic as stat requests.
- Self-heal seems to be slow. It's still being investigated but there
are some indications that we have a considerable amount of contention in
io-threads. This contention could be the cause of some other performance
issues, but we'll need to investigate more about this. There is already
some work  trying to reduce it.
- 'ls' performance is not good in some cases. When the volume has many
bricks, 'ls' performance tends to degrade. We are still investigating the
cause, but one important factor is that DHT sends readdir(p) requests to
all its subvolumes, This means that 'ls' will run at the speed of the
slower of the bricks. If any brick has an issue, or a spike in load, even
if it's transitory, it will have a bad impact in 'ls' performance. This can
be alleviated by enabling parallel-readdir and readdir-ahead option.
*Note*: There have been reports that enabling parallel-readdir causes some
entries to apparently disappear after some time (though they are still
present on the bricks). I'm not aware of the root cause yet.
- The number of threads in a server is quite high when multiple bricks
are present, even if brick-mux is used. There are some efforts  trying
to reduce this number.
We have recently started the design  of a new caching infrastructure
that should provide much better performance, specially for small files or
metadata intensive workloads. It should also provide a safe infrastructure
to keep cached information consistent on all clients.
This framework will make caching features available to any xlator that
could need them in an easy and safe way.
The current thinking is that current existing caching xlators (mostly
md-cache, io-cache and write-behind) will probably be reworked as a single
complete caching xlator, since this makes things easier.
Any feedback or ideas will be highly appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Gluster-users