[Bugs] [Bug 1467614] Gluster read/write performance improvements on NVMe backend

bugzilla at redhat.com bugzilla at redhat.com
Thu Sep 7 11:01:26 UTC 2017


https://bugzilla.redhat.com/show_bug.cgi?id=1467614



--- Comment #16 from Krutika Dhananjay <kdhananj at redhat.com> ---
Some more updates since comment #15:

NOTE: All of these tests were done on ovirt-gluster hyperconverged setup with 3
hosts, 9 vms, and bricks carved out of NVMe drives.

* iobuf.diff had a bug, and had not eliminated one case of contention on
iobuf_pool->mutex in __iobuf_put(). I corrected that and ran randrd tests
again. This time iobuf_pool->mutex didn't show up in mutrace info captured on
brick. But the iops continued to drop by 10K.

* Replacing inode_table->lock with inode->lock will lead to correctness issues
and possible corruption in inode_unref() since inode_unref() does not merely
decrement ref count but also operates on inode_table and the lists within it.
So it is necessary to acquire inode_table->lock when performing inode_unref().
The old (existing) code is correct.

* Applied https://review.gluster.org/#/c/16832/ from facebook to eliminate lock
contention on conf->mutex in io-threads. Somehow this brought the perf down.
This was with the default value of fops-per-thread-ratio - 20. This actually
caused iops to drop from ~60K to 30K.

* Default io-thread-count is 16. So I decreased io-thread-count to 1 and yet
there was no drop in IOPs. Only disabling client-io-threads altogether causes
drop in performance.

The last point is possibly means either that even the io-threads are getting
held up at lower layers - perhaps epoll, rpc, etc or not enough parallel
requests are received by client-io-threads for thread count to matter.

* Most importantly I disabled fuse event history in the crudest manner possible
(by deleting calls to fuse_log_eh() and fuse_log_eh_fop()) and I'm seeing that
the IOPs increase by 5K and disk utilization is up to 88-90%. This change takes
the total IOPs we are seeing in the best case scenario for randrds to 66K from
61K. That still did not bring down fuse thread's CPU utilization though!

That's it for now.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=x24dczSKOc&a=cc_unsubscribe


More information about the Bugs mailing list