[Bugs] [Bug 1467614] Gluster read/write performance improvements on NVMe backend

bugzilla at redhat.com bugzilla at redhat.com
Fri Sep 15 12:37:19 UTC 2017


https://bugzilla.redhat.com/show_bug.cgi?id=1467614



--- Comment #27 from Krutika Dhananjay <kdhananj at redhat.com> ---
(In reply to Krutika Dhananjay from comment #16)
> Some more updates since comment #15:
> 
> NOTE: All of these tests were done on ovirt-gluster hyperconverged setup
> with 3 hosts, 9 vms, and bricks carved out of NVMe drives.
> 
> * iobuf.diff had a bug, and had not eliminated one case of contention on
> iobuf_pool->mutex in __iobuf_put(). I corrected that and ran randrd tests
> again. This time iobuf_pool->mutex didn't show up in mutrace info captured
> on brick. But the iops continued to drop by 10K.
> 
> * Replacing inode_table->lock with inode->lock will lead to correctness
> issues and possible corruption in inode_unref() since inode_unref() does not
> merely decrement ref count but also operates on inode_table and the lists
> within it. So it is necessary to acquire inode_table->lock when performing
> inode_unref(). The old (existing) code is correct.
> 
> * Applied https://review.gluster.org/#/c/16832/ from facebook to eliminate
> lock contention on conf->mutex in io-threads. Somehow this brought the perf
> down. This was with the default value of fops-per-thread-ratio - 20. This
> actually caused iops to drop from ~60K to 30K.
> 
> * Default io-thread-count is 16. So I decreased io-thread-count to 1 and yet
> there was no drop in IOPs. Only disabling client-io-threads altogether
> causes drop in performance.
> 
> The last point is possibly means either that even the io-threads are getting
> held up at lower layers - perhaps epoll, rpc, etc or not enough parallel
> requests are received by client-io-threads for thread count to matter.
> 
> * Most importantly I disabled fuse event history in the crudest manner
> possible (by deleting calls to fuse_log_eh() and fuse_log_eh_fop()) and I'm
> seeing that the IOPs increase by 5K and disk utilization is up to 88-90%.
> This change takes the total IOPs we are seeing in the best case scenario for
> randrds to 66K from 61K. That still did not bring down fuse thread's CPU
> utilization though!

I stand corrected. The patch I was using had a bug. I fixed that and ran the
test again and now I see iops go up from 61K to ~73K. I saw disk utilization of
up to 92%.

Also, on a single brick distribute volume, I now see IOPs go up from ~18K to
~26K. The disk was 100% utilized (yes, I know that doesn't necessarily mean
we've reached a dead end, just saying :)).

All of this was with randrd.job. I am hoping some of the patches we'd tried
earlier (such as fb's patch in io-threads) will be more relevant now, because
turns out the lock contention due to event history was not only affecting the
callbacks of fops in fuse-bridge but also the call path (see FUSE_FOP, in fact
this was the bug I was mentioning above). More updates next week..

-Krutika


> 
> That's it for now.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=9Ifa4TZXYJ&a=cc_unsubscribe


More information about the Bugs mailing list