[Bugs] [Bug 1175214] [RFE] Rebalance Performance Improvements

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1175214

Ben England <bengland at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |jharriga at redhat.com



--- Comment #14 from Ben England <bengland at redhat.com> ---
Rebalance appears to be slightly slower in the newest version.  And we're not
using replication yet.  Are we using thin-p?  

Use case:  It appears we added a single brick here to the volume.  Is this
typical use case, or would it be more typical to add multiple bricks to the
volume at once?  I think this is what Intuit might want to do for example. 
This could change the measured performance significantly.

It would help if we reported data in files/sec/server and MB/s/server.  For
example, if we double the number of bricks in the volume, how much data must be
moved to  rebalance, how long will this take, is it realistic to expect a user
to wait this long for rebalance to happen, etc.  

BTW, it appears that sar is not reporting network transmit rate correctly, this
is a bug that perf team has been seeing of late in RHEL 6.6, no bz yet.  

I would guess we could do better than that with the parallel algorithm
discussed in previous meetings.  Consider RHS QE's own results for small-file
writes, etc.  Is there any explanation for why this is unexpectedly slow in the
absence of any other activity on the system?   Other than swap space
consumption, which wasn't growing, I didn't see any obvious signs of a hardware
bottleneck.  Perhaps part of the problem is that only one server is receiving
rebalanced files, is this correct?

I think a reasonable goal for the rebalancing for the average file size used
here is that we should be able to achieve 20% of per-server sequential write
rate measured by QE on the servers targeted by the rebalance, provided that
rebalance is not run in parallel with an application workload.  We have to read
the file before we can move it to its target DHT subvolume, and then we have to
delete it from its origin brick(s). So if seq. write speed is 300 MB/s/server
(I don't remember what it is anymore) in the above example, That would give you
something like 60 MB/s/server, which is roughly 1/5 day per TB, per server.  So
the time taken to fill up a 12-TB server like the above would be about 2 days.


Can someone else try some of these calculations and see if they agree?  Do you
think 20% of seq. write perf is a fair goal?  If not, what do you think is a
reasonable goal to achieve?  Keep in mind that with introduction of SSDs and
40-Gbps networking, rebalancing will be expected to run much faster than
before.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=x2L3z7sd9d&a=cc_unsubscribe