[Bugs] [Bug 1467209] [Scale] : Rebalance ETA shows the initial estimate to be ~140 days, finishes within 18 hours though.

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1467209

Nithya Balachandran <nbalacha at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED



--- Comment #1 from Nithya Balachandran <nbalacha at redhat.com> ---
--- Additional comment from Nithya Balachandran on 2017-06-14 02:15:27 EDT ---

The rebalance estimate feature works best when the files are of a uniform size.
This is not the case with this setup where the volume contains a mix of both
large and small files.


>From the logs, it looks like rebalance initially spent a lot of time migrating
very large files:


1413 [2017-06-12 13:14:26.923797] I [MSGID: 109028]
[dht-rebalance.c:4669:gf_defrag_status_get] 0-glusterfs: Files migrated: 2,
size: 21474836480, lookups: 514, failures: 0, skipped: 0
1414 [2017-06-12 13:14:28.069317] I [dht-rebalance.c:4578:gf_defrag_status_get]
0-glusterfs: TIME: num_files_lookedup=514,elapsed time =
507.000000,rate_lookedup=1.013807
1415 [2017-06-12 13:14:28.069357] I [dht-rebalance.c:4581:gf_defrag_status_get]
0-glusterfs: TIME: Estimated total time to complete = 2929242 seconds
1416 [2017-06-12 13:14:28.069369] I [dht-rebalance.c:4584:gf_defrag_status_get]
0-glusterfs: TIME: Seconds left = 2928735 seconds


So far only 2 files have been migrated but initially calculated file count
shows well over 200K files. Based on this the estimated time is roughly 140
days.  


As rebalance proceeds and starts processing the smaller files, the rate goes up
and the estimated time goes down.

This starts roughly around :
[2017-06-12 14:41:47.655006] I [dht-rebalance.c:4578:gf_defrag_status_get]
0-glusterfs: TIME: num_files_lookedup=137397,elapsed time =
5746.000000,rate_lookedup=23.911765
[2017-06-12 14:41:47.655044] I [dht-rebalance.c:4581:gf_defrag_status_get]
0-glusterfs: TIME: Estimated total time to complete = 124193 seconds
[2017-06-12 14:41:47.655058] I [dht-rebalance.c:4584:gf_defrag_status_get]
0-glusterfs: TIME: Seconds left = 118447 seconds


and the estimated time now is roughly 1/20th the originally calculated time
(roughly 32 hours).


As the rebalance proceed further,
[2017-06-13 03:23:00.853181] I [dht-rebalance.c:4578:gf_defrag_status_get]
0-glusterfs: TIME: num_files_lookedup=3557582,elapsed time =
51419.000000,rate_lookedup=69.188082
[2017-06-13 03:23:00.853216] I [dht-rebalance.c:4581:gf_defrag_status_get]
0-glusterfs: TIME: Estimated total time to complete = 51563 seconds
[2017-06-13 03:23:00.853227] I [dht-rebalance.c:4584:gf_defrag_status_get]
0-glusterfs: TIME: Seconds left = 144 seconds


The estimated time is now 51563 s (roughly 14 hours).

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.