[Gluster-users] Rebalance taking > 2 months
Nithya Balachandran
nbalacha at redhat.com
Tue Jul 31 08:28:33 UTC 2018
Hi Rusty,
A rebalance involves 2 steps:
1. Setting a new layout on a directory
2. Migrating any files inside that directory that hash to a different
subvol based on the new layout set in step 1.
A few things to keep in mind :
- Any new content created on this volume will currently go to the newly
added brick.
- Having a more equitable file distribution is beneficial but you might
not need to do a complete rebalance to do this. You can run the script on
just enough directories to free up space on your older bricks. This should
be done on bricks which contains large files to speed this up.
Do the following on one of the server nodes:
- Create a tmp mount point and mount the volume using the rebalance
volfile
- mkdir /mnt/rebal
- glusterfs -s localhost --volfile-id rebalance/data /mnt/rebal
- Select a directory in the volume which contains a lot of large files
and which has not been processed by the rebalance yet - the lower down in
the tree the better. Check the rebalance logs to figure out which dirs have
not been processed yet.
- cd /mnt/rebal/<chosen_dir>
- for dir in `find . -type d`; do echo $dir |xargs -0 -n1 -P10 bash
process_dir.sh;done
- You can run this for different values of <chosen_dir> and on multiple
server nodes in parallel as long as the directory trees for the different
<chosen_dirs> don't overlap.
- Do this for multiple directories until the disk space used reduces on
the older bricks.
This is a very simple script. Let me know how it works - we can always
tweak it for your particular data set.
>and performance is basically garbage while it rebalances
Can you provide more detail on this? What kind of effects are you seeing?
How many clients access this volume?
Regards,
Nithya
On 30 July 2018 at 22:18, Nithya Balachandran <nbalacha at redhat.com> wrote:
> I have not documented this yet - I will send you the steps tomorrow.
>
> Regards,
> Nithya
>
> On 30 July 2018 at 20:23, Rusty Bower <rusty at rustybower.com> wrote:
>
>> That would be awesome. Where can I find these?
>>
>> Rusty
>>
>> Sent from my iPhone
>>
>> On Jul 30, 2018, at 03:40, Nithya Balachandran <nbalacha at redhat.com>
>> wrote:
>>
>> Hi Rusty,
>>
>> Sorry for the delay getting back to you. I had a quick look at the
>> rebalance logs - it looks like the estimates are based on the time taken to
>> rebalance the smaller files.
>>
>> We do have a scripting option where we can use virtual xattrs to trigger
>> file migration from a mount point. That would speed things up.
>>
>>
>> Regards,
>> Nithya
>>
>> On 28 July 2018 at 07:11, Rusty Bower <rusty at rustybower.com> wrote:
>>
>>> Just wanted to ping this to see if you guys had any thoughts, or other
>>> scripts I can run for this stuff. It's still predicting another 90 days to
>>> rebalance this, and performance is basically garbage while it rebalances.
>>>
>>> Rusty
>>>
>>> On Mon, Jul 23, 2018 at 10:19 AM, Rusty Bower <rusty at rustybower.com>
>>> wrote:
>>>
>>>> datanode03 is the newest brick
>>>>
>>>> the bricks had gotten pretty full, which I think might be part of the
>>>> issue:
>>>> - datanode01 /dev/sda1 51T 48T 3.3T 94% /mnt/data
>>>> - datanode02 /dev/sda1 51T 48T 3.4T 94% /mnt/data
>>>> - datanode03 /dev/md0 128T 4.6T 123T 4% /mnt/data
>>>>
>>>> each of the bricks are on a completely separate disk from the OS
>>>>
>>>> I'll shoot you the log files offline :)
>>>>
>>>> Thanks!
>>>> Rusty
>>>>
>>>> On Mon, Jul 23, 2018 at 3:12 AM, Nithya Balachandran <
>>>> nbalacha at redhat.com> wrote:
>>>>
>>>>> Hi Rusty,
>>>>>
>>>>> Sorry I took so long to get back to you.
>>>>>
>>>>> Which is the newly added brick? I see datanode02 has not picked up
>>>>> any files for migration which is odd.
>>>>> How full are the individual bricks (df -h ) output.
>>>>> Is each of your bricks in a separate partition?
>>>>> Can you send me the rebalance logs from all 3 nodes (offline if you
>>>>> prefer)?
>>>>>
>>>>> We can try using scripts to speed up the rebalance if you prefer.
>>>>>
>>>>> Regards,
>>>>> Nithya
>>>>>
>>>>>
>>>>>
>>>>> On 16 July 2018 at 22:06, Rusty Bower <rusty at rustybower.com> wrote:
>>>>>
>>>>>> Thanks for the reply Nithya.
>>>>>>
>>>>>> 1. glusterfs 4.1.1
>>>>>>
>>>>>> 2. Volume Name: data
>>>>>> Type: Distribute
>>>>>> Volume ID: 294d95ce-0ff3-4df9-bd8c-a52fc50442ba
>>>>>> Status: Started
>>>>>> Snapshot Count: 0
>>>>>> Number of Bricks: 3
>>>>>> Transport-type: tcp
>>>>>> Bricks:
>>>>>> Brick1: datanode01:/mnt/data/bricks/data
>>>>>> Brick2: datanode02:/mnt/data/bricks/data
>>>>>> Brick3: datanode03:/mnt/data/bricks/data
>>>>>> Options Reconfigured:
>>>>>> performance.readdir-ahead: on
>>>>>>
>>>>>> 3.
>>>>>> Node Rebalanced-files
>>>>>> size scanned failures skipped status run
>>>>>> time in h:m:s
>>>>>> --------- -----------
>>>>>> ----------- ----------- ----------- -----------
>>>>>> ------------ --------------
>>>>>> localhost 36822
>>>>>> 11.3GB 50715 0 0 in progress
>>>>>> 26:46:17
>>>>>> datanode02 0
>>>>>> 0Bytes 2852 0 0 in progress
>>>>>> 26:46:16
>>>>>> datanode03 3128
>>>>>> 513.7MB 11442 0 3128 in progress
>>>>>> 26:46:17
>>>>>> Estimated time left for rebalance to complete : > 2 months. Please
>>>>>> try again later.
>>>>>> volume rebalance: data: success
>>>>>>
>>>>>> 4. Directory structure is basically an rsync backup of some old
>>>>>> systems as well as all of my personal media. I can elaborate more, but it's
>>>>>> a pretty standard filesystem.
>>>>>>
>>>>>> 5. In some folders there might be up to like 12-15 levels of
>>>>>> directories (especially the backups)
>>>>>>
>>>>>> 6. I'm honestly not sure, I can try to scrounge this number up
>>>>>>
>>>>>> 7. My guess would be > 100k
>>>>>>
>>>>>> 8. Most files are pretty large (media files), but there's a lot of
>>>>>> small files (metadata and configuration files) as well
>>>>>>
>>>>>> I've also appended a (moderately sanitized) snippet of the rebalance
>>>>>> log (let me know if you need more)
>>>>>>
>>>>>> [2018-07-16 17:37:59.979003] I [MSGID: 0]
>>>>>> [dht-rebalance.c:1799:dht_migrate_file] 0-data-dht: destination for
>>>>>> file - /this/is/a/file/path/that/exists/wz/wz/Npc.wz/2040036.img.xml
>>>>>> is changed to - data-client-2
>>>>>> [2018-07-16 17:38:00.004262] I [MSGID: 109022]
>>>>>> [dht-rebalance.c:2274:dht_migrate_file] 0-data-dht: completed
>>>>>> migration of /this/is/a/file/path/that/exis
>>>>>> ts/wz/wz/Npc.wz/2112002.img.xml from subvolume data-client-0 to
>>>>>> data-client-2
>>>>>> [2018-07-16 17:38:00.725582] I [dht-rebalance.c:4982:gf_defrag_get_estimates_based_on_size]
>>>>>> 0-glusterfs: TIME: (size) total_processed=43108305980 tmp_cnt =
>>>>>> 55419279917056,rate_processed=446597.869797, elapsed = 96526.000000
>>>>>> [2018-07-16 17:38:00.725641] I [dht-rebalance.c:5130:gf_defrag_status_get]
>>>>>> 0-glusterfs: TIME: Estimated total time to complete (size)= 124092127
>>>>>> seconds, seconds left = 123995601
>>>>>> [2018-07-16 17:38:00.725709] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5210:gf_defrag_status_get] 0-glusterfs: Rebalance
>>>>>> is in progress. Time taken is 96526.00 secs
>>>>>> [2018-07-16 17:38:00.725738] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5214:gf_defrag_status_get] 0-glusterfs: Files
>>>>>> migrated: 36876, size: 12270259289, lookups: 50715, failures: 0, skipped: 0
>>>>>> [2018-07-16 17:38:02.769121] I [dht-rebalance.c:4982:gf_defrag_get_estimates_based_on_size]
>>>>>> 0-glusterfs: TIME: (size) total_processed=43108305980 tmp_cnt =
>>>>>> 55419279917056,rate_processed=446588.616567, elapsed = 96528.000000
>>>>>> [2018-07-16 17:38:02.769207] I [dht-rebalance.c:5130:gf_defrag_status_get]
>>>>>> 0-glusterfs: TIME: Estimated total time to complete (size)= 124094698
>>>>>> seconds, seconds left = 123998170
>>>>>> [2018-07-16 17:38:02.769263] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5210:gf_defrag_status_get] 0-glusterfs: Rebalance
>>>>>> is in progress. Time taken is 96528.00 secs
>>>>>> [2018-07-16 17:38:02.769286] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5214:gf_defrag_status_get] 0-glusterfs: Files
>>>>>> migrated: 36876, size: 12270259289, lookups: 50715, failures: 0, skipped: 0
>>>>>> [2018-07-16 17:38:03.410469] I [dht-rebalance.c:1645:dht_migrate_file]
>>>>>> 0-data-dht: /this/is/a/file/path/that/exis
>>>>>> ts/wz/wz/Npc.wz/9201002.img.xml: attempting to move from
>>>>>> data-client-0 to data-client-2
>>>>>> [2018-07-16 17:38:03.416127] I [MSGID: 109022]
>>>>>> [dht-rebalance.c:2274:dht_migrate_file] 0-data-dht: completed
>>>>>> migration of /this/is/a/file/path/that/exis
>>>>>> ts/wz/wz/Npc.wz/2040036.img.xml from subvolume data-client-0 to
>>>>>> data-client-2
>>>>>> [2018-07-16 17:38:04.738885] I [dht-rebalance.c:1645:dht_migrate_file]
>>>>>> 0-data-dht: /this/is/a/file/path/that/exis
>>>>>> ts/wz/wz/Npc.wz/9110012.img.xml: attempting to move from
>>>>>> data-client-0 to data-client-2
>>>>>> [2018-07-16 17:38:04.745722] I [MSGID: 109022]
>>>>>> [dht-rebalance.c:2274:dht_migrate_file] 0-data-dht: completed
>>>>>> migration of /this/is/a/file/path/that/exis
>>>>>> ts/wz/wz/Npc.wz/9201002.img.xml from subvolume data-client-0 to
>>>>>> data-client-2
>>>>>> [2018-07-16 17:38:04.812368] I [dht-rebalance.c:4982:gf_defrag_get_estimates_based_on_size]
>>>>>> 0-glusterfs: TIME: (size) total_processed=43108308134 tmp_cnt =
>>>>>> 55419279917056,rate_processed=446579.386035, elapsed = 96530.000000
>>>>>> [2018-07-16 17:38:04.812417] I [dht-rebalance.c:5130:gf_defrag_status_get]
>>>>>> 0-glusterfs: TIME: Estimated total time to complete (size)= 124097263
>>>>>> seconds, seconds left = 124000733
>>>>>> [2018-07-16 17:38:04.812465] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5210:gf_defrag_status_get] 0-glusterfs: Rebalance
>>>>>> is in progress. Time taken is 96530.00 secs
>>>>>> [2018-07-16 17:38:04.812489] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5214:gf_defrag_status_get] 0-glusterfs: Files
>>>>>> migrated: 36877, size: 12270261443, lookups: 50715, failures: 0, skipped: 0
>>>>>> [2018-07-16 17:38:04.992413] I [dht-rebalance.c:1645:dht_migrate_file]
>>>>>> 0-data-dht: /this/is/a/file/path/that/exis
>>>>>> ts/wz/wz/Npc.wz/2050000.img.xml: attempting to move from
>>>>>> data-client-0 to data-client-2
>>>>>> [2018-07-16 17:38:04.994122] I [MSGID: 109022]
>>>>>> [dht-rebalance.c:2274:dht_migrate_file] 0-data-dht: completed
>>>>>> migration of /this/is/a/file/path/that/exis
>>>>>> ts/wz/wz/Npc.wz/9110012.img.xml from subvolume data-client-0 to
>>>>>> data-client-2
>>>>>> [2018-07-16 17:38:06.855618] I [dht-rebalance.c:4982:gf_defrag_get_estimates_based_on_size]
>>>>>> 0-glusterfs: TIME: (size) total_processed=43108318798 tmp_cnt =
>>>>>> 55419279917056,rate_processed=446570.244043, elapsed = 96532.000000
>>>>>> [2018-07-16 17:38:06.855719] I [dht-rebalance.c:5130:gf_defrag_status_get]
>>>>>> 0-glusterfs: TIME: Estimated total time to complete (size)= 124099804
>>>>>> seconds, seconds left = 124003272
>>>>>> [2018-07-16 17:38:06.855770] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5210:gf_defrag_status_get] 0-glusterfs: Rebalance
>>>>>> is in progress. Time taken is 96532.00 secs
>>>>>> [2018-07-16 17:38:06.855793] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5214:gf_defrag_status_get] 0-glusterfs: Files
>>>>>> migrated: 36879, size: 12270266602, lookups: 50715, failures: 0, skipped: 0
>>>>>> [2018-07-16 17:38:08.511064] I [dht-rebalance.c:1645:dht_migrate_file]
>>>>>> 0-data-dht: /this/is/a/file/path/that/exis
>>>>>> ts/wz/wz/Npc.wz/9201055.img.xml: attempting to move from
>>>>>> data-client-0 to data-client-2
>>>>>> [2018-07-16 17:38:08.533029] I [MSGID: 109022]
>>>>>> [dht-rebalance.c:2274:dht_migrate_file] 0-data-dht: completed
>>>>>> migration of /this/is/a/file/path/that/exis
>>>>>> ts/wz/wz/Npc.wz/2050000.img.xml from subvolume data-client-0 to
>>>>>> data-client-2
>>>>>> [2018-07-16 17:38:08.899708] I [dht-rebalance.c:4982:gf_defrag_get_estimates_based_on_size]
>>>>>> 0-glusterfs: TIME: (size) total_processed=43108318798 tmp_cnt =
>>>>>> 55419279917056,rate_processed=446560.991961, elapsed = 96534.000000
>>>>>> [2018-07-16 17:38:08.899791] I [dht-rebalance.c:5130:gf_defrag_status_get]
>>>>>> 0-glusterfs: TIME: Estimated total time to complete (size)= 124102375
>>>>>> seconds, seconds left = 124005841
>>>>>> [2018-07-16 17:38:08.899842] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5210:gf_defrag_status_get] 0-glusterfs: Rebalance
>>>>>> is in progress. Time taken is 96534.00 secs
>>>>>> [2018-07-16 17:38:08.899865] I [MSGID: 109028]
>>>>>> [dht-rebalance.c:5214:gf_defrag_status_get] 0-glusterfs: Files
>>>>>> migrated: 36879, size: 12270266602, lookups: 50715, failures: 0, skipped: 0
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 16, 2018 at 7:37 AM, Nithya Balachandran <
>>>>>> nbalacha at redhat.com> wrote:
>>>>>>
>>>>>>> If possible, please send the rebalance logs as well.
>>>>>>>
>>>>>>>
>>>>>>> On 16 July 2018 at 10:14, Nithya Balachandran <nbalacha at redhat.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Rusty,
>>>>>>>>
>>>>>>>> We need the following information:
>>>>>>>>
>>>>>>>> 1. The exact gluster version you are running
>>>>>>>> 2. gluster volume info <volname>
>>>>>>>> 3. gluster rebalance status
>>>>>>>> 4. Information on the directory structure and file locations on
>>>>>>>> your volume.
>>>>>>>> 5. How many levels of directories
>>>>>>>> 6. How many files and directories in each level
>>>>>>>> 7. How many directories and files in total (a rough estimate)
>>>>>>>> 8. Average file size
>>>>>>>>
>>>>>>>> Please note that having a rebalance running in the background
>>>>>>>> should not affect your volume access in any way. However I would like to
>>>>>>>> know why only 6000 files have been scanned in 6 hours.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Nithya
>>>>>>>>
>>>>>>>>
>>>>>>>> On 16 July 2018 at 06:13, Rusty Bower <rusty at rustybower.com> wrote:
>>>>>>>>
>>>>>>>>> Hey folks,
>>>>>>>>>
>>>>>>>>> I just added a new brick to my existing gluster volume, but *gluster
>>>>>>>>> volume rebalance data status* is telling me the
>>>>>>>>> following: Estimated time left for rebalance to complete : > 2 months.
>>>>>>>>> Please try again later.
>>>>>>>>>
>>>>>>>>> I already did a fix-mapping, but this thing is absolutely crawling
>>>>>>>>> trying to rebalance everything (last estimate was ~40 years)
>>>>>>>>>
>>>>>>>>> Any thoughts on if this is a bug, or ways to speed this up? It's
>>>>>>>>> taking ~6 hours to scan 6000 files, which seems unreasonably slow.
>>>>>>>>>
>>>>>>>>> Thanks
>>>>>>>>> Rusty
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180731/de38b53c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: process_dir.sh
Type: application/x-shellscript
Size: 203 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180731/de38b53c/attachment.bin>
More information about the Gluster-users
mailing list