[Gluster-devel] Rebalance improvement design

[Benjamin Turner]

Thanks Vijay!  I forgot to upgrade the kernel(thinp 6.6 perf bug gah)
before I created this data set, so its a bit smaller:

total threads = 16
total files = 7,060,700 (64 kb files, 100 files per dir)
total data =   430.951 GB
 88.26% of requested files processed, minimum is  70.00
10101.355737 sec elapsed time
698.985382 files/sec
698.985382 IOPS
43.686586 MB/sec

I updated everything and ran the rebalanace
on glusterfs-3.8dev-0.107.git275f724.el6.x86_64.:

[root at gqas001 ~]# gluster v rebalance testvol status
                                    Node Rebalanced-files          size
  scanned      failures       skipped               status   run time in
secs
                               ---------      -----------   -----------
-----------   -----------   -----------         ------------
--------------
                               localhost          1327346        81.0GB
  3999140             0             0            completed
55088.00
      gqas013.sbu.lab.eng.bos.redhat.com                0        0Bytes
        1             0             0            completed
26070.00
      gqas011.sbu.lab.eng.bos.redhat.com                0        0Bytes
        0             0             0               failed
0.00
      gqas014.sbu.lab.eng.bos.redhat.com                0        0Bytes
        0             0             0               failed
0.00
      gqas016.sbu.lab.eng.bos.redhat.com          1325857        80.9GB
  4000865             0             0            completed
55088.00
      gqas015.sbu.lab.eng.bos.redhat.com                0        0Bytes
        0             0             0               failed
0.00
volume rebalance: testvol: success:


A couple observations:

I am seeing lots of threads / processes running:

[root at gqas001 ~]# ps -eLf | grep glu | wc -l
96 <- 96 gluster threads
[root at gqas001 ~]# ps -eLf | grep rebal | wc -l
36 <- 36 rebal threads.

Is this tunible?  Is there a use case where we would need to limit this?
Just curious, how did we arrive at 36 rebal threads?

# cat /var/log/glusterfs/testvol-rebalance.log | wc -l
4,577,583
[root at gqas001 ~]# ll /var/log/glusterfs/testvol-rebalance.log -h
-rw------- 1 root root 1.6G May  3 12:29
/var/log/glusterfs/testvol-rebalance.log

:) How big is this going to get when I do the 10-20 TB?  I'll keep tabs on
this, my default test setup only has:

[root at gqas001 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_gqas001-lv_root   50G  4.8G   42G  11% /
tmpfs                  24G     0   24G   0% /dev/shm
/dev/sda1             477M   65M  387M  15% /boot
/dev/mapper/vg_gqas001-lv_home  385G   71M  366G   1% /home
/dev/mapper/gluster_vg-lv_bricks  9.5T  219G  9.3T   3% /bricks

Next run I want to fill up a 10TB cluster and double the # of bricks to
simulate running out of space doubling capacity.  Any other fixes or
changes that need to go in before I try a larger data set?  Before that I
may run my performance regression suite against a system while a rebal is
in progress and check how it affects performance.  I'll turn both these
cases into perf regression tests that I run with iozone smallfile and such,
any other use cases I should add?  Should I add hard / soft links /
whatever else tot he data set?

-b


On Sun, May 3, 2015 at 11:48 AM, Vijay Bellur <vbellur at redhat.com> wrote:

> On 05/01/2015 10:23 AM, Benjamin Turner wrote:
>
>> Ok I have all my data created and I just started the rebalance.  One
>> thing to not in the client log I see the following spamming:
>>
>> [root at gqac006 ~]# cat /var/log/glusterfs/gluster-mount-.log | wc -l
>> 394042
>>
>> [2015-05-01 00:47:55.591150] I [MSGID: 109036]
>> [dht-common.c:6478:dht_log_new_layout_for_dir_selfheal] 0-testvol-dht:
>> Setting layout of
>> /file_dstdir/
>> gqac006.sbu.lab.eng.bos.redhat.com/thrd_05/d_001/d_000/d_004/d_006
>> <
>> http://gqac006.sbu.lab.eng.bos.redhat.com/thrd_05/d_001/d_000/d_004/d_006
>> >
>> with [Subvol_name: testvol-replicate-0, Err: -1 , Start: 0 , Stop:
>> 2141429669 ], [Subvol_name: testvol-replicate-1, Err: -1 , Start:
>> 2141429670 , Stop: 4294967295 ],
>> [2015-05-01 00:47:55.596147] I
>> [dht-selfheal.c:1587:dht_selfheal_layout_new_directory] 0-testvol-dht:
>> chunk size = 0xffffffff / 19920276 = 0xd7
>> [2015-05-01 00:47:55.596177] I
>> [dht-selfheal.c:1626:dht_selfheal_layout_new_directory] 0-testvol-dht:
>> assigning range size 0x7fa39fa6 to testvol-replicate-1
>>
>
>
> I also noticed the same set of excessive logs in my tests. Have sent
> across a patch [1] to address this problem.
>
> -Vijay
>
> [1] http://review.gluster.org/10281
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20150504/cf2d3af3/attachment.html>