[Gluster-users] Questions about gluster reblance

Fri Sep 12 03:55:59 UTC 2014

Hello Shyam. Thanks for the reply. Please see my reply below, starting with [paul:]

Please add me in address list besides gluster-uses when replying so that I can easier
reply since I subscribed gluster-users with the digest mode (No other choice if I
remember correctly.)

Date: Wed, 10 Sep 2014 10:36:41 -0400
From: Shyam <srangana at redhat.com>
To: gluster-users at gluster.org
Subject: Re: [Gluster-users] Questions about gluster reblance
Message-ID: <541061F9.7000800 at redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed

On 09/10/2014 03:27 AM, Paul Guo wrote:
> Hello,
>
> Recently I spent a bit time understanding rebalance since I want to know its
> performance given that there could be more and more bricks to be added into
> my glusterfs volume and there will be more and more files and directories
> in the existing glusterfs volume. During the test I saw something which I'm
> really confused about.
>
> Steps:
>
> SW versions: glusterfs 3.4.4 + centos 6.5
> Inital Configuration: replica 2, lab1:/brick1 + lab2:/brick1
>
> fuse_mount it on /mnt
> cp -rf /sbin /mnt (~300+ files under /sbin)
> add two more bricks: lab1:/brick2 + lab2:/brick2.
> run gluster reblance.
>
> 1) fix-layout only (e.g. gluster volume rebalance g1 fix-layout start)?
>
> After rebalance is done (observed via "gluster volume rebalance g1
> status"),?
> I found there is no file under lab1:/brick2/sbin. The hash ranges of
> new brick?lab1:/brick2/sbin and old brick lab1:/brick1/sbin appear to
> be ok.
>
> [root at lab1 Desktop]# getfattr -dm. -e hex /brick2/sbin
> getfattr: Removing leading '/' from absolute path names
> # file: brick2/sbin
> trusted.gfid=0x35976c2034d24dc2b0639fde18de007d
> trusted.glusterfs.dht=0x00000001000000007fffffffffffffff
>
> [root at lab1 Desktop]# getfattr -dm. -e hex /brick1/sbin
> getfattr: Removing leading '/' from absolute path names
> # file: brick1/sbin
> trusted.gfid=0x35976c2034d24dc2b0639fde18de007d
> trusted.glusterfs.dht=0x0000000100000000000000007ffffffe
> ?
> The question is: AFAIK, fix-layout would create "linkto" files
> (files with "linkto" xattr and with sticky bit set only)
> for those ones whose hash values belong
> to the new subvol. so there should have been some "linkto" files
> under lab1:/brick2, but no one now, why?

fix-layout only fixes the layout, i.e spreads the layout to the newer 
bricks (or bricks previously not participating in the layout). It would 
not create the linkto files.

Post fix-layout, if one were to perform a lookup on a file, that should 
have belonged to the newer brick as per the layout and hash of that file 
name, one can see the linkto file being present.

Hope this explains (1).

[paul:]

After fix-layout is complete, I mount the volume on /mnt, then 
run "ls -l /mnt/sbin/*" and "file /mnt/sbin/*",
and then I found just several linkto files are created while most files,
which should have been created under the new brick (i.e. brick2),
are not created.
[root at lab1 ~]# ls -l /brick2/sbin
total 0
---------T 2 root root 0 Sep 12 09:26 dmraid
---------T 2 root root 0 Sep 12 09:26 initctl
---------T 2 root root 0 Sep 12 09:26 ip6tables-multi
---------T 2 root root 0 Sep 12 09:26 portreserve
---------T 2 root root 0 Sep 12 09:26 reboot
---------T 2 root root 0 Sep 12 09:26 swapon
[root at lab1 ~]# getfattr -dm. -e hex /brick2/sbin
getfattr: Removing leading '/' from absolute path names
# file: brick2/sbin
trusted.gfid=0x94bc07cd18914a91ab12fbe931c63431
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff

[root at lab1 ~]# ./gethash.py reboot
0xd48b11f6L
[root at lab1 ~]# ./gethash.py swapon
0x93129578L

The hash values of reboot & swapon are in the range of
/brick2/sbin (i.e. 7fffFFFF - ffffFFFF) so the linkto files
for the two binaries are expected, but there are more
linkto files missing, e.g. xfsdump
[root at lab1 ~]# ./gethash.py xfsdump
0xc17ff86bL

Even I umount /mnt, stop-then-start the volume,
restart glusterd, remount /mnt and then do the experiment
again, I still find no more linkto files under /brick2/sbin.

>
> 2) fix-layout + data_migrate (e.g. gluster volume rebalance g1 start)
>
> After migration is done, I saw linkto files under brick2/sbin.?
> There are totally 300+ files under system /sbin. Under brick2/sbin,
> I found the 300+ files are all there! either migrated or linkto-ed.
>
> -rwxr-xr-x 2 root root   17400 Sep 10 12:02 vmcore-dmesg
> ---------T 2 root root       0 Sep 10 12:03 weak-modules
> ---------T 2 root root       0 Sep 10 12:03 wipefs
> -rwxr-xr-x 2 root root  295656 Sep 10 12:02 xfsdump
> -rwxr-xr-x 2 root root  510000 Sep 10 12:02 xfs_repair
> -rwxr-xr-x 2 root root  348088 Sep 10 12:02 xfsrestore
>
> And under brick1/sbin, those migrated files are gone as expected.
> There are near to 150 files under brick/sbin.
> ?
> This confuses me since creating those linkto files seems to
> be unnecessary, at least for files whose hash values do not belong
> to the subvol. (My understanding is that if a file's hash value is
> in the range of a subvol then it will be stored in that subvol.)

Can you check if a lookup of the file post rebalance clears up these 
_stale_ linkto files?

[paul:] No. Those linkto files are still there after I run "file /mnt/sbin/*".

How did you compute the hash of these files and decide that they do not 
belong to the new brick (i.e brick2)? I did them on my end and you are 
right (based on the layout you presented above), but I am curious as to 
how you arrived at the same conclusion.

[paul:]
The hash calculating script comes from
http://joejulian.name/blog/dht-misses-are-expensive/
It uses language binding by calling gf_dm_hashfn() in the
glusterfs C code finally. I added debug code in glusterfs
and double-confirmed that the script works correctly.
See more analysis below,
[root at lab1 ~]# ls -l /brick2/sbin
....
-rwxr-xr-x 2 root root   16576 Sep 12 11:26 wipefs
---------T 2 root root       0 Sep 12 11:27 xfsdump
[root at lab1 ~]# ./gethash.py xfsdump
0xc17ff86bL
[root at lab1 ~]# ./gethash.py wipefs
0x6afa24a9L

[root at lab1 ~]# getfattr -dm. -e hex /brick2/sbin

getfattr: Removing leading '/' from absolute path names
# file: brick2/sbin
trusted.gfid=0xddd06defaf1242b4b8ec5d41fdaa01e3
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff

[paul:] It means that wipefs belongs to /brick1/sbin
and xfsdump belongs to /brick2/sbin, but under /brick2/sbin,
wipefs is migrated and xfsdump is linkto-ed. This does not
make sense. Or I did something wrong?
This is another issue besides the uncesessary linkto file issue.

Rebalance could choose to not move files but just create the linkto 
files based on space usage between the source and target bricks etc. Not 
stating this is what happened here, but a possibility.

[paul:] All the bricks are empty before testing. They are virtual disk partitions
in virtualbox centos 6.5 guests.

>
> I quickly looked at the code. gf_defrag_start_crawl() appears to
> be the function for this operation. I do see code that does file migration
> from the code path, but debugging code shows that those "linkto" files
> seem to be not created by gf_defrag_start_crawl(). I'm not that familar with
> the code detail and the theory so I'm not sure who created those
> "linkto" files and why the "linkto" file are created.

I am going to leave this part as, dht_linkfile_create does this and 
mostly would happen during lookup.

[paul:] I added debug code in dht_linkfile_create(). It appears that
it is called for those migrated files only.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140912/27dc98e2/attachment.html>