[Gluster-users] Questions about gluster reblance

[Paul Guo]

Hello,


Recently I spent a bit time understanding rebalance since I want to know its
performance given that there could be more and more bricks to be added into
my glusterfs volume and there will be more and more files and directories
in the existing glusterfs volume. During the test I saw something which I'm
really confused about.


Steps:


SW versions: glusterfs 3.4.4 + centos 6.5
Inital Configuration: replica 2, lab1:/brick1 + lab2:/brick1


fuse_mount it on /mnt
cp -rf /sbin /mnt (~300+ files under /sbin)
add two more bricks: lab1:/brick2 + lab2:/brick2.
run gluster reblance.


1) fix-layout only (e.g. gluster volume rebalance g1 fix-layout start)‍


After rebalance is done (observed via "gluster volume rebalance g1 status"),‍
I found there is no file under lab1:/brick2/sbin. The hash ranges of
new brick ‍lab1:/brick2/sbin and old brick lab1:/brick1/sbin appear to
be ok.


[root at lab1 Desktop]# getfattr -dm. -e hex /brick2/sbin
getfattr: Removing leading '/' from absolute path names
# file: brick2/sbin
trusted.gfid=0x35976c2034d24dc2b0639fde18de007d
trusted.glusterfs.dht=0x00000001000000007fffffffffffffff


[root at lab1 Desktop]# getfattr -dm. -e hex /brick1/sbin
getfattr: Removing leading '/' from absolute path names
# file: brick1/sbin
trusted.gfid=0x35976c2034d24dc2b0639fde18de007d
trusted.glusterfs.dht=0x0000000100000000000000007ffffffe

‍

The question is: AFAIK, fix-layout would create "linkto" files
(files with "linkto" xattr and with sticky bit set only)
for those ones whose hash values belong
to the new subvol. so there should have been some "linkto" files
under lab1:/brick2, but no one now, why?


2) fix-layout + data_migrate (e.g. gluster volume rebalance g1 start)


After migration is done, I saw linkto files under brick2/sbin.‍
There are totally 300+ files under system /sbin. Under brick2/sbin,
I found the 300+ files are all there! either migrated or linkto-ed.


-rwxr-xr-x 2 root root   17400 Sep 10 12:02 vmcore-dmesg
---------T 2 root root       0 Sep 10 12:03 weak-modules
---------T 2 root root       0 Sep 10 12:03 wipefs
-rwxr-xr-x 2 root root  295656 Sep 10 12:02 xfsdump
-rwxr-xr-x 2 root root  510000 Sep 10 12:02 xfs_repair
-rwxr-xr-x 2 root root  348088 Sep 10 12:02 xfsrestore



And under brick1/sbin, those migrated files are gone as expected.
There are near to 150 files under brick/sbin.
‍

This confuses me since creating those linkto files seems to
be unnecessary, at least for files whose hash values do not belong
to the subvol. (My understanding is that if a file's hash value is
in the range of a subvol then it will be stored in that subvol.)


I quickly looked at the code. gf_defrag_start_crawl() appears to
be the function for this operation. I do see code that does file migration
from the code path, but debugging code shows that those "linkto" files
seem to be not created by gf_defrag_start_crawl(). I'm not that familar with
the code detail and the theory so I'm not sure who created those
"linkto" files and why the "linkto" file are created.


Thanks,
Paul
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140910/6c74a58a/attachment.html>