[Gluster-users] Problem rebalancing a distributed volume
Kyle Johnson
kjohnson at gnulnx.net
Thu Jul 14 05:47:17 UTC 2016
Hi Susant,
The cause of the issue seems to be that gluster on freebsd is unable to
correctly determine the available disk space; it is reporting that I
have 12.6PB of total disk space, when in reality I have only 40T. As
such, the bricks on that server have much more weight than those on the
other server. As a temporary workaround, I disabled
cluster.weighted-rebalance.
Output from CentOS (192.168.110.2):
[kyle at colossus ~]$ sudo gluster volume status ftp detail
Status of volume: ftp
------------------------------------------------------------------------------
Brick : Brick 192.168.110.1:/tank/bricks/ftp
TCP Port : 49159
RDMA Port : 0
Online : Y
Pid : 69547
File System : N/A
Device : N/A
Mount Options : N/A
Inode Size : N/A
Disk Space Free : 2.5PB
Total Disk Space : 12.6PB
Inode Count : 21097228996
Free Inodes : 21087359717
------------------------------------------------------------------------------
Brick : Brick 192.168.110.2:/ftp/bricks/ftp
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 1087
File System : zfs
Device : storage/bricks
Mount Options : rw,noatime
Inode Size : N/A
Disk Space Free : 39.3TB
Total Disk Space : 46.3TB
Inode Count : 84298675428
Free Inodes : 84297711138
[kyle at colossus ~]$ sudo zfs list storage/bricks
NAME USED AVAIL REFER MOUNTPOINT
storage/bricks 7.05T 39.3T 7.05T /ftp/bricks
Output from FreeBSD (192.168.110.1):
$ zfs list -r tank
NAME USED AVAIL REFER MOUNTPOINT
tank 40.5T 9.82T 153K /tank
tank/bricks 40.5T 9.82T 40.5T /tank/bricks
As you can see, the Disk Space Free and Total Disk Space is way off. In
addition, gluster is not able to determine the file system, device, and
mount options.
At least regarding the device and mount options, it looks like
cli-command-volume.c only checks linux hosts (see
https://github.com/gluster/glusterfs/blob/70fd68d94f768c098b3178c151fa92c5079a8cfd/cli/src/cli-cmd-volume.c#L1980
and
https://github.com/gluster/glusterfs/blob/70fd68d94f768c098b3178c151fa92c5079a8cfd/cli/src/cli-cmd-volume.c#L1996)
I'll follow up to this thread as I find more information.
Kyle
On 07/06/2016 11:43 PM, Susant Palai wrote:
> Hi,
> Please pass on the rebalance log from the 1st server for more analysis which can be found under /var/log/glusterfs/"$VOL-rebalance.log".
> And also we need the current layout xattrs from both the bricks, which can be extracted by the following command.
> "getfattr -m . -de hex <$BRICK_PATH>".
>
> Thanks,
> Susant
>
> ----- Original Message -----
>> From: "Kyle Johnson" <kjohnson at gnulnx.net>
>> To: gluster-users at gluster.org
>> Sent: Tuesday, 5 July, 2016 10:58:09 PM
>> Subject: [Gluster-users] Problem rebalancing a distributed volume
>>
>>
>>
>>
>> Hello everyone,
>>
>> I am having trouble with a distributed volume. In short, the rebalance
>> command does not seem to work for me: Existing files are not migrated, and
>> new files are not created on the new brick.
>>
>> I am running glusterfs 3.7.6 on two servers:
>>
>> 1) FreeBSD 10.3-RELEASE (colossus2 - 192.168.110.1)
>> 2) CentOS 6.7 (colossus - 192.168.110.2)
>>
>> The bricks are zfs-backed on both servers, and the network consists of two
>> direct-connected cat6 cables on 10gig NICs. The NICs are bonded (lagg'd)
>> together with mode 4 (LACP).
>>
>> Here is what I am seeing:
>>
>> root at colossus ~]# gluster volume create fubar 192.168.110.2:/ftp/bricks/fubar
>> volume create: fubar: success: please start the volume to access data
>> [root at colossus ~]# gluster volume start fubar
>> volume start: fubar: success
>> [root at colossus ~]# mount -t glusterfs 192.168.110.2:/fubar /mnt/test
>> [root at colossus ~]# touch /mnt/test/file{1..100}
>> [root at colossus ~]# ls / mnt/test / | wc -l
>> 100
>> [root at colossus ~]# ls /ftp/bricks/fubar | wc -l
>> 100
>>
>> # So far, so good.
>>
>> [root at colossus ~]# gluster volume add-brick fubar
>> 192.168.110.1:/tank/bricks/fubar
>> volume add-brick: success
>>
>> # For good measure, I'll do an explicit fix-layout first.
>>
>> [root at colossus ~]# gluster volume rebalance fubar fix-layout start
>> volume rebalance: fubar: success: Rebalance on fubar has been started
>> successfully. Use rebalance status command to check status of the rebalance
>> process.
>> ID: 2da23238-dbe4-4759-97b2-08879db271e7
>>
>> [root at colossus ~]# gluster volume rebalance fubar status
>> Node Rebalanced-files size scanned failures skipped status run time in secs
>> --------- ----------- ----------- ----------- ----------- -----------
>> ------------ --------------
>> localhost 0 0Bytes 0 0 0 fix-layout completed 0.00
>> 192.168.110.1 0 0Bytes 0 0 0 fix-layout completed 0.00
>> volume rebalance: fubar: success
>>
>> # Now to do the actual rebalance.
>>
>> [root at colossus ~]# gluster volume rebalance fubar start
>> volume rebalance: fubar: success: Rebalance on fubar has been started
>> successfully. Use rebalance status command to check status of the rebalance
>> process.
>> ID: 67160a67-01b2-4a51-9a11-114aa6269ee9
>>
>> [root at colossus ~]# gluster volume rebalance fubar status
>> Node Rebalanced-files size scanned failures skipped status run time in secs
>> --------- ----------- ----------- ----------- ----------- -----------
>> ------------ --------------
>> localhost 0 0Bytes 100 0 0 completed 0.00
>> 192.168.110.1 0 0Bytes 0 0 0 completed 0.00
>> volume rebalance: fubar: success
>> [root at colossus ~]# ls / mnt/test / | wc -l
>> 101
>> [root at colossus ~]# ls / ftp/bricks/fubar / | wc -l
>> 100
>>
>> # As the output shows, 100 files were scanned, but none were moved.
>>
>> # And for another test, I'll create 100 new post-fix-layout files
>>
>> [root at colossus ~]# touch /mnt/test/file{101..200}
>> [root at colossus ~]# ls / ftp/bricks/fubar / | wc -l
>> 199
>>
>>
>> # And as you can see here, they were all created on the first server. The
>> second server isn't touched at all.
>>
>>
>> Not sure if this is relevant, but if I create the volume with both bricks to
>> begin with, files are properly distributed.
>>
>> Thanks!
>> Kyle
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list