[Bugs] [Bug 1473133] New: dht/cluster: rebalance/ remove-brick should honor min-free-disk
bugzilla at redhat.com
bugzilla at redhat.com
Thu Jul 20 05:49:31 UTC 2017
https://bugzilla.redhat.com/show_bug.cgi?id=1473133
Bug ID: 1473133
Summary: dht/cluster: rebalance/remove-brick should honor
min-free-disk
Product: GlusterFS
Version: 3.10
Component: distribute
Assignee: spalai at redhat.com
Reporter: spalai at redhat.com
CC: bugs at gluster.org, nbalacha at redhat.com
Depends On: 1441508
Blocks: 1360317, 1473132
+++ This bug was initially created as a clone of Bug #1441508 +++
Description of problem:
Currently rebalance/remove-brick process does not honor min-free-disk. Hence,
often making the disk full scenario.
As part of solution they should check if brick can accommodate file within the
min-free-disk boundary. If not, then it should find a new brick which has good
enough space (obviously min-free-disk limit checked) to migrate the file.
--- Additional comment from Worker Ant on 2017-04-12 11:57:41 MVT ---
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor
min-free-disk) posted (#5) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-12 12:13:47 MVT ---
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor
min-free-disk) posted (#6) for review on master by Atin Mukherjee
(amukherj at redhat.com)
--- Additional comment from Worker Ant on 2017-04-12 14:07:32 MVT ---
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor
min-free-disk) posted (#7) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-12 16:24:16 MVT ---
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor
min-free-disk) posted (#8) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-12 17:10:50 MVT ---
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor
min-free-disk) posted (#9) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-12 19:23:51 MVT ---
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor
min-free-disk) posted (#10) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-12 22:13:06 MVT ---
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor
min-free-disk) posted (#11) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-13 10:33:14 MVT ---
REVIEW: https://review.gluster.org/17034 (cluster/dht: Make rebalance honor
min-free-disk) posted (#12) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-13 14:57:45 MVT ---
COMMIT: https://review.gluster.org/17034 committed in master by Raghavendra G
(rgowdapp at redhat.com)
------
commit c2ea4463ab4ce04936a824f7e8c7c133aabe1381
Author: Susant Palai <spalai at redhat.com>
Date: Tue Apr 11 17:27:17 2017 +0530
cluster/dht: Make rebalance honor min-free-disk
test: Manual
created files of size 1K on 2 brick(of size 1GB) setup .
added a brick of size 16GB.
set min-free-disk to 12GB(so that first two bricks won't receive any
files).
removed one of the 1st brick of size 1GB.
Logs from test:
[2017-04-12 08:52:08.196484] W [MSGID: 0]
[dht-rebalance.c:895:__dht_check_free_space]
0-test1-dht: Write will cross min-free-disk for file - /tile32 on subvol -
test1-client-1.
Looking for new subvol.
[2017-04-12 08:52:08.196904] I [MSGID: 0]
[dht-rebalance.c:925:__dht_check_free_space]
0-test1-dht: new target found - test1-client-2 for file - /tile32
- Post migration we have two files. The new destination (/brick/1) has the
data file
[root at vm1 ~]# ll /brick/1/tile32
-rw-r--r--. 2 root root 0 Apr 12 14:22 /brick/1/tile32
- On the old target the linkto file is there with linkto xattr pointing to
/brick/1
[root at vm1 ~]# ll /tmp/2/tile32
---------T. 2 root root 1000 Apr 12 14:22 /tmp/2/tile32
[root at vm1 ~]# getfattr -m . -de text /tmp/2/tile32
getfattr: Removing leading '/' from absolute path names
security.selinux="unconfined_u:object_r:user_tmp_t:s0"
trusted.gfid="����:Aс�#�/'b2"
trusted.glusterfs.dht.linkto="test1-client-2"
Marking ./tests/features/worm_sh.t as bad test.
Reason being, this patch failed on master branch as well and it has nothing
to do with rebalance/remove-brick.
BUG: 1441508
Change-Id: I90bae251cda3d957a49cdceda90cd08311a392fb
Signed-off-by: Susant Palai <spalai at redhat.com>
Reviewed-on: https://review.gluster.org/17034
Smoke: Gluster Build System <jenkins at build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
Reviewed-by: Amar Tumballi <amarts at redhat.com>
Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
--- Additional comment from Worker Ant on 2017-04-17 12:35:26 MVT ---
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source
for min-free-disk check) posted (#1) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-17 12:41:54 MVT ---
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source
for min-free-disk check) posted (#2) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-17 12:45:39 MVT ---
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source
for min-free-disk check) posted (#3) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-17 14:52:21 MVT ---
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source
for min-free-disk check) posted (#4) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-17 15:09:08 MVT ---
REVIEW: https://review.gluster.org/17064 (cluster/dht: ignore rebalance source
for min-free-disk check) posted (#5) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-18 12:27:50 MVT ---
REVIEW: https://review.gluster.org/17064 (cluster/dht: Skip file migration if
the subvol that meets min-free-disk criteria happens to be the same subvol
containing data-file) posted (#6) for review on master by Susant Palai
(spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-19 10:35:18 MVT ---
COMMIT: https://review.gluster.org/17064 committed in master by Raghavendra G
(rgowdapp at redhat.com)
------
commit b350bcd6a3db2e92d1baa47a5ec02efd09f76f16
Author: Susant Palai <spalai at redhat.com>
Date: Mon Apr 17 13:00:54 2017 +0530
cluster/dht: Skip file migration if the subvol that meets min-free-disk
criteria happens to be the same subvol containing data-file
Rebalance need to figure out a new subvol in case the hashed subvol
does not have enough space. In the process of figuring out the new subvol,
we need to ignore the source subvol, otherwise it will lead to data loss.
Test: Manual
Ran the following
sizeof /tmp/1: 1.5GB
sizeof /brick/1: 16GB
sizeof /tmp/2: 1.5GB
<start>
glusterd; gluster v create test1 vm1:/brick/1 vm1:/tmp/1;
gluster v start test1;
mount -t glusterfs vm1:test1 /mnt;
for i in {1..2000}
do
dd if=/dev/zero of=/mnt/file$i bs=1KB count=1 &> /dev/null;
done
gluster v add-brick test1 vm1:/tmp/2
gluster v set test1 min-free-disk 12GB
gluster v remove-brick test1 vm1:/tmp/1 star
<end>
file count and data were intact.
Change-Id: Ib8fc8467a3d48a7c12958824c4f0b88e160b86c1
BUG: 1441508
Signed-off-by: Susant Palai <spalai at redhat.com>
Reviewed-on: https://review.gluster.org/17064
Smoke: Gluster Build System <jenkins at build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
--- Additional comment from Worker Ant on 2017-04-24 16:06:58 MVT ---
REVIEW: https://review.gluster.org/17104 (cluster/dht: correct space check for
rebalance) posted (#1) for review on master by Susant Palai (spalai at redhat.com)
--- Additional comment from Worker Ant on 2017-04-25 21:28:22 MVT ---
COMMIT: https://review.gluster.org/17104 committed in master by Raghavendra G
(rgowdapp at redhat.com)
------
commit dba51428dc16762a9489249f6d9b2e64c0b8303e
Author: Susant Palai <spalai at redhat.com>
Date: Mon Apr 24 16:27:23 2017 +0530
cluster/dht: correct space check for rebalance
With rebalance doing fallocate on destination, we don't need to
add file size to the "destination available space" to decide whether
to migrate the file or not.
Notes: Fallocate would have already occupied the file size space on
destination
Change-Id: If7f6a6654e6257726680cf20d618482a6e9095a6
BUG: 1441508
Signed-off-by: Susant Palai <spalai at redhat.com>
Reviewed-on: https://review.gluster.org/17104
Smoke: Gluster Build System <jenkins at build.gluster.org>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
Reviewed-by: Amar Tumballi <amarts at redhat.com>
Reviewed-by: N Balachandran <nbalacha at redhat.com>
Reviewed-by: Raghavendra G <rgowdapp at redhat.com>
--- Additional comment from Worker Ant on 2017-05-16 09:58:51 MVT ---
REVIEW: https://review.gluster.org/17305 (cluster/dht: Fix crash in dht rmdir)
posted (#1) for review on master by N Balachandran (nbalacha at redhat.com)
--- Additional comment from Nithya Balachandran on 2017-05-16 10:05:26 MVT ---
(In reply to Worker Ant from comment #19)
> REVIEW: https://review.gluster.org/17305 (cluster/dht: Fix crash in dht
> rmdir) posted (#1) for review on master by N Balachandran
> (nbalacha at redhat.com)
Please ignore this - I had entered the wrong BZ for the patch.
--- Additional comment from Shyamsundar on 2017-05-30 23:49:52 MVT ---
This bug is getting closed because a release has been made available that
should address the reported issue. In case the problem is still not fixed with
glusterfs-3.11.0, please open a new bug report.
glusterfs-3.11.0 has been announced on the Gluster mailinglists [1], packages
for several distributions should become available in the near future. Keep an
eye on the Gluster Users mailinglist [2] and the update infrastructure for your
distribution.
[1] http://lists.gluster.org/pipermail/announce/2017-May/000073.html
[2] https://www.gluster.org/pipermail/gluster-users/
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1360317
[Bug 1360317] [GSS] glusterfs doesn't respect cluster.min-free-disk on
remove-brick operation
https://bugzilla.redhat.com/show_bug.cgi?id=1441508
[Bug 1441508] dht/cluster: rebalance/remove-brick should honor
min-free-disk
https://bugzilla.redhat.com/show_bug.cgi?id=1473132
[Bug 1473132] dht/cluster: rebalance/remove-brick should honor
min-free-disk
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=o92G5EIPwt&a=cc_unsubscribe
More information about the Bugs
mailing list