[Bugs] [Bug 1220051] New: Data Tiering: Volume inconsistency errors getting logged when attaching uneven(odd) number of hot bricks in hot tier(pure distribute tier layer) to a dist-rep volume
bugzilla at redhat.com
bugzilla at redhat.com
Sat May 9 13:01:09 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1220051
Bug ID: 1220051
Summary: Data Tiering: Volume inconsistency errors getting
logged when attaching uneven(odd) number of hot bricks
in hot tier(pure distribute tier layer) to a dist-rep
volume
Product: GlusterFS
Version: 3.7.0
Component: tiering
Keywords: Triaged
Severity: urgent
Priority: urgent
Assignee: bugs at gluster.org
Reporter: rkavunga at redhat.com
QA Contact: bugs at gluster.org
CC: bugs at gluster.org, nchilaka at redhat.com,
vagarwal at redhat.com
Depends On: 1216898
Blocks: 1186580 (qe_tracker_everglades), 1199352
(glusterfs-3.7.0)
+++ This bug was initially created as a clone of Bug #1216898 +++
Description of problem:
=======================
When we try to attach a hot tier with only distribute(no replica set in hot
layer) hot bricks, with the distribute being odd number of bricks(say 1 hot
brick only), then there are errors displayed for volume inconsistency as below:
Also it can be seen that the number of bricks formula is wrong(though i guess
this may be fixed as part of displaying hot and cold tier seperately)
Number of Bricks: 2 x 2 = 5
[2015-04-29 07:07:42.214122] I
[glusterd-brick-ops.c:388:__glusterd_handle_add_brick] 0-management: Received
add brick req
[2015-04-29 07:07:42.218608] E
[glusterd-brick-ops.c:1343:glusterd_op_stage_add_brick] 0-management: Unable to
get replica count
[2015-04-29 07:07:42.231549] E [run.c:190:runner_log] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f9f6dda7116] (-->
/lib64/libglusterfs.so.0(runner_log+0xfc)[0x7f9f6ddf419c] (-->
/usr/lib64/glusterfs/3.7.0alpha0/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x47a)[0x7f9f62c181ba]
(-->
/usr/lib64/glusterfs/3.7.0alpha0/xlator/mgmt/glusterd.so(+0x51e15)[0x7f9f62b99e15]
(-->
/usr/lib64/glusterfs/3.7.0alpha0/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x46)[0x7f9f62ba0176]
))))) 0-management: Failed to execute script:
/var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh
--volname=vol200 --version=1 --volume-op=add-brick
--gd-workdir=/var/lib/glusterd
[2015-04-29 07:07:42.231625] I
[glusterd-brick-ops.c:1108:glusterd_op_perform_add_bricks] 0-management: type
is set 5, need to change it
[2015-04-29 07:07:42.237710] E
[glusterd-volgen.c:2713:volgen_graph_build_clients] 0-: volume inconsistency:
total number of bricks (5) is not divisible with number of bricks per cluster
(2) in a multi-cluster setup
[2015-04-29 07:07:42.237741] E
[glusterd-volgen.c:4976:glusterd_create_volfiles] 0-management: Could not
generate trusted client volfiles
[2015-04-29 07:07:42.237752] E
[glusterd-brick-ops.c:1919:glusterd_op_add_brick] 0-: Unable to add bricks
[2015-04-29 07:07:42.237770] E [glusterd-syncop.c:1330:gd_commit_op_phase]
0-management: Commit of operation 'Volume Add brick' failed on localhost
[2015-04-29 07:07:47.096802] I
[glusterd-handler.c:1317:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req
Version-Release number of selected component (if applicable):
============================================================
glusterfs 3.7.0alpha0 built on Apr 28 2015 01:37:11
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
[root at yarrow ~]# rpm -qa|grep gluster
glusterfs-libs-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-server-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-fuse-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-api-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-cli-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
[root at yarrow ~]#
Steps to Reproduce:
==================
1.create a dist-rep volume and start the volume
[root at yarrow ~]# gluster v create vol200 replica 2
moonshine:/brick_100G_9/vol200 yarrow:/brick_100G_9/vol200
moonshine:/brick_100G_10/vol200 yarrow:/brick_100G_10/vol200 force
volume create: vol200: success: please start the volume to access data
[root at yarrow ~]# gluster v info vol200
Volume Name: vol200
Type: Distributed-Replicate
Volume ID: bb0c179a-c466-4137-a8bd-110788481aa1
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: moonshine:/brick_100G_9/vol200
Brick2: yarrow:/brick_100G_9/vol200
Brick3: moonshine:/brick_100G_10/vol200
Brick4: yarrow:/brick_100G_10/vol200
[root at yarrow ~]# gluster v start vol200
volume start: vol200: success
2.attach a tier with only one hot brick and view the logs
[root at yarrow ~]# gluster v attach-tier vol200 yarrow:/ssdbricks_50G_1/vol200
volume add-brick: failed: Commit failed on localhost. Please check the log file
for more details.
[root at yarrow ~]# gluster v info vol200
Volume Name: vol200
Type: Tier
Volume ID: bb0c179a-c466-4137-a8bd-110788481aa1
Status: Started
Number of Bricks: 2 x 2 = 5
Transport-type: tcp
Bricks:
Brick1: yarrow:/ssdbricks_50G_1/vol200
Brick2: moonshine:/brick_100G_9/vol200
Brick3: yarrow:/brick_100G_9/vol200
Brick4: moonshine:/brick_100G_10/vol200
Brick5: yarrow:/brick_100G_10/vol200
Actual results:
Expected results:
Additional info:
--- Additional comment from nchilaka on 2015-04-29 03:28:03 EDT ---
logs available at [root at rhsqe-repo sosreports]# pwd
/home/repo/sosreports/1216898
--- Additional comment from Anand Avati on 2015-05-02 09:51:28 EDT ---
REVIEW: http://review.gluster.org/10494 (tiering: Remove unwanted check for
tier type volume) posted (#1) for review on master by mohammed rafi kc
(rkavunga at redhat.com)
--- Additional comment from Anand Avati on 2015-05-04 11:53:09 EDT ---
REVIEW: http://review.gluster.org/10494 (tiering: Remove unwanted check for
tier type volume) posted (#2) for review on master by mohammed rafi kc
(rkavunga at redhat.com)
--- Additional comment from Anand Avati on 2015-05-05 01:25:01 EDT ---
REVIEW: http://review.gluster.org/10494 (tiering: Remove unwanted check for
tier type volume) posted (#3) for review on master by mohammed rafi kc
(rkavunga at redhat.com)
--- Additional comment from Anand Avati on 2015-05-08 17:56:53 EDT ---
REVIEW: http://review.gluster.org/10494 (tiering: Remove unwanted check for
tier type volume) posted (#4) for review on master by mohammed rafi kc
(rkavunga at redhat.com)
--- Additional comment from Anand Avati on 2015-05-08 23:14:27 EDT ---
COMMIT: http://review.gluster.org/10494 committed in master by Vijay Bellur
(vbellur at redhat.com)
------
commit 5fda96d954dd06dbacc6b2851d4ba058ec1295a2
Author: Mohammed Rafi KC <rkavunga at redhat.com>
Date: Sat May 2 19:19:27 2015 +0530
tiering: Remove unwanted check for tier type volume
Change-Id: I2def61ebf348558e5f6a138265e3329d9a5407a3
BUG: 1216898
Signed-off-by: Mohammed Rafi KC <rkavunga at redhat.com>
Reviewed-on: http://review.gluster.org/10494
Reviewed-by: Dan Lambright <dlambrig at redhat.com>
Tested-by: Gluster Build System <jenkins at build.gluster.com>
Tested-by: NetBSD Build System
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1186580
[Bug 1186580] QE tracker bug for Everglades
https://bugzilla.redhat.com/show_bug.cgi?id=1199352
[Bug 1199352] GlusterFS 3.7.0 tracker
https://bugzilla.redhat.com/show_bug.cgi?id=1216898
[Bug 1216898] Data Tiering: Volume inconsistency errors getting logged when
attaching uneven(odd) number of hot bricks in hot tier(pure distribute tier
layer) to a dist-rep volume
--
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list