[Bugs] [Bug 1216898] New: Data Tiering: Volume inconsistency errors getting logged when attaching uneven(odd) number of hot bricks in hot tier(pure distribute tier layer) to a dist-rep volume

Wed Apr 29 07:18:36 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1216898

            Bug ID: 1216898
           Summary: Data Tiering: Volume inconsistency errors getting
                    logged when attaching uneven(odd) number of hot bricks
                    in hot tier(pure distribute tier layer) to a dist-rep
                    volume
           Product: GlusterFS
           Version: mainline
         Component: tiering
          Assignee: bugs at gluster.org
          Reporter: nchilaka at redhat.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org

Description of problem:
=======================
When we try to attach a hot tier with only distribute(no replica set in hot
layer) hot bricks, with the distribute being odd number of bricks(say 1 hot
brick only), then there are errors displayed for volume inconsistency as below:

Also it can be seen that the number of bricks formula is wrong(though i guess
this may be fixed as part of displaying hot and cold tier  seperately)
Number of Bricks: 2 x 2 = 5

[2015-04-29 07:07:42.214122] I
[glusterd-brick-ops.c:388:__glusterd_handle_add_brick] 0-management: Received
add brick req
[2015-04-29 07:07:42.218608] E
[glusterd-brick-ops.c:1343:glusterd_op_stage_add_brick] 0-management: Unable to
get replica count
[2015-04-29 07:07:42.231549] E [run.c:190:runner_log] (-->
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x186)[0x7f9f6dda7116] (-->
/lib64/libglusterfs.so.0(runner_log+0xfc)[0x7f9f6ddf419c] (-->
/usr/lib64/glusterfs/3.7.0alpha0/xlator/mgmt/glusterd.so(glusterd_hooks_run_hooks+0x47a)[0x7f9f62c181ba]
(-->
/usr/lib64/glusterfs/3.7.0alpha0/xlator/mgmt/glusterd.so(+0x51e15)[0x7f9f62b99e15]
(-->
/usr/lib64/glusterfs/3.7.0alpha0/xlator/mgmt/glusterd.so(glusterd_op_commit_perform+0x46)[0x7f9f62ba0176]
))))) 0-management: Failed to execute script:
/var/lib/glusterd/hooks/1/add-brick/pre/S28Quota-enable-root-xattr-heal.sh
--volname=vol200 --version=1 --volume-op=add-brick
--gd-workdir=/var/lib/glusterd
[2015-04-29 07:07:42.231625] I
[glusterd-brick-ops.c:1108:glusterd_op_perform_add_bricks] 0-management: type
is set 5, need to change it
[2015-04-29 07:07:42.237710] E
[glusterd-volgen.c:2713:volgen_graph_build_clients] 0-: volume inconsistency:
total number of bricks (5) is not divisible with number of bricks per cluster
(2) in a multi-cluster setup
[2015-04-29 07:07:42.237741] E
[glusterd-volgen.c:4976:glusterd_create_volfiles] 0-management: Could not
generate trusted client volfiles
[2015-04-29 07:07:42.237752] E
[glusterd-brick-ops.c:1919:glusterd_op_add_brick] 0-: Unable to add bricks
[2015-04-29 07:07:42.237770] E [glusterd-syncop.c:1330:gd_commit_op_phase]
0-management: Commit of operation 'Volume Add brick' failed on localhost    
[2015-04-29 07:07:47.096802] I
[glusterd-handler.c:1317:__glusterd_handle_cli_get_volume] 0-glusterd: Received
get vol req

Version-Release number of selected component (if applicable):
============================================================
glusterfs 3.7.0alpha0 built on Apr 28 2015 01:37:11
Repository revision: git://git.gluster.com/glusterfs.git
Copyright (c) 2006-2011 Gluster Inc. <http://www.gluster.com>
GlusterFS comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GlusterFS under the terms of the GNU General
Public License.
[root at yarrow ~]# rpm -qa|grep gluster
glusterfs-libs-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-server-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-fuse-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-api-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
glusterfs-cli-3.7.0alpha0-0.17.gited96153.el7.centos.x86_64
[root at yarrow ~]# 

Steps to Reproduce:
==================
1.create a dist-rep volume and start the volume
[root at yarrow ~]# gluster v create vol200 replica 2
moonshine:/brick_100G_9/vol200 yarrow:/brick_100G_9/vol200
moonshine:/brick_100G_10/vol200 yarrow:/brick_100G_10/vol200 force
volume create: vol200: success: please start the volume to access data
[root at yarrow ~]# gluster v info vol200

Volume Name: vol200
Type: Distributed-Replicate
Volume ID: bb0c179a-c466-4137-a8bd-110788481aa1
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: moonshine:/brick_100G_9/vol200
Brick2: yarrow:/brick_100G_9/vol200
Brick3: moonshine:/brick_100G_10/vol200
Brick4: yarrow:/brick_100G_10/vol200
[root at yarrow ~]# gluster v start vol200
volume start: vol200: success

2.attach a tier with only one hot brick and view the logs

[root at yarrow ~]# gluster v attach-tier vol200 yarrow:/ssdbricks_50G_1/vol200
volume add-brick: failed: Commit failed on localhost. Please check the log file
for more details.
[root at yarrow ~]# gluster v info vol200

Volume Name: vol200
Type: Tier
Volume ID: bb0c179a-c466-4137-a8bd-110788481aa1
Status: Started
Number of Bricks: 2 x 2 = 5
Transport-type: tcp
Bricks:
Brick1: yarrow:/ssdbricks_50G_1/vol200
Brick2: moonshine:/brick_100G_9/vol200
Brick3: yarrow:/brick_100G_9/vol200
Brick4: moonshine:/brick_100G_10/vol200
Brick5: yarrow:/brick_100G_10/vol200

Actual results:

Expected results:

Additional info:

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.