[Bugs] [Bug 1511310] New: Test bug-1483058-replace-brick-quorum-validation.t fails inconsistently

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1511310

            Bug ID: 1511310
           Summary: Test bug-1483058-replace-brick-quorum-validation.t
                    fails inconsistently
           Product: GlusterFS
           Version: mainline
         Component: glusterd
          Assignee: bugs at gluster.org
          Reporter: rgowdapp at redhat.com
                CC: bugs at gluster.org



Description of problem:
I ran into this failure [1] during regression runs for patch [2]. On running
the test on my local machine, it fails inconsistently. Failed test was:


TEST 15 (line 49): gluster --mode=script --wignore
--glusterd-sock=/d/backends/1/glusterd/gd.sock
--log-file=/var/log/glusterfs/bug-1483058-replace-brick-quorum-validation.t_cli1.log
volume replace-brick patchy 127.1.1.2:/d/backends/2/patchy1
127.1.1.1:/d/backends/1/patchy1_new commit force
volume replace-brick: failed: Quorum not met. Volume operation not allowed.
./tests/bugs/glusterd/bug-1483058-replace-brick-quorum-validation.t .. 15/15
RESULT 15: 1
./tests/bugs/glusterd/bug-1483058-replace-brick-quorum-validation.t .. Failed
1/15 subtests 

Test Summary Report
-------------------
./tests/bugs/glusterd/bug-1483058-replace-brick-quorum-validation.t (Wstat: 0
Tests: 15 Failed: 1)
  Failed test:  15
Files=1, Tests=15, 39 wallclock secs ( 0.03 usr  0.00 sys +  1.74 cusr  0.98
csys =  2.75 CPU)

On looking at one of glusterd logs, I found:
[2017-11-09 06:06:09.387014]:++++++++++
G_LOG:./tests/bugs/glusterd/bug-1483058-replace-brick-quorum-validation.t:
TEST: 49 gluster --mode=script --wignore
--glusterd-sock=/d/backends/1/glusterd/gd.sock
--log-file=/var/log/glusterfs/bug-1483058-replace-brick-quorum-validation.t_cli1.log
volume replace-brick patchy 127.1.1.2:/d/backends/2/patchy1
127.1.1.1:/d/backends/1/patchy1_new commit force ++++++++++
The message "I [MSGID: 106487]
[glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req" repeated 5 times between [2017-11-09 06:06:03.713593]
and [2017-11-09 06:06:09.371221]
[2017-11-09 06:06:09.511510] I [MSGID: 106505]
[glusterd-replace-brick.c:67:__glusterd_handle_replace_brick] 0-management:
Received replace brick req
[2017-11-09 06:06:09.511673] I [MSGID: 106503]
[glusterd-replace-brick.c:148:__glusterd_handle_replace_brick] 0-management:
Received replace-brick commit force request.
[2017-11-09 06:06:10.205940] E [MSGID: 106001]
[glusterd-replace-brick.c:228:glusterd_op_stage_replace_brick] 0-management:
Server quorum not met. Rejecting operation.
[2017-11-09 06:06:10.205972] W [MSGID: 106122]
[glusterd-mgmt.c:168:gd_mgmt_v3_pre_validate_fn] 0-management: Replace-brick
prevalidation failed.
[2017-11-09 06:06:10.205987] E [MSGID: 106122]
[glusterd-mgmt.c:1036:glusterd_mgmt_v3_pre_validate] 0-management: Pre
Validation failed for operation Replace brick on local node
[2017-11-09 06:06:10.206000] E [MSGID: 106122]
[glusterd-replace-brick.c:660:glusterd_mgmt_v3_initiate_replace_brick_cmd_phases]
0-management: Pre Validation Failed

Note that I didn't find any log related to tmp mount done during replace brick.
Also glustershd.log didn't reflect that replace brick is succeeded. It had the
old brick in graph.

Looking at this log, I fail to understand how [2] could've affected this
failure. I am running tests without [2] just to eliminate [2] as the root
cause. Will report back once tests are complete.

[1] https://build.gluster.org/job/centos6-regression/7327/console
[2] https://review.gluster.org/18681

Version-Release number of selected component (if applicable):
mainline

How reproducible:
inconsistently

Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.