[Bugs] [Bug 1392853] New: Hosted Engine VM paused post replace-brick operation

bugzilla at redhat.com bugzilla at redhat.com
Tue Nov 8 11:18:05 UTC 2016


https://bugzilla.redhat.com/show_bug.cgi?id=1392853

            Bug ID: 1392853
           Summary: Hosted Engine VM paused post replace-brick operation
           Product: GlusterFS
           Version: 3.7.15
         Component: sharding
          Keywords: Triaged
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: kdhananj at redhat.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org, rhinduja at redhat.com,
                    rhs-bugs at redhat.com, sasundar at redhat.com,
                    storage-qa-internal at redhat.com
        Depends On: 1370350, 1392445
            Blocks: 1392844, 1392846



+++ This bug was initially created as a clone of Bug #1392445 +++

+++ This bug was initially created as a clone of Bug #1370350 +++

Description of problem:
-----------------------
After replacing the defunct brick of replica 3 sharded volume, the hosted
engine VM running with its image on that volume went to paused state.

Fuse mount logs showed EIO

Version-Release number of selected component (if applicable):
--------------------------------------------------------------
RHGS 3.1.3
RHV 4.0.2

How reproducible:
-----------------
1/1

Steps to Reproduce:
-------------------
1. Create a replica 3 sharded volume optimized for VM store
2. Create a hosted engine VM with this volume as a 'data domain'
3. After hosted-engine up and operational, kill one of the bricks of the volume
4. Add a new node to the cluster
5. Replace the old-brick with the new-brick from the newly added node to the
cluster

Actual results:
---------------
hosted-engine vm went to paused state, fuse mount showed EIO with error 'Lookup
on shard 3 failed.'

Expected results:
-----------------
There shouldn't be any error messages, after performing replace brick operation

--- Additional comment from SATHEESARAN on 2016-08-25 22:38:05 EDT ---


1. Setup details
----------------
   4 Nodes in the cluster, all running RHGS 3.1.3 ( glusterfs-3.7.9.10.el7rhgs
)
   Hosts are :
   cambridge.lab.eng.blr.redhat.com ( 10.70.37.73 )
   zod.lab.eng.blr.redhat.com ( 10.70.37.76 )
   tettnang.lab.eng.blr.redhat.com ( 10.70.37.77 ) <-- This host is defunct for
more than 12 hours

   ** Here is the newly added host ** :
   yarrow.lab.eng.blr.redhat.com ( 10.70.37.78 )


2. Peer status
--------------

[root at cambridge ~]# gluster pe s
Number of Peers: 3

Hostname: tettnang-nic2.lab.eng.blr.redhat.com
Uuid: d9cd9f98-6dc3-436f-b6ca-aab0b059fc41
State: Peer in Cluster (Disconnected)
Other names:
10.70.36.77

Hostname: 10.70.36.76
Uuid: 7bd8c3ea-4b88-431f-b62e-a330b9ae6b9a
State: Peer in Cluster (Connected)

Hostname: yarrow.lab.eng.blr.redhat.com
Uuid: 3d98430d-50f3-4aa8-9389-ac133a58c9b3
State: Peer in Cluster (Connected)

----------

[root at cambridge ~]# gluster pool list
UUID                    Hostname                                State
d9cd9f98-6dc3-436f-b6ca-aab0b059fc41    tettnang-nic2.lab.eng.blr.redhat.com   
Disconnected 
7bd8c3ea-4b88-431f-b62e-a330b9ae6b9a    10.70.36.76                            
Connected 
3d98430d-50f3-4aa8-9389-ac133a58c9b3    yarrow.lab.eng.blr.redhat.com          
Connected 
3c3ad2a9-e8f8-496b-bdde-1c0777552dee    localhost                              
Connected 

3. Gluster volume details
-------------------------
[root at cambridge ~]# gluster volume info enginevol

Volume Name: enginevol
Type: Replicate
Volume ID: 46a261df-f527-479c-9776-4bdb21fe19b1
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.70.36.73:/rhgs/engine/enginebrick
Brick2: 10.70.36.76:/rhgs/engine/enginebrick
Brick3: yarrow.lab.eng.blr.redhat.com:/rhgs/engine/enginebrick <-- new brick
Options Reconfigured:
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
cluster.eager-lock: enable
network.remote-dio: off
cluster.quorum-type: auto
cluster.server-quorum-type: server
storage.owner-uid: 36
storage.owner-gid: 36
features.shard: on
features.shard-block-size: 512MB
performance.low-prio-threads: 32
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
performance.strict-o-direct: on
network.ping-timeout: 30
user.cifs: off
nfs.disable: on

Note: the old brick was - 10.70.36.77:/rhgs/engine/enginebrick

4. volume status
----------------
[root at cambridge ~]# gluster volume status enginevol
Status of volume: enginevol
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.70.36.73:/rhgs/engine/enginebrick  49152     0          Y       18194
Brick 10.70.36.76:/rhgs/engine/enginebrick  49152     0          Y       15595
Brick yarrow.lab.eng.blr.redhat.com:/rhgs/e
ngine/enginebrick                           49152     0          Y       19457
Self-heal Daemon on localhost               N/A       N/A        Y       12863
Self-heal Daemon on yarrow.lab.eng.blr.redh
at.com                                      N/A       N/A        Y       19462
Self-heal Daemon on 10.70.36.76             N/A       N/A        Y       32641

Task Status of Volume enginevol
------------------------------------------------------------------------------
There are no active volume tasks

--- Additional comment from SATHEESARAN on 2016-08-25 22:39:22 EDT ---

self-heal and split-brain info

[root at cambridge ~]# gluster volume heal enginevol info
Brick 10.70.36.73:/rhgs/engine/enginebrick
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.81 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.1 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.6 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.24 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.3 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.72 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.84 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.18 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.7 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.8 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.36 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.4 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.9 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.57 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.39 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.12 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.90 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.42 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.10 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.96 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.21 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.87 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.45 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.99 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.48 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.60 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.15 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.27 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.54 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.69 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.78 
Status: Connected
Number of entries: 31

Brick 10.70.36.76:/rhgs/engine/enginebrick
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.81 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.1 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.6 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.24 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.3 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.72 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.84 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.18 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.7 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.8 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.36 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.4 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.9 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.57 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.39 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.12 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.90 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.42 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.10 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.96 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.21 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.87 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.45 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.99 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.48 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.60 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.15 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.27 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.54 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.69 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.78 
Status: Connected
Number of entries: 31

Brick yarrow.lab.eng.blr.redhat.com:/rhgs/engine/enginebrick
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.81 
/.shard/853758b3-f79d-4114-86b4-c9e4fe1f97db.3 
Status: Connected
Number of entries: 2

[root at cambridge ~]# gluster volume heal enginevol info split-brain
Brick 10.70.36.73:/rhgs/engine/enginebrick
Status: Connected
Number of entries in split-brain: 0

Brick 10.70.36.76:/rhgs/engine/enginebrick
Status: Connected
Number of entries in split-brain: 0

Brick yarrow.lab.eng.blr.redhat.com:/rhgs/engine/enginebrick
Status: Connected
Number of entries in split-brain: 0

--- Additional comment from SATHEESARAN on 2016-08-25 22:45:40 EDT ---

[2016-08-25 08:16:44.373211] W [MSGID: 114031]
[client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-0: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid
argument]
[2016-08-25 08:16:44.373283] W [MSGID: 114031]
[client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-1: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid
argument]
[2016-08-25 08:16:44.373343] W [MSGID: 114031]
[client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-2: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid
argument]
[2016-08-25 08:16:44.374685] E [MSGID: 133010]
[shard.c:1582:shard_common_lookup_shards_cbk] 2-enginevol-shard: Lookup on
shard 3 failed. Base file gfid = 853758b3-f79d-4114-86b4-c9e4fe1f97db
[Input/output error]
[2016-08-25 08:16:44.374734] W [fuse-bridge.c:2224:fuse_readv_cbk]
0-glusterfs-fuse: 3986689: READ => -1 (Input/output error)
[2016-08-25 08:37:22.183269] W [MSGID: 114031]
[client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-0: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid
argument]
[2016-08-25 08:37:22.183355] W [MSGID: 114031]
[client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-1: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid
argument]
[2016-08-25 08:37:22.183414] W [MSGID: 114031]
[client-rpc-fops.c:2974:client3_3_lookup_cbk] 2-enginevol-client-2: remote
operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Invalid
argument]
[2016-08-25 08:37:22.184586] E [MSGID: 133010]
[shard.c:1582:shard_common_lookup_shards_cbk] 2-enginevol-shard: Lookup on
shard 3 failed. Base file gfid = 853758b3-f79d-4114-86b4-c9e4fe1f97db
[Input/output error]
[2016-08-25 08:37:22.184625] W [fuse-bridge.c:2224:fuse_readv_cbk]
0-glusterfs-fuse: 4023987: READ => -1 (Input/output error)

--- Additional comment from SATHEESARAN on 2016-08-25 22:46:06 EDT ---

comment3 is the snip from fuse mount.log

--- Additional comment from SATHEESARAN on 2016-08-30 23:05:51 EDT ---

I couldn't hit back for the second time, when Krutika asked for debug enabled
logs.

Later, Krutika also confirmed that the community user too seeing this problem

--- Additional comment from Krutika Dhananjay on 2016-09-13 09:58:51 EDT ---

(In reply to SATHEESARAN from comment #5)
> I couldn't hit back for the second time, when Krutika asked for debug
> enabled logs.
> 
> Later, Krutika also confirmed that the community user too seeing this problem

I stand corrected. I figured later that it was with granular-entry-heal enabled
and a case where the same brick was wiped off and healed. The issue there was a
combination of dated documentation and the lack of reset-brick functionality.

I did see logs of the kind you have pasted in comment #3, of lookups failing
with EINVAL but no input/output errors.

-Krutika

--- Additional comment from Krutika Dhananjay on 2016-09-13 10:06:52 EDT ---

Sas,

Do you have the logs from this run, of the bricks, shds and the clients?

-Krutika

--- Additional comment from SATHEESARAN on 2016-10-14 06:55:56 EDT ---

(In reply to Krutika Dhananjay from comment #7)
> Sas,
> 
> Do you have the logs from this run, of the bricks, shds and the clients?
> 
> -Krutika

Hi Krutika,

I have missed the logs.
I will try to reproduce this issue.

But could you guess any problems with the error messages in comment3 ?

--- Additional comment from Krutika Dhananjay on 2016-10-24 03:08:04 EDT ---

Not quite, Sas. The EINVAL seems to be getting propagated by the brick(s) since
protocol/client which is the lowest layer in the client stack is receiving
EINVAL from over the network.

-Krutika

--- Additional comment from Red Hat Bugzilla Rules Engine on 2016-11-07
05:23:09 EST ---

This bug is automatically being provided 'pm_ack+' for the release flag
'rhgs‑3.2.0', the current release of Red Hat Gluster Storage 3 under active
development, having been appropriately marked for the release, and having been
provided ACK from Development and QE

If the 'blocker' flag had been proposed/set on this BZ, it has now been unset,
since the 'blocker' flag is not valid for the current phase of RHGS 3.2.0
development

--- Additional comment from Worker Ant on 2016-11-07 09:17:21 EST ---

REVIEW: http://review.gluster.org/15788 (features/shard: Fill loc.pargfid too
for named lookups on individual shards) posted (#1) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)

--- Additional comment from Worker Ant on 2016-11-08 03:45:10 EST ---

REVIEW: http://review.gluster.org/15788 (features/shard: Fill loc.pargfid too
for named lookups on individual shards) posted (#2) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)

--- Additional comment from Worker Ant on 2016-11-08 06:05:31 EST ---

COMMIT: http://review.gluster.org/15788 committed in master by Pranith Kumar
Karampuri (pkarampu at redhat.com) 
------
commit e9023083b3a165390a8cc8fc77253f354744e81a
Author: Krutika Dhananjay <kdhananj at redhat.com>
Date:   Mon Nov 7 16:06:56 2016 +0530

    features/shard: Fill loc.pargfid too for named lookups on individual shards

    On a sharded volume when a brick is replaced while IO is going on, named
    lookup on individual shards as part of read/write was failing with
    ENOENT on the replaced brick, and as a result AFR initiated name heal in
    lookup callback. But since pargfid was empty (which is what this patch
    attempts to fix), the resolution of the shards by protocol/server used
    to fail and the following pattern of logs was seen:

    Brick-logs:

    [2016-11-08 07:41:49.387127] W [MSGID: 115009]
    [server-resolve.c:566:server_resolve] 0-rep-server: no resolution type
    for (null) (LOOKUP)
    [2016-11-08 07:41:49.387157] E [MSGID: 115050]
    [server-rpc-fops.c:156:server_lookup_cbk] 0-rep-server: 91833: LOOKUP(null)
   
(00000000-0000-0000-0000-000000000000/16d47463-ece5-4b33-9c93-470be918c0f6.82)
    ==> (Invalid argument) [Invalid argument]

    Client-logs:
    [2016-11-08 07:41:27.497687] W [MSGID: 114031]
    [client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-0: remote
    operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
    [Invalid argument]
    [2016-11-08 07:41:27.497755] W [MSGID: 114031]
    [client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-1: remote
    operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
    [Invalid argument]
    [2016-11-08 07:41:27.498500] W [MSGID: 114031]
    [client-rpc-fops.c:2930:client3_3_lookup_cbk] 2-rep-client-2: remote
    operation failed. Path: (null) (00000000-0000-0000-0000-000000000000)
    [Invalid argument]
    [2016-11-08 07:41:27.499680] E [MSGID: 133010]

    Also, this patch makes AFR by itself choose a non-NULL pargfid even if
    its ancestors fail to initialize all pargfid placeholders.

    Change-Id: I5f85b303ede135baaf92e87ec8e09941f5ded6c1
    BUG: 1392445
    Signed-off-by: Krutika Dhananjay <kdhananj at redhat.com>
    Reviewed-on: http://review.gluster.org/15788
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
    Reviewed-by: Ravishankar N <ravishankar at redhat.com>
    Reviewed-by: Pranith Kumar Karampuri <pkarampu at redhat.com>
    Smoke: Gluster Build System <jenkins at build.gluster.org>


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1370350
[Bug 1370350] Hosted Engine VM paused post replace-brick operation
https://bugzilla.redhat.com/show_bug.cgi?id=1392445
[Bug 1392445] Hosted Engine VM paused post replace-brick operation
https://bugzilla.redhat.com/show_bug.cgi?id=1392844
[Bug 1392844] Hosted Engine VM paused post replace-brick operation
https://bugzilla.redhat.com/show_bug.cgi?id=1392846
[Bug 1392846] Hosted Engine VM paused post replace-brick operation
-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list