[Gluster-users] Shards and heal question post move-brick

David Gossage dgossage at carouselchecks.com
Wed Jul 20 06:29:43 UTC 2016 

So I have enabled shading on 3.7.11, moved all VM images off and on and
everything seemed fine no issues.

Volume Name: GLUSTER1
Type: Replicate
Volume ID: 167b8e57-28c3-447a-95cc-8410cbdf3f7f
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ccgl1.gl.local:/gluster1/BRICK1/1
Brick2: ccgl2.gl.local:/gluster1/BRICK1/1
Brick3: ccgl4.gl.local:/gluster1/BRICK1/1
Options Reconfigured:
nfs.enable-ino32: off
nfs.addr-namelookup: off
nfs.disable: on
performance.strict-write-ordering: off
cluster.background-self-heal-count: 16
cluster.self-heal-window-size: 1024
server.allow-insecure: on
cluster.server-quorum-type: server
cluster.quorum-type: auto
network.remote-dio: enable
cluster.eager-lock: enable
performance.stat-prefetch: on
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
storage.owner-gid: 36
storage.owner-uid: 36
performance.readdir-ahead: on
features.shard: on
features.shard-block-size: 64MB
diagnostics.brick-log-level: WARNING


I added a new server(ccgl4) to replace one that nic died.  Probed, and
moved brick then kicked off heal.

When I did so on the smaller volume that just hosts my hosted engine it
completed so fast that engine didnt even hiccup.

When I did same procedure on larger volume all VM's paused.  I'm guesing
just too many shards healing at once maybe no biggie.  I let heal go went
to bed for few hours while it healed 600GB of shards.

This morning looks like I may have some split-brain heals going on.  All
VM's started just fine after I unpausd/re-started them as needed.  So far
they have been up services seem ok, nothign seems read only etc.

But I have shards that still seem healing after a bit.
split-brain currently shows 0 entries on all 3 nodes (1 example below)
[root at ccgl1 ~]# gluster volume heal GLUSTER1 info split-brain
Brick ccgl1.gl.local:/gluster1/BRICK1/1
Status: Connected
Number of entries in split-brain: 0

Brick ccgl2.gl.local:/gluster1/BRICK1/1
Status: Connected
Number of entries in split-brain: 0

Brick ccgl4.gl.local:/gluster1/BRICK1/1
Status: Connected
Number of entries in split-brain: 0


However for example I get different results running (and sorry for the
copy/paste storm incoming)

[root at ccgl1 ~]# gluster volume heal GLUSTER1 info
Brick ccgl1.gl.local:/gluster1/BRICK1/1
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47
/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226
/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192
/.shard - Possibly undergoing heal

/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274
/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32
/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49
/.shard - Possibly undergoing heal

Status: Connected
Number of entries: 21

Brick ccgl2.gl.local:/gluster1/BRICK1/1
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274
/.shard - Possibly undergoing heal

/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47
/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47
/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48
/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218
/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95
/.shard - Possibly undergoing heal

Status: Connected
Number of entries: 21

Brick ccgl4.gl.local:/gluster1/BRICK1/1
/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49
/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209
/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217
/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32
Status: Connected
Number of entries: 19

[root at ccgl2 ~]# gluster volume heal GLUSTER1 info
Brick ccgl1.gl.local:/gluster1/BRICK1/1
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47
/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226
/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192
/.shard - Possibly undergoing heal

/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274
/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32
/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49
/.shard/1cec1185-ad25-414b-9ae3-c17b5ef0d064.3
/.shard - Possibly undergoing heal

Status: Connected
Number of entries: 22

Brick ccgl2.gl.local:/gluster1/BRICK1/1
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274
/.shard - Possibly undergoing heal

/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47
/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47
/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48
/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218
/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95
/.shard - Possibly undergoing heal

/.shard/900421ff-b10d-404a-bbe7-173af11a69dd.377
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.248
Status: Connected
Number of entries: 23

Brick ccgl4.gl.local:/gluster1/BRICK1/1
/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49
/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209
/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217
/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.45
/.shard/996ba563-c7c6-4448-9d94-2dee6c90b8c3.200
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.47
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.47
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.311
Status: Connected
Number of entries: 24



[root at ccgl4 ~]# gluster volume heal GLUSTER1 info
Brick ccgl1.gl.local:/gluster1/BRICK1/1
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47
/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226
/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192
/.shard - Possibly undergoing heal

/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274
/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32
/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.123
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.45
/.shard - Possibly undergoing heal

Status: Connected
Number of entries: 23

Brick ccgl2.gl.local:/gluster1/BRICK1/1
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274
/.shard - Possibly undergoing heal

/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47
/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47
/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48
/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218
/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.2
/.shard - Possibly undergoing heal

/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.248
Status: Connected
Number of entries: 23

Brick ccgl4.gl.local:/gluster1/BRICK1/1
/.shard/4c7d44fc-a0c1-413b-8dc4-2abbbe1d4d4f.49
/.shard/91e12c0a-2557-4410-bf7d-834604f221f0.95
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.274
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.48
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.1
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.49
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.49
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.192
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.47
/.shard/76b7fe00-a2b1-4b77-b129-c5643e0cffa7.10
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.201
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.209
/.shard/f9a7f3c5-4c13-4020-b560-1f4f7b1e3c42.47
/.shard/d5a328be-03d0-42f7-a443-248290849e7d.48
/.shard/18843fb4-e31c-4fc3-b519-cc6e5e947813.226
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.217
/.shard/5b708220-7e27-4a27-a2ac-98a6ae7c693d.226
/.shard/241a55ed-f0d5-4dbc-a6ce-ab784a0ba6ff.218
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.32
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.45
/.shard/996ba563-c7c6-4448-9d94-2dee6c90b8c3.200
/.shard/996ba563-c7c6-4448-9d94-2dee6c90b8c3.160
/.shard/585c8361-f6a9-48d4-9295-260ec657de1e.46
Status: Connected
Number of entries: 23


Should I just let it keep going?

*David Gossage*
*Carousel Checks Inc. | System Administrator*
*Office* 708.613.2284
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160720/2ee8a66a/attachment.html>

More information about the Gluster-users mailing list