<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<br>
<div class="moz-cite-prefix">On 07/20/2017 02:20 PM, yayo (j) wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">Hi, </div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">Thank you for the answer and sorry
for delay:</div>
<div class="gmail_quote"><br>
</div>
<div class="gmail_quote">2017-07-19 16:55 GMT+02:00
Ravishankar N <span dir="ltr"><<a
href="mailto:ravishankar@redhat.com" target="_blank"
moz-do-not-send="true">ravishankar@redhat.com</a>></span>:</div>
<div class="gmail_quote"><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 1. What does the glustershd.log
say on all 3 nodes when you run the command? Does it
complain anything about these files?<br>
</div>
</blockquote>
<div><br>
</div>
<div>No, glustershd.log is clean, no extra log after command
on all 3 nodes</div>
</div>
</div>
</div>
</blockquote>
<br>
Could you check if the self-heal daemon on all nodes is connected to
the 3 bricks? You will need to check the glustershd.log for that.<br>
If it is not connected, try restarting the shd using `gluster volume
start engine force`, then launch the heal command like you did
earlier and see if heals happen.<br>
<br>
If it doesn't, please provide the getfattr outputs of the 12 files
from all 3 nodes using `getfattr -d -m . -e hex <i>/gluster/engine/brick/</i>path-to-file`
?<br>
<br>
Thanks,<br>
Ravi<br>
<br>
<blockquote type="cite"
cite="mid:CAGK=3kySygAmyLtfR2JrHBM7MtoB4v6mYifTNk3mE1qwcsFjEg@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 2. Are these 12 files also present
in the 3rd data brick?<br>
</div>
</blockquote>
<div><br>
</div>
<div>I've checked right now: all files exists in all 3
nodes </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> 3. Can you provide the output of
`gluster volume info` for the this volume?</div>
</blockquote>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<blockquote style="margin:0px 0px 0px
40px;border:none;padding:0px">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Volume Name: engine</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Type: Replicate</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Status: Started</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Snapshot Count: 0</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Number of Bricks: 1 x 3 = 3</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Transport-type: tcp</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Bricks:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick1: node01:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick2: node02:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Brick3: node04:/gluster/engine/brick</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>Options Reconfigured:</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>nfs.disable: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.readdir-ahead: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>transport.address-family: inet</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>storage.owner-uid: 36</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.quick-read: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.read-ahead: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.io-cache: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.stat-prefetch: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.low-prio-threads: 32</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>network.remote-dio: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.eager-lock: enable</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.quorum-type: auto</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.server-quorum-type: server</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.data-self-heal-algorithm: full</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.locking-scheme: granular</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.shd-max-threads: 8</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.shd-wait-qlength: 10000</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>features.shard: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>user.cifs: off</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>storage.owner-gid: 36</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>features.shard-block-size: 512MB</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>network.ping-timeout: 30</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>performance.strict-o-direct: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>cluster.granular-entry-heal: on</i></div>
</div>
</div>
</div>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div><i>auth.allow: *</i></div>
</div>
</div>
</div>
</blockquote>
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div> server.allow-insecure: on</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"><span class="gmail-"><br>
<blockquote type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Some extra info:</div>
<div><br>
</div>
<div>We have recently changed the gluster
from: 2 (full repliacated) + 1 arbiter to
3 full replicated cluster </div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br>
</span> Just curious, how did you do this?
`remove-brick` of arbiter brick followed by an
`add-brick` to increase to replica-3?<br>
<br>
</div>
</blockquote>
<div><br>
</div>
<div>Yes</div>
<br>
<div>
<div class="gmail-hljs gmail-nginx"
style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace">
<div><br>
</div>
<div>
<div><span class="gmail-hljs-attribute">#gluster</span>
volume remove-brick engine replica <span
class="gmail-hljs-number"
style="color:rgb(136,0,0)">2</span>
node03:/gluster/data/brick force <b>(OK!)</b></div>
<div><br>
</div>
<div>#gluster volume heal engine <span
class="gmail-hljs-literal"
style="color:rgb(120,169,96)">info</span> <b>(<span
class="gmail-hljs-literal"
style="color:rgb(120,169,96)">no</span>
entries!)</b></div>
<div><br>
</div>
<div>#gluster volume add-brick engine replica <span
class="gmail-hljs-number"
style="color:rgb(136,0,0)">3</span>
node04:/gluster/engine/brick <b>(OK!)</b></div>
<div><br>
</div>
<div><b>After some minutes</b></div>
<div><br>
</div>
<div>[root<span class="gmail-hljs-variable"
style="color:rgb(188,96,96)">@node01</span> ~]<span
class="gmail-hljs-comment"
style="color:rgb(136,136,136)"># gluster volume
heal engine info</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node01:/gluster/engine/brick</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Status: Connected</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Number of entries:
0</span></div>
<div><br>
</div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node02:/gluster/engine/brick</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Status: Connected</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Number of entries:
0</span></div>
<div><br>
</div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Brick
node04:/gluster/engine/brick</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Status: Connected</span></div>
<div><span class="gmail-hljs-comment"
style="color:rgb(136,136,136)">Number of entries:
0</span></div>
</div>
</div>
</div>
<br>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF"> Thanks,<br>
Ravi</div>
</blockquote>
<div><br>
</div>
<div>Another extra info (I don't know if this can be the
problem): Five days ago A black out has suddenly shut down
the networks switch (also gluster network) of node 03 and
04 ... But I don't know this problem is in place after
this black out </div>
<div><br>
</div>
<div>Thank you!</div>
</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>