<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">Hi, </div><div class="gmail_quote"><br></div><div class="gmail_quote">Thank you for the answer and sorry for delay:</div><div class="gmail_quote"><br></div><div class="gmail_quote">2017-07-19 16:55 GMT+02:00 Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span>:</div><div class="gmail_quote"><br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
1. What does the glustershd.log say on all 3 nodes when you run the
command? Does it complain anything about these files?<br></div></blockquote><div><br></div><div>No, glustershd.log is clean, no extra log after command on all 3 nodes</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
2. Are these 12 files also present in the 3rd data brick?<br></div></blockquote><div><br></div><div>I've checked right now: all files exists in all 3 nodes </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
3. Can you provide the output of `gluster volume info` for the this
volume?</div></blockquote><div><br></div><div><br></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Volume Name: engine</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Type: Replicate</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Volume ID: d19c19e3-910d-437b-8ba7-4f2a23d17515</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Status: Started</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Snapshot Count: 0</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Number of Bricks: 1 x 3 = 3</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Transport-type: tcp</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Bricks:</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Brick1: node01:/gluster/engine/brick</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Brick2: node02:/gluster/engine/brick</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Brick3: node04:/gluster/engine/brick</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>Options Reconfigured:</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>nfs.disable: on</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>performance.readdir-ahead: on</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>transport.address-family: inet</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>storage.owner-uid: 36</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>performance.quick-read: off</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>performance.read-ahead: off</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>performance.io-cache: off</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>performance.stat-prefetch: off</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>performance.low-prio-threads: 32</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>network.remote-dio: off</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>cluster.eager-lock: enable</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>cluster.quorum-type: auto</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>cluster.server-quorum-type: server</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>cluster.data-self-heal-algorithm: full</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>cluster.locking-scheme: granular</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>cluster.shd-max-threads: 8</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>cluster.shd-wait-qlength: 10000</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>features.shard: on</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>user.cifs: off</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>storage.owner-gid: 36</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>features.shard-block-size: 512MB</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>network.ping-timeout: 30</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>performance.strict-o-direct: on</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>cluster.granular-entry-heal: on</i></div></div></div></div><div class="gmail_extra"><div class="gmail_quote"><div><div><i>auth.allow: *</i></div></div></div></div></blockquote><div class="gmail_extra"><div class="gmail_quote"><div><div> server.allow-insecure: on</div></div><div><br></div><div><br></div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF"><span class="gmail-"><br>
<blockquote type="cite">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div>Some extra info:</div>
<div><br>
</div>
<div>We have recently changed the gluster from: 2 (full
repliacated) + 1 arbiter to 3 full replicated cluster </div>
</div>
</blockquote>
</div>
</div>
</blockquote>
<br></span>
Just curious, how did you do this? `remove-brick` of arbiter brick
followed by an `add-brick` to increase to replica-3?<br>
<br></div></blockquote><div><br></div><div>Yes</div><div></div><br><div><div class="gmail-hljs gmail-nginx" style="display:block;overflow-x:auto;padding:0.5em;color:rgb(51,51,51);background:rgb(248,248,248);font-family:monospace"><div><br></div><div><div><span class="gmail-hljs-attribute">#gluster</span> volume remove-brick engine replica <span class="gmail-hljs-number" style="color:rgb(136,0,0)">2</span> node03:/gluster/data/brick force <b>(OK!)</b></div><div><br></div><div>#gluster volume heal engine <span class="gmail-hljs-literal" style="color:rgb(120,169,96)">info</span> <b>(<span class="gmail-hljs-literal" style="color:rgb(120,169,96)">no</span> entries!)</b></div><div><br></div><div>#gluster volume add-brick engine replica <span class="gmail-hljs-number" style="color:rgb(136,0,0)">3</span> node04:/gluster/engine/brick <b>(OK!)</b></div><div><br></div><div><b>After some minutes</b></div><div><br></div><div>[root<span class="gmail-hljs-variable" style="color:rgb(188,96,96)">@node01</span> ~]<span class="gmail-hljs-comment" style="color:rgb(136,136,136)"># gluster volume heal engine info</span></div><div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node01:/gluster/engine/brick</span></div><div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div><div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div><div><br></div><div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node02:/gluster/engine/brick</span></div><div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div><div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div><div><br></div><div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Brick node04:/gluster/engine/brick</span></div><div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Status: Connected</span></div><div><span class="gmail-hljs-comment" style="color:rgb(136,136,136)">Number of entries: 0</span></div></div></div></div><br><div><div></div></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div bgcolor="#FFFFFF">
Thanks,<br>
Ravi</div></blockquote><div><br></div><div>Another extra info (I don't know if this can be the problem): Five days ago A black out has suddenly shut down the networks switch (also gluster network) of node 03 and 04 ... But I don't know this problem is in place after this black out </div><div><br></div><div>Thank you!</div></div><div><br></div>
</div></div>