<div dir="ltr">And what does glusterd log indicate for these failures?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 5, 2017 at 8:43 PM, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div class="h5"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 5, 2017 at 5:02 PM, Sahina Bose <span dir="ltr"><<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 5, 2017 at 8:16 PM, Gianluca Cecchi <span dir="ltr"><<a href="mailto:gianluca.cecchi@gmail.com" target="_blank">gianluca.cecchi@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span>On Wed, Jul 5, 2017 at 7:42 AM, Sahina Bose <span dir="ltr"><<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><div class="gmail_quote"><div><div class="m_-7125121196685184234gmail-m_6600024723269394955m_3192401637786308022gmail-h5"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><br>...<br><br>then the commands I need to run would be:<br><br>gluster volume reset-brick export ovirt01.localdomain.local:/glu<wbr>ster/brick3/export start<br>gluster volume reset-brick export ovirt01.localdomain.local:/glu<wbr>ster/brick3/export gl01.localdomain.local:/gluste<wbr>r/brick3/export commit force<br><br></div><div class="gmail_quote">Correct?<br></div></div></div></blockquote><div><br></div></div></div><div>Yes, correct. gl01.localdomain.local should resolve correctly on all 3 nodes.<br></div></div></div></div></blockquote><div><br></div><div><br></div></span><div>It fails at first step:</div><div><br></div><div> [root@ovirt01 ~]# gluster volume reset-brick export ovirt01.localdomain.local:/glu<wbr>ster/brick3/export start</div><div>volume reset-brick: failed: Cannot execute command. The cluster is operating at version 30712. reset-brick command reset-brick start is unavailable in this version.</div><div>[root@ovirt01 ~]# </div><div><br></div><div>It seems somehow in relation with this upgrade not of the commercial solution Red Hat Gluster Storage</div><div><a href="https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Installation_Guide/chap-Upgrading_Red_Hat_Storage.html" target="_blank">https://access.redhat.com/docu<wbr>mentation/en-US/Red_Hat_Storag<wbr>e/3.1/html/Installation_Guide/<wbr>chap-Upgrading_Red_Hat_Storage<wbr>.html</a><br></div><div><br></div><div>So ti seems I have to run some command of type:</div><div><br></div><div>gluster volume set all cluster.op-version XXXXX<br></div><div><br></div><div>with XXXXX > 30712</div><div><br></div><div>It seems that latest version of commercial Red Hat Gluster Storage is 3.1 and its op-version is indeed 30712..</div><div><br></div><div>So the question is which particular op-version I have to set and if the command can be set online without generating disruption....</div></div></div></div></blockquote><div><br></div><div>It should have worked with the glusterfs 3.10 version from Centos repo. Adding gluster-users for help on the op-version<br> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><br></div><div>Thanks,</div><div>Gianluca</div></div></div></div>
</blockquote></div><br></div></div>
</blockquote></div><br></div><div class="gmail_extra"><br></div></div></div><div class="gmail_extra">It seems op-version is not updated automatically by default, so that it can manage mixed versions while you update one by one...</div><div class="gmail_extra"><br></div><div class="gmail_extra">I followed what described here:</div><div class="gmail_extra"><div class="gmail_extra"><a href="https://gluster.readthedocs.io/en/latest/Upgrade-Guide/op_version/" target="_blank">https://gluster.readthedocs.<wbr>io/en/latest/Upgrade-Guide/op_<wbr>version/</a></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">- Get current version:</div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 ~]# gluster volume get all cluster.op-version</div><div class="gmail_extra">Option Value </div><div class="gmail_extra">------ ----- </div><div class="gmail_extra">cluster.op-version 30712 </div><div class="gmail_extra">[root@ovirt01 ~]# </div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">- Get maximum version I can set for current setup:</div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 ~]# gluster volume get all cluster.max-op-version</div><div class="gmail_extra">Option Value </div><div class="gmail_extra">------ ----- </div><div class="gmail_extra">cluster.max-op-version 31000 </div><div class="gmail_extra">[root@ovirt01 ~]# </div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">- Get op version information for all the connected clients:</div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 ~]# gluster volume status all clients | grep ":49" | awk '{print $4}' | sort | uniq -c </div><div class="gmail_extra"> 72 31000</div><div class="gmail_extra">[root@ovirt01 ~]# </div><div class="gmail_extra"><br></div><div class="gmail_extra">--> ok</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">- Update op-version</div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 ~]# gluster volume set all cluster.op-version 31000</div><div class="gmail_extra">volume set: success</div><div class="gmail_extra">[root@ovirt01 ~]# </div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">- Verify:</div><div class="gmail_extra">[root@ovirt01 ~]# gluster volume get all cluster.op-versionOption Value </div><div class="gmail_extra">------ ----- </div><div class="gmail_extra">cluster.op-version 31000 </div><div class="gmail_extra">[root@ovirt01 ~]# </div><div class="gmail_extra"><br></div><div class="gmail_extra">--> ok</div><span class=""><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 ~]# gluster volume reset-brick export ovirt01.localdomain.local:/<wbr>gluster/brick3/export start</div></span><div class="gmail_extra">volume reset-brick: success: reset-brick start operation successful</div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 ~]# gluster volume reset-brick export ovirt01.localdomain.local:/<wbr>gluster/brick3/export gl01.localdomain.local:/<wbr>gluster/brick3/export commit force</div><div class="gmail_extra">volume reset-brick: failed: Commit failed on ovirt02.localdomain.local. Please check log file for details.</div><div class="gmail_extra">Commit failed on ovirt03.localdomain.local. Please check log file for details.</div><div class="gmail_extra">[root@ovirt01 ~]# </div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 bricks]# gluster volume info export</div><div class="gmail_extra"> </div><div class="gmail_extra">Volume Name: export</div><div class="gmail_extra">Type: Replicate</div><div class="gmail_extra">Volume ID: b00e5839-becb-47e7-844f-<wbr>6ce6ce1b7153</div><div class="gmail_extra">Status: Started</div><div class="gmail_extra">Snapshot Count: 0</div><div class="gmail_extra">Number of Bricks: 1 x (2 + 1) = 3</div><div class="gmail_extra">Transport-type: tcp</div><div class="gmail_extra">Bricks:</div><div class="gmail_extra">Brick1: gl01.localdomain.local:/<wbr>gluster/brick3/export</div><div class="gmail_extra">Brick2: ovirt02.localdomain.local:/<wbr>gluster/brick3/export</div><div class="gmail_extra">Brick3: ovirt03.localdomain.local:/<wbr>gluster/brick3/export (arbiter)</div><div class="gmail_extra">Options Reconfigured:</div><div class="gmail_extra">transport.address-family: inet</div><div class="gmail_extra">performance.readdir-ahead: on</div><div class="gmail_extra">performance.quick-read: off</div><div class="gmail_extra">performance.read-ahead: off</div><div class="gmail_extra">performance.io-cache: off</div><div class="gmail_extra">performance.stat-prefetch: off</div><div class="gmail_extra">cluster.eager-lock: enable</div><div class="gmail_extra">network.remote-dio: off</div><div class="gmail_extra">cluster.quorum-type: auto</div><div class="gmail_extra">cluster.server-quorum-type: server</div><div class="gmail_extra">storage.owner-uid: 36</div><div class="gmail_extra">storage.owner-gid: 36</div><div class="gmail_extra">features.shard: on</div><div class="gmail_extra">features.shard-block-size: 512MB</div><div class="gmail_extra">performance.low-prio-threads: 32</div><div class="gmail_extra">cluster.data-self-heal-<wbr>algorithm: full</div><div class="gmail_extra">cluster.locking-scheme: granular</div><div class="gmail_extra">cluster.shd-wait-qlength: 10000</div><div class="gmail_extra">cluster.shd-max-threads: 6</div><div class="gmail_extra">network.ping-timeout: 30</div><div class="gmail_extra">user.cifs: off</div><div class="gmail_extra">nfs.disable: on</div><div class="gmail_extra">performance.strict-o-direct: on</div><div class="gmail_extra">[root@ovirt01 bricks]# gluster volume reset-brick export ovirt02.localdomain.local:/<wbr>gluster/brick3/export start</div><div class="gmail_extra">volume reset-brick: success: reset-brick start operation successful</div><div class="gmail_extra">[root@ovirt01 bricks]# gluster volume reset-brick export ovirt02.localdomain.local:/<wbr>gluster/brick3/export gl02.localdomain.local:/<wbr>gluster/brick3/export commit force</div><div class="gmail_extra">volume reset-brick: failed: Commit failed on localhost. Please check log file for details.</div><div class="gmail_extra">[root@ovirt01 bricks]#</div><div class="gmail_extra"><br></div><div class="gmail_extra">I proceed (I have actually nothing on export volume...)</div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 bricks]# gluster volume reset-brick export ovirt02.localdomain.local:/<wbr>gluster/brick3/export start</div><div class="gmail_extra">volume reset-brick: success: reset-brick start operation successful</div><div class="gmail_extra">[root@ovirt01 bricks]# gluster volume reset-brick export ovirt02.localdomain.local:/<wbr>gluster/brick3/export gl02.localdomain.local:/<wbr>gluster/brick3/export commit force</div><div class="gmail_extra">volume reset-brick: failed: Commit failed on localhost. Please check log file for details.</div><div class="gmail_extra">[root@ovirt01 bricks]# </div><div class="gmail_extra"><br></div><div class="gmail_extra">Again error</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 bricks]# gluster volume info export</div><div class="gmail_extra"> </div><div class="gmail_extra">Volume Name: export</div><div class="gmail_extra">Type: Replicate</div><div class="gmail_extra">Volume ID: b00e5839-becb-47e7-844f-<wbr>6ce6ce1b7153</div><div class="gmail_extra">Status: Started</div><div class="gmail_extra">Snapshot Count: 0</div><div class="gmail_extra">Number of Bricks: 0 x (2 + 1) = 2</div><div class="gmail_extra">Transport-type: tcp</div><div class="gmail_extra">Bricks:</div><div class="gmail_extra">Brick1: gl01.localdomain.local:/<wbr>gluster/brick3/export</div><div class="gmail_extra">Brick2: ovirt03.localdomain.local:/<wbr>gluster/brick3/export</div><div class="gmail_extra">Options Reconfigured:</div><div class="gmail_extra">transport.address-family: inet</div><div class="gmail_extra">performance.readdir-ahead: on</div><div class="gmail_extra">performance.quick-read: off</div><div class="gmail_extra">performance.read-ahead: off</div><div class="gmail_extra">performance.io-cache: off</div><div class="gmail_extra">performance.stat-prefetch: off</div><div class="gmail_extra">cluster.eager-lock: enable</div><div class="gmail_extra">network.remote-dio: off</div><div class="gmail_extra">cluster.quorum-type: auto</div><div class="gmail_extra">cluster.server-quorum-type: server</div><div class="gmail_extra">storage.owner-uid: 36</div><div class="gmail_extra">storage.owner-gid: 36</div><div class="gmail_extra">features.shard: on</div><div class="gmail_extra">features.shard-block-size: 512MB</div><div class="gmail_extra">performance.low-prio-threads: 32</div><div class="gmail_extra">cluster.data-self-heal-<wbr>algorithm: full</div><div class="gmail_extra">cluster.locking-scheme: granular</div><div class="gmail_extra">cluster.shd-wait-qlength: 10000</div><div class="gmail_extra">cluster.shd-max-threads: 6</div><div class="gmail_extra">network.ping-timeout: 30</div><div class="gmail_extra">user.cifs: off</div><div class="gmail_extra">nfs.disable: on</div><div class="gmail_extra">performance.strict-o-direct: on</div><div class="gmail_extra">[root@ovirt01 bricks]# </div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">The last</div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 bricks]# gluster volume reset-brick export ovirt03.localdomain.local:/<wbr>gluster/brick3/export start</div><div class="gmail_extra">volume reset-brick: success: reset-brick start operation successful</div><div class="gmail_extra">[root@ovirt01 bricks]# gluster volume reset-brick export ovirt03.localdomain.local:/<wbr>gluster/brick3/export gl03.localdomain.local:/<wbr>gluster/brick3/export commit force</div><div class="gmail_extra">volume reset-brick: failed: Commit failed on localhost. Please check log file for details.</div><div class="gmail_extra">[root@ovirt01 bricks]# </div><div class="gmail_extra"><br></div><div class="gmail_extra">again error</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">[root@ovirt01 bricks]# gluster volume info export</div><div class="gmail_extra"> </div><div class="gmail_extra">Volume Name: export</div><div class="gmail_extra">Type: Replicate</div><div class="gmail_extra">Volume ID: b00e5839-becb-47e7-844f-<wbr>6ce6ce1b7153</div><div class="gmail_extra">Status: Started</div><div class="gmail_extra">Snapshot Count: 0</div><div class="gmail_extra">Number of Bricks: 0 x (2 + 1) = 1</div><div class="gmail_extra">Transport-type: tcp</div><div class="gmail_extra">Bricks:</div><div class="gmail_extra">Brick1: gl01.localdomain.local:/<wbr>gluster/brick3/export</div><div class="gmail_extra">Options Reconfigured:</div><div class="gmail_extra">transport.address-family: inet</div><div class="gmail_extra">performance.readdir-ahead: on</div><div class="gmail_extra">performance.quick-read: off</div><div class="gmail_extra">performance.read-ahead: off</div><div class="gmail_extra">performance.io-cache: off</div><div class="gmail_extra">performance.stat-prefetch: off</div><div class="gmail_extra">cluster.eager-lock: enable</div><div class="gmail_extra">network.remote-dio: off</div><div class="gmail_extra">cluster.quorum-type: auto</div><div class="gmail_extra">cluster.server-quorum-type: server</div><div class="gmail_extra">storage.owner-uid: 36</div><div class="gmail_extra">storage.owner-gid: 36</div><div class="gmail_extra">features.shard: on</div><div class="gmail_extra">features.shard-block-size: 512MB</div><div class="gmail_extra">performance.low-prio-threads: 32</div><div class="gmail_extra">cluster.data-self-heal-<wbr>algorithm: full</div><div class="gmail_extra">cluster.locking-scheme: granular</div><div class="gmail_extra">cluster.shd-wait-qlength: 10000</div><div class="gmail_extra">cluster.shd-max-threads: 6</div><div class="gmail_extra">network.ping-timeout: 30</div><div class="gmail_extra">user.cifs: off</div><div class="gmail_extra">nfs.disable: on</div><div class="gmail_extra">performance.strict-o-direct: on</div><div class="gmail_extra">[root@ovirt01 bricks]# </div><div class="gmail_extra"><br></div><div class="gmail_extra">See here for gluster log in gzip format....</div><div class="gmail_extra"><a href="https://drive.google.com/file/d/0BwoPbcrMv8mvQmlYZjAySTZKTzQ/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/<wbr>0BwoPbcrMv8mvQmlYZjAySTZKTzQ/<wbr>view?usp=sharing</a><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">The first command executed at 14:57 and the other two at 15:04</div><div class="gmail_extra"><br></div><div class="gmail_extra">This is what seen by oVirt right now for the volume</div><div class="gmail_extra"><a href="https://drive.google.com/file/d/0BwoPbcrMv8mvNFAyd043TnNwSEU/view?usp=sharing" target="_blank">https://drive.google.com/file/<wbr>d/<wbr>0BwoPbcrMv8mvNFAyd043TnNwSEU/<wbr>view?usp=sharing</a><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">(After the first command I saw 2 of 3 up)</div><span class="HOEnZb"><font color="#888888"><div class="gmail_extra"><br></div><div class="gmail_extra">Gianluca</div></font></span></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>