<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hello all fellow GlusterFriends,</div><div class=""><br class=""></div><div class="">I would like you to comment / correct my upgrade procedure steps on replica 2 volume of 3.7.x gluster.</div><div class="">Than I would like to change replica 2 to replica 3 in order to correct quorum issue that Infrastructure currently has.</div><div class=""><br class=""></div><div class=""><b class="">Infrastructure setup:</b></div><div class="">- all clients running on same nodes as servers (FUSE mounts)</div><div class="">- under gluster there is ZFS pool running as raidz2 with SSD ZLOG/ZIL cache</div><div class="">- all two hypervisor running as GlusterFS nodes and also Qemu compute nodes (Ubuntu 16.04 LTS)</div><div class="">- we are running Qemu VMs that accesses VMs disks via gfapi (Opennebula)</div><div class="">- we currently run : 1x2 , Type: Replicate volume</div><div class=""><br class=""></div><div class=""><b class="">Current Versions :</b></div><div class="">glusterfs-* [package] 3.7.6-1ubuntu1</div><div class="">qemu-*<span class="Apple-tab-span" style="white-space:pre"> </span>[package] 2.5+dfsg-5ubuntu10.2glusterfs3.7.14xenial1</div><div class=""><br class=""></div><div class=""><b class="">What we need : (New versions)</b></div><div class="">- upgrade GlusterFS to 3.12 LTM version (Ubuntu 16.06 LTS packages are EOL - see <a href="https://www.gluster.org/community/release-schedule/" class="">https://www.gluster.org/community/release-schedule/</a>)</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- I want to use <a href="https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.12" class="">https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.12</a> as package repository for 3.12</div><div class="">- upgrade Qemu (with build-in support for libgfapi) - <a href="https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.12" class="">https://launchpad.net/~monotek/+archive/ubuntu/qemu-glusterfs-3.12</a></div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- (sadly Ubuntu has packages build without libgfapi support)</div><div class="">- add third node to replica setup of volume (this is probably most dangerous operation)</div><div class=""><br class=""></div><div class=""><b class="">Backup Phase</b></div><div class="">- backup "NFS storage” - raw DATA that runs on VMs</div><div class="">- stop all running VMs</div><div class="">- backup all running VMs (Qcow2 images) outside of gluster</div><div class=""><br class=""></div><div class=""><b class="">Upgrading Gluster Phase</b></div><div class="">- killall glusterfs glusterfsd glusterd (on every server)</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>(this should stop all gluster services - server and client as it runs on same nodes)</div><div class="">- install new Gluster Server and Client packages from repository mentioned upper (on every server) </div><div class="">- install new Monotek's qemu glusterfs package with gfapi enabled support (on every server) </div><div class="">- /etc/init.d/glusterfs-server start (on every server)</div><div class="">- /etc/init.d/glusterfs-server status - verify that all runs ok (on every server)</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- check :</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- gluster volume info</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- gluster volume status</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>- check gluster FUSE clients, if mounts working as expected</div><div class="">- test if various VMs are able tu boot and run as expected (if libgfapi works in Qemu)</div><div class="">- reboot all nodes - do system upgrade of packages</div><div class="">- test and check again</div><div class=""><br class=""></div><div class=""><b class="">Adding third node to replica 2 setup (replica 2 => replica 3)</b></div><div class="">(volumes will be mounted and up after upgrade and we tested VMs are able to be served with libgfapi = upgrade of gluster sucessfuly completed)</div><div class="">(next we extend replica 2 to replica 3 while volumes are mounted but no data is touched = no running VMs, only glusterfs servers and clients on nodes)</div><div class="">- issue command : gluster volume add-brick volume replica 3 node3.san:/tank/gluster/brick1 (on new single node - node3)</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>so we change : </div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Bricks:</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Brick1: node1.san:/tank/gluster/brick1</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Brick2: node2.san:/tank/gluster/brick1</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>to :</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Bricks:</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Brick1: node1.san:/tank/gluster/brick1</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Brick2: node2.san:/tank/gluster/brick1</div><div class=""><span class="Apple-tab-span" style="white-space:pre"> </span>Brick3: node3.san:/tank/gluster/brick1</div><div class="">- check gluster status</div><div class="">- (is rebalance / heal required here ?)</div><div class="">- start all VMs and start celebration :)</div><div class=""><br class=""></div><div class=""><b class="">My Questions</b></div><div class="">- is heal and rebalance necessary in order to upgrade replica 2 to replica 3 ?</div><div class="">- is this upgrade procedure OK ? What more/else should I do in order to do this upgrade correctly ?</div><div class=""><br class=""></div><div class="">Many thanks to all for support. Hope my little preparation howto will help others to solve same situation.</div><div class=""><br class=""></div><div class="">Best Regards,</div><div class="">Martin</div></body></html>