[Gluster-users] Upgrade Gluster 3.7 to 3.12 and add 3rd replica [howto/help]

[Martin Toth]

Hi Diego,

I’ve tried to upgrade and then extend gluster with 3rd node in virtualbox test environment and all went without problems.
Sharding will not help me at this time so I will consider upgrading 1G to 10G before this procedure in production. That should lower downtime - healing time of VM image files on Gluster.

I hope healing will take as short as possible on 10G.

Additional info for Gluster/Qemu Users:
- Ubuntu does not have Qemu compiled with libgfapi support so I’ve created PPA for that :
	https://launchpad.net/~snowmanko/+archive/ubuntu/qemu-glusterfs-3.12 <https://launchpad.net/~snowmanko/+archive/ubuntu/qemu-glusterfs-3.12> (I will try to make this repo up to date)
	- it’s tested against glusterfs3.12.1 version (libgfapi works as expected with this repo)

- Moreover related to this problem - there is MIR - https://bugs.launchpad.net/ubuntu/+source/glusterfs/+bug/1274247 <https://bugs.launchpad.net/ubuntu/+source/glusterfs/+bug/1274247> - it’s now accepted, I am really excited to see libgfapi compiled by default in Ubuntu Qemu packages in near future

Thanks for support.

BR,

Martin

> On 22 Sep 2017, at 14:50, Diego Remolina <dijuremo at gmail.com> wrote:
> 
> Hi Martin,
> 
>> Do you mean latest package from Ubuntu repository or latest package from
>> Gluster PPA (3.7.20-ubuntu1~xenial1).
>> Currently I am using Ubuntu repository package, but want to use PPA for
>> upgrade because Ubuntu has old packages of Gluster in repo.
> 
> When you switch to PPA, make sure to download and keep a copy of each
> set of gluster deb packages, otherwise if you ever want to back out an
> upgrade to an older release, you will have to download the source deb
> file and build it yourself, because PPAs only keep the latest version
> for binaries.
> 
>> 
>> I do not use sharding because all bricks has same size, so it will not
>> speedup healing of VMs images in case of heal operation. Volume is 3TB, how
>> long does it take to heal on 2x1gbit (linux bond) connection, can you
>> approximate ?
> 
> Sharding is not so much about brick size. Sharding is about preventing
> a whole large VM file being locked when it is being healed. Also
> minimizes the amount of data copied because gluster only heals smaller
> pieces versus a whole VM image.
> 
> Say your 100GB IMG needs to be healed, the file is locked while it
> gets copied from one server to the other and the running VM may not be
> able to use it while the heal is going, so your VM may in fact stop
> working or have I/O errors. With sharding, VMs are cut into, well,
> shards, largest shard is 512MB, then the heal process only locks the
> shards being healed. So gluster only heals the shards that changed
> which are much smaller and faster to copy, and do not need to lock the
> whole 100GB IMG file which takes longer to copy, just the shard being
> healed. Do note that if you had never used sharding, if you turn it on
> it will *not* convert your older files. Also you should *never* turn
> on sharding and then back off, as that will result in corrupted VM
> image files. Once it is on, if you want to turn it off, stop your VMs,
> then move all VM IMG files elsewhere, turn off sharding and then copy
> the files back  to the volume after disabling sharding.
> 
> As for speed, I really cannot tell you as it depends on the disks,
> netowr, etc. For example, I have a two node setup plus an arbiter (2
> nodes with bricks, one is just the arbiter to keep quorum if one of
> the brick servers goes down). I recently replaced the HDDs in one
> machine as the drives hit the 5 year age mark. So I took the 12 drives
> out, added 24 drives to the machine (we had unused slots),
> reconfigured raid 6 and left it initializing in the background and
> started the heal of 13.1TB of data. My servers are connected via
> 10Gbit (I am not seeing reads/writes over 112MB/s) and this process
> started last Monday at 7;20PM and it is not done yet. It is missing
> healing about 40GB still. Now my servers are used as a file server,
> which means lots of small files which take longer to heal. I would
> think your VM images will heal much faster.
> 
>> I want to turn every VM off because its required for upgrading gluster
>> procedure, thats why I want to add 3rd brick (3rd replica) at this time
>> (after upgrade when VMs will be offline).
>> 
> 
> You could even attempt an online upgrade if you try to add the new
> node/brick running 3.12 to the mix before upgrading from 3.7.x on the
> other nodes. However, I am not sure how that is going to work. With
> such a difference in versions, it may not work well.
> 
> If you can afford the downtime to upgrade, that will be the safest option.
> 
> Diego

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20171001/5648bfe4/attachment.html>