[Gluster-users] Upgrade Gluster 3.7 to 3.12 and add 3rd replica [howto/help]

[Diego Remolina]

Hi Martin,

> Do you mean latest package from Ubuntu repository or latest package from
> Gluster PPA (3.7.20-ubuntu1~xenial1).
> Currently I am using Ubuntu repository package, but want to use PPA for
> upgrade because Ubuntu has old packages of Gluster in repo.

When you switch to PPA, make sure to download and keep a copy of each
set of gluster deb packages, otherwise if you ever want to back out an
upgrade to an older release, you will have to download the source deb
file and build it yourself, because PPAs only keep the latest version
for binaries.

>
> I do not use sharding because all bricks has same size, so it will not
> speedup healing of VMs images in case of heal operation. Volume is 3TB, how
> long does it take to heal on 2x1gbit (linux bond) connection, can you
> approximate ?

Sharding is not so much about brick size. Sharding is about preventing
a whole large VM file being locked when it is being healed. Also
minimizes the amount of data copied because gluster only heals smaller
pieces versus a whole VM image.

Say your 100GB IMG needs to be healed, the file is locked while it
gets copied from one server to the other and the running VM may not be
able to use it while the heal is going, so your VM may in fact stop
working or have I/O errors. With sharding, VMs are cut into, well,
shards, largest shard is 512MB, then the heal process only locks the
shards being healed. So gluster only heals the shards that changed
which are much smaller and faster to copy, and do not need to lock the
whole 100GB IMG file which takes longer to copy, just the shard being
healed. Do note that if you had never used sharding, if you turn it on
it will *not* convert your older files. Also you should *never* turn
on sharding and then back off, as that will result in corrupted VM
image files. Once it is on, if you want to turn it off, stop your VMs,
then move all VM IMG files elsewhere, turn off sharding and then copy
the files back  to the volume after disabling sharding.

As for speed, I really cannot tell you as it depends on the disks,
netowr, etc. For example, I have a two node setup plus an arbiter (2
nodes with bricks, one is just the arbiter to keep quorum if one of
the brick servers goes down). I recently replaced the HDDs in one
machine as the drives hit the 5 year age mark. So I took the 12 drives
out, added 24 drives to the machine (we had unused slots),
reconfigured raid 6 and left it initializing in the background and
started the heal of 13.1TB of data. My servers are connected via
10Gbit (I am not seeing reads/writes over 112MB/s) and this process
started last Monday at 7;20PM and it is not done yet. It is missing
healing about 40GB still. Now my servers are used as a file server,
which means lots of small files which take longer to heal. I would
think your VM images will heal much faster.

> I want to turn every VM off because its required for upgrading gluster
> procedure, thats why I want to add 3rd brick (3rd replica) at this time
> (after upgrade when VMs will be offline).
>

You could even attempt an online upgrade if you try to add the new
node/brick running 3.12 to the mix before upgrading from 3.7.x on the
other nodes. However, I am not sure how that is going to work. With
such a difference in versions, it may not work well.

If you can afford the downtime to upgrade, that will be the safest option.

Diego