[Gluster-devel] Rolling upgrades from glusterfs 3.4 to 3.5

[Justin Clift]

On 12/06/2014, at 6:47 PM, Pranith Kumar Karampuri wrote:
> On 06/12/2014 11:16 PM, Anand Avati wrote:
<snip>
>> The client can actually be fixed to be compatible with both old and new servers. We can change the errno from ESTALE to ENOENT before doing the GFID mismatch check in client_lookup_cbk.
> But this will require a two hop upgrade. Is that normal/acceptable?


Thoughts on this from Corvid Tech:

> From: "David F. Robinson" <david.robinson at corvidtec.com>
> Subject: Re: Rolling upgrades discussion on mailing list
> Date: 12 June 2014 11:01:27 PM GMT+01:00
> 
> Read through the discussion.  Not sure I fully understood it, but here are my thoughts regardless...
> 
> - We have roughly 1000-nodes (HPC setup) and 100 workstations that hopefully will be using gluster as the primary storage system (if we can resolve the few outstanding issues)... If these all had gluster-3.4.x rpm's and I wanted to upgrade to 3.5.x rpm's, I don't particular care if I had to incrementally update the packages (3.4.5 --> 3.4.6 --> 3.5.0 --> 3.5.1).  Requiring a minimum version seems reasonable and workable to me as long as there is a documented process to update.
> - What I wouldn't want to do if it was avoidable was to have to stop all i/o on all of the hpc-nodes and workstations for the update.  If I could do incremental 'rpm -Uvh' for the various versions "without" killing the currently ongoing i/o, that is what would be most important.  It wasn't clear to me if this was not possible at all, or not possible unless you upgraded all of the clients before upgrading the server.
> - If the problem is that you have to update all of the clients BEFORE updating the server, that doesn't sound unworkable as long as there is  a mechanism to list all clients that are connected and display the gluster version each client is using.
> - Also, a clearly documented process to update would be great.  We have lots of questions and taking the entire storage device offline to do an upgrade isn't a really feasible.  My assumption after reading the link you sent is that we would have to upgrade all of the client software (assumes that a newer client can still write to an older server version... i.e. client 3.5.2 can write to a 3.4.5 server).  This isn't a big deal and seems reasonable, if there is some method to find all of the connected clients.
> 
> - From the server side, I would like to be able to take one of the nodes/bricks offline nicely (i.e. have the current i/o finish, accept no more i/o requests, and then take the node offline for writes) and do the update. Then bring that node back up, let the heal finish, and move on to the next node/brick.  We are told that simply taking a node offline will not interrupt the i/o if you are using a replica.  We have experienced mixed results with this.  I believe that the active i/o to the brick fails but additional i/o fails over to the other brick.  A nicer process to take a brick offline would be desirable.
>    - The other suggestion I have seen for server software updates is to migrate all of the data from a node/brick to the other bricks of the system (gluster remove-brick xxx), remove this brick from the pool, do the software update, and then add it back to the pool. The issue here is that each of our bricks is roughly 100TB. Moving all of that data to the other bricks in the system will take a very long time. Not a practical solution for us...
> 
> 
> I don't mind having to use a process (incrementally upgrade all clients, take bricks offline, update bricks),  as long as I didn't have to turn off all i/o to the primary storage system for the entire company in order to execute an upgrade... The number one requirement for us would be to have a process to do an upgrade on a "live" system... Who cares if the process is painful... That is the IT guys problem :)...


+ Justin

--
GlusterFS - http://www.gluster.org

An open source, distributed file system scaling to several
petabytes, and handling thousands of clients.

My personal twitter: twitter.com/realjustinclift