[Gluster-devel] Rolling upgrades from glusterfs 3.4 to 3.5

[Ravishankar N]

On 06/12/2014 11:13 PM, Anand Avati wrote:
> On Thu, Jun 12, 2014 at 10:33 AM, Vijay Bellur <vbellur at redhat.com 
> <mailto:vbellur at redhat.com>> wrote:
>
>     On 06/12/2014 06:52 PM, Ravishankar N wrote:
>
>         Hi Vijay,
>
>         Since glusterfs 3.5, posix_lookup() sends ESTALE instead of
>         ENOENT [1]
>         when when a parent gfid (entry) is not present on the brick . In a
>         replicate set up, this causes a problem because AFR gives more
>         priority
>         to ESTALE than ENOENT, causing IO to fail [2]. The fix is in
>         progress at
>         [3] and is client-side specific , and would make it to 3.5.2
>
>         But we will still hit the problem when rolling upgrade is
>         performed from
>         3.4 to 3.5,  unless the clients are also upgraded to 3.5: To
>         elaborate
>         an example:
>
>         0) Create a 1x2 volume using 2 nodes and mount it from client. All
>         machines are glusterfs 3.4
>         1) Perform for i in {1..30}; do mkdir $i; tar xf
>         glusterfs-3.5git.tar.gz
>         -C $i& done
>         2) While this is going on, kill one of the node in the replica
>         pair and
>         upgrade it to glusterfs 3.5 (simulating rolling upgrade)
>         3) After a while, kill all tar processes
>         4) Create a backup directory and move all 1..30 dirs inside
>         'backup'
>         5) Start the untar processes in 1) again
>         6) Bring up the upgraded node. Tar fails with estale errors.
>
>         Essentially the errors occur because [3] is a client side fix. But
>         rolling upgrades are targeted at servers while the older
>         clients still
>         need to access them without issues.
>
>         A solution is to have a fix in the posix translator wherein
>         the newer
>         client passes it's version (3.5) to posix_lookup() which then
>         sends
>         ESTALE if version is 3.5 or newer but sends ENOENT instead if
>         it is an
>         older client. Does this seem okay?
>
>
>     Cannot think of a better solution to this. Seamless rolling
>     upgrades are necessary for us and the proposed fix does seem okay
>     for that reason.
>
>     Thanks,
>     Vijay
>
>
> I also like Justin's proposal, of having fixes in 3.4.X and requiring 
> clients to be at least 3.4.X in order to have rolling upgrade to 
> 3.5.Y. This way we can add the "special fix" in 3.4.X client (just 
> like the 3.5.2 client). Ravi's proposal "works", but all LOOKUPs will 
> have an extra xattr, and we will be carrying forward the compat code 
> burden for a very long time. Whereas a 3.4.X client fix will remain in 
> 3.4 branch.
>
> Thanks
>

I have sent a fix for review (http://review.gluster.org/#/c/8080/) . The 
change is in the server side only. I reckon if we are asking users to 
upgrade clients to a 3.4.x  which anyway involves app downtime, we might 
as well ask them to upgrade to 3.5.

The fix is only sent on 3.5 - it does not need to go to master as I 
understand from Pranith that we only support compatibility between the 
current two releases. (meaning 3.6 servers require clients to be at at 
least 3.5 and not lower).

Regards,
Ravi

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140618/be9f2503/attachment.html>