[Gluster-devel] Support to reclaim locks (posix) provided lkowner & range matches

[Vijay Bellur]

On 07/26/2016 05:56 AM, Soumya Koduri wrote:
> Hi Vijay,
>
> On 07/26/2016 12:13 AM, Vijay Bellur wrote:
>> On 07/22/2016 08:44 AM, Soumya Koduri wrote:
>>> Hi,
>>>
>>> In certain scenarios (esp.,in highly available environments), the
>>> application may have to fail-over/connect to a different glusterFS
>>> client while the I/O is happening. In such cases until there is a ping
>>> timer expiry and glusterFS server cleans up the locks held by the older
>>> glusterFS client, the application will not be able to reclaim their lost
>>> locks. To avoid that we need support in Gluster to let clients reclaim
>>> the existing locks provided lkwoner and the lock range matches.
>>
>>
>> If the server detects a disconnection, it goes about cleaning up the
>> locks held by the disconnected client. Only if the failover connection
>> happens before this server cleanup the outlined scheme would work.Since
>> there is no ping timer on the server, do you propose to have a grace
>> timer on the server?
>
> But we are looking for a solution which can work in active-active
> configuration as well. We need to handle cases where in the connection
> between server and the old-client is still in use, which can happen
> during load-balancing or failback.
>
> Different cases which I can outline are:
>
> Application Client - (AC)
> Application/GlusterClient 1 - GC1
> Application/GlusterClient 2 - GC2
> Gluster Server (GS)
>
> 1) Active-Passive config  (service gone down)
>
> AC ----> GC1  ----> GS (GC2 is not active)
>
>     | (failover)
>     v
>
> AC ----> GC2  ----> GS (GC1 connection gets dropped and GC2 establishes
> connection)
>
> In this case, we can have grace timer to allow reclaims only for certain
> time post GC2 (any) rpc connection establishment.
>
> 2) Active-Active config  (service gone down)
>
> AC ----> GC1  ----> GS
>              ^
>              |
>          GC2  -------
>
>     | (failover)
>     v
>
> AC ----> GC2  ----> GS (GC1 connection gets dropped)
>
> The grace timer then shall not get triggered in this case. But at-least
> the locks from GC1 gets cleaned post its connection cleanup.
>

grace timer is not required if lock reclamation can happen before the 
old connection between GC1 & GS gets dropped. Is this guaranteed to 
happen every time?

>
> 3) Active-Active config  (both the services active/load-balancing)
> This is the trick one.
>
> AC ----> GC1  ----> GS
>              ^
>              |
>          GC2  -------
>
>     | (load-balancing/failback)
>     v
>
>      GC1  ----> GS
>              ^
>              |
> AC ----> GC2  -------
>
> The locks taken by GC1 shall end up being on the server for ever unless
> we restart either GC1 or the server.
>

Yes, this is trickier. The behavior is dependent on how the application 
performs a failback. How do we handle this with Ganesha today? Since the 
connection between nfs client and Ganesha/GC1 is broken, would it not 
send cleanup requests on locks it held on behalf of that client?


> Considering above cases, looks like we may need to allow reclaim of the
> locks all the time. Please suggest if I have missed out any details.
>

I agree that lock reclamation is needed. Grace timeout behavior does 
need more thought for all these cases. Given the involved nature of this 
problem, it might be better to write down a more detailed spec that 
discusses all these cases for a more thorough review.

>>
>>>
>>> For client-side support, I am thinking if we can integrate with the new
>>> lock API being introduced as part of mandatory lock support in gfapi [2]
>>>
>>
>> Is glfs_file_lock() planned to be used here? If so, how do we specify
>> that it is a reclaim lock in this api?
>
> Yes. We have been discussing on that patch-set if we can use the same
> API. We should either have a separate field to pass reclaim flag or if
> we choose not to change its definition, then probably can have
> additional lock types -
>
> GLFS_LK_ADVISORY
> GLFS_LK_MANDATORY
>
> New lock-types
> GLFS_LK_RECLAIM_ADVISORY
> GLFS_LK_RECLAIM_MANDATORY
>

Either approach seems reasonable to me.

>>
>> We also would need to pass the reclaim_lock flag over rpc.
>
> To avoid new fop/rpc changes, I was considering to take xdata approach
> (similar to the way lock mode is passed in xdata for mandatory lock
> support) since the processing of reclamation doesn't differ much from
> the existing lk fop except for conflicting lock checks.
>

This looks ok to me.

Thanks,
Vijay