[Gluster-devel] Support to reclaim locks (posix) provided lkowner & range matches

[Soumya Koduri]

Hi Vijay,

On 07/26/2016 12:13 AM, Vijay Bellur wrote:
> On 07/22/2016 08:44 AM, Soumya Koduri wrote:
>> Hi,
>>
>> In certain scenarios (esp.,in highly available environments), the
>> application may have to fail-over/connect to a different glusterFS
>> client while the I/O is happening. In such cases until there is a ping
>> timer expiry and glusterFS server cleans up the locks held by the older
>> glusterFS client, the application will not be able to reclaim their lost
>> locks. To avoid that we need support in Gluster to let clients reclaim
>> the existing locks provided lkwoner and the lock range matches.
>
>
> If the server detects a disconnection, it goes about cleaning up the
> locks held by the disconnected client. Only if the failover connection
> happens before this server cleanup the outlined scheme would work.Since
> there is no ping timer on the server, do you propose to have a grace
> timer on the server?

But we are looking for a solution which can work in active-active 
configuration as well. We need to handle cases where in the connection 
between server and the old-client is still in use, which can happen 
during load-balancing or failback.

Different cases which I can outline are:

Application Client - (AC)
Application/GlusterClient 1 - GC1
Application/GlusterClient 2 - GC2
Gluster Server (GS)

1) Active-Passive config  (service gone down)

AC ----> GC1  ----> GS (GC2 is not active)

	| (failover)
	v

AC ----> GC2  ----> GS (GC1 connection gets dropped and GC2 establishes 
connection)

In this case, we can have grace timer to allow reclaims only for certain 
time post GC2 (any) rpc connection establishment.

2) Active-Active config  (service gone down)

AC ----> GC1  ----> GS
		     ^
		     |
          GC2  -------

	| (failover)
	v

AC ----> GC2  ----> GS (GC1 connection gets dropped)

The grace timer then shall not get triggered in this case. But at-least 
the locks from GC1 gets cleaned post its connection cleanup.


3) Active-Active config  (both the services active/load-balancing)
This is the trick one.

AC ----> GC1  ----> GS
		     ^
		     |
          GC2  -------

	| (load-balancing/failback)
	v

	 GC1  ----> GS
		     ^
		     |
AC ----> GC2  -------

The locks taken by GC1 shall end up being on the server for ever unless 
we restart either GC1 or the server.

Considering above cases, looks like we may need to allow reclaim of the 
locks all the time. Please suggest if I have missed out any details.

>
>>
>> For client-side support, I am thinking if we can integrate with the new
>> lock API being introduced as part of mandatory lock support in gfapi [2]
>>
>
> Is glfs_file_lock() planned to be used here? If so, how do we specify
> that it is a reclaim lock in this api?

Yes. We have been discussing on that patch-set if we can use the same 
API. We should either have a separate field to pass reclaim flag or if 
we choose not to change its definition, then probably can have 
additional lock types -

GLFS_LK_ADVISORY
GLFS_LK_MANDATORY

New lock-types
GLFS_LK_RECLAIM_ADVISORY
GLFS_LK_RECLAIM_MANDATORY

>
> We also would need to pass the reclaim_lock flag over rpc.

To avoid new fop/rpc changes, I was considering to take xdata approach 
(similar to the way lock mode is passed in xdata for mandatory lock 
support) since the processing of reclamation doesn't differ much from 
the existing lk fop except for conflicting lock checks.

http://review.gluster.org/#/c/14986/2/xlators/features/locks/src/posix.c

Please let me know your thoughts.

Thanks,
Soumya

>
> -Vijay