[Gluster-users] GlusterFS 3.0.2 small file read performance benchmark

[Ed W]

Well, "oplocks" are an SMB definition, but the basic concept of 
opportunistic locking is independent of the filesystem.  For example it 
appears that "oplocks" now appear in the NFS v4 standard under the name 
"delegations" (I would assume some variation of oplocks also exists in 
GFS and OCFS, but I'm not familiar with them)

The basic concept would potentially provide a huge performance boost for 
glusterfs because it allows cache coherent writeback caching.

In fact lets cut to the chase - what we desire is cache coherent 
writeback caching, ie reads to one server can be served from local 
client cache, but if the file is changed elsewhere then instantly our 
cache here is invalidated, and likewise we can write at will to a local 
copy of the file and allow it to get out of sync with the other servers, 
but as soon as some other server tries to read/write to our file then we 
must be notified and flush our cache (and request alternative locks or 
fall back to sync reads/writes)

How do we do this?  Well in NFS v3 and before and I believe in Glusterfs 
there is implemented only a "cache and hope" option, which caches data 
for a second or so and hopes the file doesn't change under us.  The 
improved algorithm is "opportunistic locking" where the client indicates 
to the server the desire to work with some data locally and get it out 
of sync with the server - the server then tracks that reservation and if 
some other client wants to access the data it pushes a lock break to the 
original client and informs it that it needs to fsync and run without 
the oplock

I believe that an oplock service this could be implemented via a new 
translator which works in conjunction with the read and writeback 
caching. Effectively it would be a two way lock manager, but it's job is 
somewhat simpler in that all it needs do is vary the existing caches on 
a per file basis.  So for example if we read some attributes for some 
files then at present they are blindly cached for X ms and then dropped, 
but our oplock translator will instead allow the attributes to be cached 
indefinitely until we get a push notification from the server side that 
our cache must be invalidated.  Same also with writes - we can use 
writeback cache as long as no one else has tried to read or write to our 
file, but as soon as someone else touches it we need to fsync and run 
without cache

I have had a very quick glance at the current locks module and it's 
quite a bit more complex than I might have guessed...  I had wondered if 
it might not be possible to make the locks module talk to the cache 
module and add server side lock breaking through that module?  
Essentially it's the addition of the "push" lock breaking which helps, 
so if we are reading away and some other client modifies a file then we 
need a feedback loop to invalide our read cache

Perhaps this is all implemented in glusterfs already though and I'm just 
missing the point...

Cheers

Ed W

On 02/03/2010 18:52, Tejas N. Bhise wrote:
> Ed,
>
> oplocks are implemented by SAMBA and it would not be a part of GlusterFS per se till we implement a native SAMBA translator ( something that would replace the SAMBA server itself with a thin SAMBA kind of a layer on top of GlusterFS itself ). We are doing that for NFS by building an NFS translator.
>
> At some point, it would be interesting to explore, clustered SAMBA using ctdb, where two GlusterFS clients can export the same volume. ctdb itself seems to be coming up well now.
>
> Regards,
> Tejas.
>
> ----- Original Message -----
> From: "Ed W"<lists at wildgooses.com>
> To: "Gluster Users"<gluster-users at gluster.org>
> Sent: Wednesday, March 3, 2010 12:10:47 AM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi
> Subject: Re: [Gluster-users] GlusterFS 3.0.2 small file	read	performance	benchmark
>
> On 01/03/2010 20:44, Ed W wrote:
>    
>> I believe samba (and probably others) use a two way lock escalation
>> facility to mitigate a similar problem.  So you can "read-lock" or
>> phrased differently, "express your interest in caching some
>> files/metadata" and then if someone changes what you are watching the
>> lock break is pushed to you to invalidate your cache.
>>      
> Seems NFS v4 implements something similar via "delegations" (not
> believed implemented in linux NFSv4 though...)
>
> In samba the equivalent are called "op locks"
>
> I guess this would be a great project for someone interested to work on
> - op-lock translator for gluster
>
> Ed W
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>