[Gluster-users] Concurrency limitation?

[Jeff Darcy]

On Tue, 07 Feb 2012 20:03:18 +0100
Arnold Krille <arnold at arnoldarts.de> wrote:

> > GlusterFS is a file-level protocol, more like NFS, and as far as I
> > know there is no inherent locking between clients.
> 
> There has to be locking. Otherwise two apps on two machines opening
> the same file for writing would destroy each others changes.

That's not exactly true.  For one thing, non-overlapping non-extending
writes won't affect each other.

> Therefor
> one client has to gather locks on all brick filesystems (which is the
> same as synchronizing access with all other clients).

Again, not really.  For a write, only *that file* would need to be
locked, and only on the bricks that actually contain that file.
Without replication or striping, the number of bricks holding a file is
exactly one, and all necessary synchronization can occur within that
one server.  With striping, serialization only needs to occur across
bricks holding stripes affected by that write - often just one again.
With replication, there does need to be more complex coordination and
there's a whole protocol in place to handle that . . . but, again, it
only affects those bricks holding replicas of the file being written.
Locking across all bricks in a volume only needs to occur for volume
level operations (e.g. volume parameter changes).

> > Why, when I simply read
> > a file, would a lock need to be created anywhere?
> 
> One client opens the file for reading, the other opens the file for
> trunc|write. What do you get on the first client? How should this
> scenario be any save without some kind of locking. And since its a
> distributed filesystem, it has to be distributed locking.

There are other solutions besides locking.  One is cache invalidation -
a write causes other clients to be notified that any cached data
related to the just-written range is invalid.  Another is polling -
part of a read is to validate the freshness of cached data (e.g. by
checking timestamps or generation numbers) before using it.  There are
still other "optimistic concurrency control" alternatives that I won't
go into right now, because I think the point is sufficiently made.

> I might be wrong and glusterfs really doesn't do any locking to
> prevent concurrent write accesses. But if thats true, I think this
> rules out glusterfs for any usage above "proof-of-concept". And thats
> why I doubt that glusterfs doesn't do any locking (or synchronization
> of the fs-locks underneath).

It does synchronization, but it's much finer-grained than you seem to
think and it uses many techniques besides locking.