[Gluster-devel] distributed locking

[Székelyi Szabolcs]

Hi,

I'm trying to put together a fully redundant, no-SPOF storage system
using GlusterFS.

Everything works fine as long as I don't introduce locking, which is
essential if one wishes to use the storage eg. as a backend for a mail
server.

On each server I have a features/posix-locks translator on top of a
storage/posix brick. This volume is exported via protocol/server, and
AFR'ed together with the same (features/posix-locks) bricks from the
other servers. The order of the subvolumes for each AFR is the same on
all servers. (So that in theory, for the first time, all servers ask the
first server to lock a file, then all of them ask the second server, and
so on).

The resulting AFR volume is exported again, and mounted into a directory
(using 'localhost' as remote-host). You can find the exact configuration
at aht end of this mail. GlusterFS is a fresh tla checkout.

I expect that if a file is locked on one of the servers, the same file
should get locked on the other servers, too. So a client trying to lock
the same file on another server, it should get blocked or EAGAIN returned.

Now I test this setup with locktests, suggested on the GlusterFS QA
page. I use the network-based tests to simulate multiple server
processes locking and releasing files.

Here are the results from the host running the master locking process:

> ./locktests -n 30 -f /mnt/glusterfs/locktest -c 2
> Init
> process initalization
> ............................................................
> --------------------------------------
> 
> TEST : TRY TO WRITE ON A READ  LOCK:==============================
> TEST : TRY TO WRITE ON A WRITE LOCK:==============================
> TEST : TRY TO READ  ON A READ  LOCK:==============================
> TEST : TRY TO READ  ON A WRITE LOCK:==============================
> TEST : TRY TO SET A READ  LOCK ON A READ  LOCK:==============================
> TEST : TRY TO SET A WRITE LOCK ON A WRITE LOCK:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> TEST : TRY TO SET A WRITE LOCK ON A READ  LOCK:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> TEST : TRY TO SET A READ  LOCK ON A WRITE LOCK:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> TEST : TRY TO READ LOCK THE WHOLE FILE BYTE BY BYTE:==============================
> TEST : TRY TO WRITE LOCK THE WHOLE FILE BYTE BY BYTE:==============================
> 
> process number : 30 - Remote clients: 2 local client 1 - Total client 3 - Total concurent tests: 90
> process number running test successfully :
> 90 process of 90 successfully ran test : WRITE ON A READ  LOCK
> 90 process of 90 successfully ran test : WRITE ON A WRITE LOCK
> 90 process of 90 successfully ran test : READ  ON A READ  LOCK
> 90 process of 90 successfully ran test : READ  ON A WRITE LOCK
> 90 process of 90 successfully ran test : SET A READ  LOCK ON A READ  LOCK
> 88 process of 90 successfully ran test : SET A WRITE LOCK ON A WRITE LOCK
> 88 process of 90 successfully ran test : SET A WRITE LOCK ON A READ  LOCK
> 30 process of 90 successfully ran test : SET A READ  LOCK ON A WRITE LOCK
> 30 process of 90 successfully ran test : READ LOCK THE WHOLE FILE BYTE BY BYTE
> 30 process of 90 successfully ran test : WRITE LOCK THE WHOLE FILE BYTE BY BYTE

On the other two slave servers I see:

> 0
> Init
> process initalization
> ............................................................Maitre CLient - fdServeur=3
> ==============================
> ==============================
> ==============================
> ==============================
> ==============================
> =xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> =xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
> ==============================
> 
> ==============================
> 
> ==============================
> Fin du programme en cours...
> Bye :)

Do you have any idea how to configure GlusterFS so the test runs
successfully? Is a setup like this possible with the current posix-locks
translator?

Thanks,
--
cc

Here are my configs:

==server (this is from the server named 'drbubo', others are similar)==

volume data-posix
  type storage/posix
  option directory /srv/glusterfs
end-volume

volume data-drbubo
  type features/posix-locks
  subvolumes data-posix
end-volume

volume data-ursula
  type protocol/client
  option transport-type tcp/client
  option remote-host ursula
  option remote-subvolume data-ursula
end-volume

volume data-sf3
  type protocol/client
  option transport-type tcp/client
  option remote-host sf3
  option remote-subvolume data-sf3
end-volume

volume data-afr
  type cluster/afr
  subvolumes data-ursula data-drbubo data-sf3
end-volume

volume server
  type protocol/server
  subvolumes data-drbubo data-afr
  option transport-type tcp/server
  option auth.ip.data-drbubo.allow <IPs of all servers>
  option auth.ip.data-afr.allow   127.0.0.1
end-volume

==client==

volume client
  type protocol/client
  option transport-type tcp/client
  option remote-host localhost
  option remote-subvolume data-afr
end-volume