[Gluster-devel] File lock problem on new files

[Kevan Benson]

I posted this a while back: 
http://lists.nongnu.org/archive/html/gluster-devel/2007-08/msg00124.html

In summary: Performing a flock operation on a newly created file does 
not always lock it from other clients.

Process: Open a new file and obtain a lock (flock in my tests) within 
the same process.  Before closing the file, open and try to obtain a 
lock from a separate process from a different glusterfs client.  There 
is no blocking.

Notes: If I create the file beforehand (touch it), the flock operation 
performs as expected.  Looking at the servers, the file is created 
immediately from the first client, the second client is just not 
blocking on the flock call.

Here's a trasncript of operations on the servers, to illustrate the 
problem.  Both servers are time synced, drift between them is sub 1 second:

# Client A
# ll /mnt/glusterfs/
total 8
-rwxr-xr-x  1 root root 746 Sep 21 12:02 testlock.pl

# Client B
# ll /mnt/glusterfs/
total 8
-rwxr-xr-x  1 root root 746 Sep 21 12:02 testlock.pl

# Client A
# /mnt/glusterfs/testlock.pl /mnt/glusterfs/testfile 10
testing flock on lockfile /mnt/glusterfs/testfile with 10 second delay 
on php5-a.graphita.com
opening file
Fri Sep 21 14:26:19 2007 :: opened file
Fri Sep 21 14:26:19 2007 :: locking file
Fri Sep 21 14:26:19 2007 :: locked file
Fri Sep 21 14:26:29 2007 :: unlocked file

# Client B
# /mnt/glusterfs/testlock.pl /mnt/glusterfs/testfile 10
testing flock on lockfile /mnt/glusterfs/testfile with 10 second delay 
on php5-b.graphita.com
opening file
Fri Sep 21 14:26:21 2007 :: opened file
Fri Sep 21 14:26:21 2007 :: locking file
Fri Sep 21 14:26:21 2007 :: locked file
Fri Sep 21 14:26:31 2007 :: unlocked file


Notice that the second client ignored the lock of the first?  Not let's 
try this again but with the file already existing:

# Client A
# rm -f /mnt/glusterfs/testfile
# ll /mnt/glusterfs/
total 8
-rwxr-xr-x  1 root root 746 Sep 21 12:02 testlock.pl
# touch /mnt/glusterfs/testfile
# ll /mnt/glusterfs/
total 12
-rw-r--r--  1 root root   0 Sep 21 14:29 testfile
-rwxr-xr-x  1 root root 746 Sep 21 12:02 testlock.pl

# Client B
# ll /mnt/glusterfs/
total 12
-rw-r--r--  1 root root   0 Sep 21 14:29 testfile
-rwxr-xr-x  1 root root 746 Sep 21 12:02 testlock.pl

# Client A
# /mnt/glusterfs/testlock.pl /mnt/glusterfs/testfile 10
testing flock on lockfile /mnt/glusterfs/testfile with 10 second delay 
on php5-a.graphita.com
opening file
Fri Sep 21 14:30:33 2007 :: opened file
Fri Sep 21 14:30:33 2007 :: locking file
Fri Sep 21 14:30:33 2007 :: locked file
Fri Sep 21 14:30:43 2007 :: unlocked file

# Client B
# /mnt/glusterfs/testlock.pl /mnt/glusterfs/testfile 10
testing flock on lockfile /mnt/glusterfs/testfile with 10 second delay 
on php5-b.graphita.com
opening file
Fri Sep 21 14:30:36 2007 :: opened file
Fri Sep 21 14:30:36 2007 :: locking file
Fri Sep 21 14:30:43 2007 :: locked file
Fri Sep 21 14:30:53 2007 :: unlocked file


Now it flocks fine.  Client B had to wait for Client A to unlock for it 
could lock.  Order of operations doesn't seem to matter, nor which 
client creates the file compared to which locks first when the file is 
created prior to lock testing.

Here's the contents of testlock.pl:
#!/usr/bin/perl

$| = 1;
use Fcntl ':flock';

my $file = shift or "./testlockfile";
my $delay = shift or 10;
my $hostname = `hostname`;
chomp($hostname);
print "testing flock on lockfile $file with $delay second delay on 
$hostname\n";
print "opening file\n";
open(my $testfile, '>>', $file) or die("$!\n");
print localtime(time())." :: opened file\n";
print localtime(time())." :: locking file\n";
flock($testfile, LOCK_EX) or die("Error locking :: $!\n");
print localtime(time())." :: locked file\n";
print $testfile "$hostname start ".localtime(time())."\n";
sleep $delay;
print $testfile "$hostname end ".localtime(time())."\n";
flock($testfile, LOCK_UN) or die("Error locking :: $!\n");
print localtime(time())." :: unlocked file\n";
exit 0;