[Bugs] [Bug 1718562] New: flock failure (regression)

bugzilla at redhat.com bugzilla at redhat.com
Sat Jun 8 16:34:10 UTC 2019


            Bug ID: 1718562
           Summary: flock failure (regression)
           Product: GlusterFS
           Version: 6
          Hardware: x86_64
                OS: Linux
            Status: NEW
         Component: locks
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: jaco at uls.co.za
                CC: bugs at gluster.org
  Target Milestone: ---
    Classification: Community

Description of problem:

after a small number of flock rounds the lock remains behind indefinitely until
cleared with volume clear-locks, whereafter which normal operation resumes

I suspect this happens when there is contention on the lock.

I've got a setup where these locks are used syncronization mechanism.  So a
process on host a will take the lock, and release it on shutdown, at which
point another host is likely already trying to obtain the lock, and never
manages to do so (clearing granted allows the lock to proceed, but randomly
clearing locks is a high-risk operation).

Version-Release number of selected component (if applicable):  glusterfs 6.1
(confirmed working correctly on 3.12.3 and 4.0.2, suspected correct on 4.1.5
but no longer have a setup with 4.1.5 around).

How reproducible:  Trivial.  In the mentioned application it's on almost every
single lock attempt as far as I can determine.

Steps to Reproduce:

morpheus ~ # gluster volume info shared

Volume Name: shared
Type: Replicate
Volume ID: a4410662-b6e0-4ed0-b1e0-a1cbf168029c
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: morpheus:/mnt/gluster/shared
Brick2: r2d2:/mnt/gluster/shared
Options Reconfigured:
transport.address-family: inet
nfs.disable: on

morpheus ~ # mkdir /mnt/t
morpheus ~ # mount -t glusterfs localhost:shared /mnt/t
morpheus ~ # 

r2d2 ~ # mkdir /mnt/t
r2d2 ~ # mount -t glusterfs localhost:shared /mnt/t
r2d2 ~ # 

morpheus ~ # cd /mnt/t/
morpheus ~ # ls -l
total 0
morpheus /mnt/t # exec 3>lockfile; c=0; while flock -w 10 -x 3; do (( c++ ));
echo "Iteration $c passed"; exec 3<&-; exec 3>lockfile; done; echo "Failed
after $c iterations"; exec 3<&-
Iteration 1 passed
Iteration 2 passed
Iteration 3 passed

r2d2 /mnt/t # exec 3>lockfile; c=0; while flock -w 10 -x 3; do (( c++ )); echo
"Iteration $c passed"; exec 3<&-; exec 3>lockfile; done; echo "Failed after $c
iterations"; exec 3<&-
Iteration 1 passed
Iteration 2 passed
Failed after 2 iterations
r2d2 /mnt/t #

Iteration 100 passed
Iteration 101 passed
Iteration 102 passed
Failed after 102 iterations
morpheus /mnt/t # 

The two mounts failed at the same time, morpheus just passed more iterations
due to being started first.

Only iterating on one host I've had to stop it with ^C around 10k iterations,
which to me is sufficient indication that it's contention related.

After the above failure, I need to either rm the file and then it works again,
or I need to issue "gluster volume clear-locks shared /lockfile kind granted

On /tmp on my local machine I can run as much invocations of the loop above as
I want without issues (ext4 filesystem).

On glusterfs 3.12.3 and 4.0.2 I tried the above too, and stopped them after 10k

I have not observed the behaviour on glusterfs 4.1.5 which we used for a very
long time.

I either need a fix for this, or a way (prefereably with little no downtime,
total around 1.8TB of data) to downgrade glusterfs back to 4.1.X.  Or a way to
get around this reliably from within my application code (mostly control
scripts written in bash).

You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.

More information about the Bugs mailing list