[Bugs] [Bug 1718562] New: flock failure (regression)
bugzilla at redhat.com
bugzilla at redhat.com
Sat Jun 8 16:34:10 UTC 2019
https://bugzilla.redhat.com/show_bug.cgi?id=1718562
Bug ID: 1718562
Summary: flock failure (regression)
Product: GlusterFS
Version: 6
Hardware: x86_64
OS: Linux
Status: NEW
Component: locks
Severity: urgent
Assignee: bugs at gluster.org
Reporter: jaco at uls.co.za
CC: bugs at gluster.org
Target Milestone: ---
Classification: Community
Description of problem:
after a small number of flock rounds the lock remains behind indefinitely until
cleared with volume clear-locks, whereafter which normal operation resumes
again.
I suspect this happens when there is contention on the lock.
I've got a setup where these locks are used syncronization mechanism. So a
process on host a will take the lock, and release it on shutdown, at which
point another host is likely already trying to obtain the lock, and never
manages to do so (clearing granted allows the lock to proceed, but randomly
clearing locks is a high-risk operation).
Version-Release number of selected component (if applicable): glusterfs 6.1
(confirmed working correctly on 3.12.3 and 4.0.2, suspected correct on 4.1.5
but no longer have a setup with 4.1.5 around).
How reproducible: Trivial. In the mentioned application it's on almost every
single lock attempt as far as I can determine.
Steps to Reproduce:
morpheus ~ # gluster volume info shared
Volume Name: shared
Type: Replicate
Volume ID: a4410662-b6e0-4ed0-b1e0-a1cbf168029c
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: morpheus:/mnt/gluster/shared
Brick2: r2d2:/mnt/gluster/shared
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
morpheus ~ # mkdir /mnt/t
morpheus ~ # mount -t glusterfs localhost:shared /mnt/t
morpheus ~ #
r2d2 ~ # mkdir /mnt/t
r2d2 ~ # mount -t glusterfs localhost:shared /mnt/t
r2d2 ~ #
morpheus ~ # cd /mnt/t/
morpheus ~ # ls -l
total 0
morpheus /mnt/t # exec 3>lockfile; c=0; while flock -w 10 -x 3; do (( c++ ));
echo "Iteration $c passed"; exec 3<&-; exec 3>lockfile; done; echo "Failed
after $c iterations"; exec 3<&-
Iteration 1 passed
Iteration 2 passed
Iteration 3 passed
...
r2d2 /mnt/t # exec 3>lockfile; c=0; while flock -w 10 -x 3; do (( c++ )); echo
"Iteration $c passed"; exec 3<&-; exec 3>lockfile; done; echo "Failed after $c
iterations"; exec 3<&-
Iteration 1 passed
Iteration 2 passed
Failed after 2 iterations
r2d2 /mnt/t #
Iteration 100 passed
Iteration 101 passed
Iteration 102 passed
Failed after 102 iterations
morpheus /mnt/t #
The two mounts failed at the same time, morpheus just passed more iterations
due to being started first.
Only iterating on one host I've had to stop it with ^C around 10k iterations,
which to me is sufficient indication that it's contention related.
After the above failure, I need to either rm the file and then it works again,
or I need to issue "gluster volume clear-locks shared /lockfile kind granted
posix"
On /tmp on my local machine I can run as much invocations of the loop above as
I want without issues (ext4 filesystem).
On glusterfs 3.12.3 and 4.0.2 I tried the above too, and stopped them after 10k
iterations.
I have not observed the behaviour on glusterfs 4.1.5 which we used for a very
long time.
I either need a fix for this, or a way (prefereably with little no downtime,
total around 1.8TB of data) to downgrade glusterfs back to 4.1.X. Or a way to
get around this reliably from within my application code (mostly control
scripts written in bash).
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list