[Bugs] [Bug 1412886] New: Extra lookup/ fstats are sent over the network when a brick is down.
bugzilla at redhat.com
bugzilla at redhat.com
Fri Jan 13 04:13:21 UTC 2017
https://bugzilla.redhat.com/show_bug.cgi?id=1412886
Bug ID: 1412886
Summary: Extra lookup/fstats are sent over the network when a
brick is down.
Product: GlusterFS
Version: 3.9
Component: replicate
Keywords: Triaged
Assignee: bugs at gluster.org
Reporter: ravishankar at redhat.com
CC: bugs at gluster.org
Depends On: 1409206
Blocks: 1410025
+++ This bug was initially created as a clone of Bug #1409206 +++
Description of problem:
When a brick is killed in a replica and `dd` is run, we see a lot of fstats
being sent over the network, with a small (but very real) reduction in write
throughput.
Throughput and profile info on a random brick when all bricks are up:
==================================================================
root at tuxpad fuse_mnt$ dd if=/dev/zero of=FILE bs=1024 count=10240
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 2.9847 s, 3.5 MB/s
Brick: 127.0.0.2:/home/ravi/bricks/brick1
-----------------------------------------
Cumulative Stats:
Block Size: 1024b+
No. of Reads: 0
No. of Writes: 10240
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2 RELEASE
0.00 0.00 us 0.00 us 0.00 us 6 RELEASEDIR
0.01 30.50 us 27.00 us 34.00 us 2 INODELK
0.01 70.00 us 70.00 us 70.00 us 1 OPEN
0.01 35.50 us 19.00 us 52.00 us 2 FLUSH
0.03 96.50 us 83.00 us 110.00 us 2 GETXATTR
0.04 253.00 us 253.00 us 253.00 us 1 TRUNCATE
0.07 225.50 us 202.00 us 249.00 us 2 FXATTROP
0.15 153.17 us 47.00 us 656.00 us 6 STATFS
0.17 537.00 us 207.00 us 867.00 us 2 XATTROP
0.22 685.00 us 22.00 us 1348.00 us 2 FINODELK
0.62 255.67 us 104.00 us 928.00 us 15 LOOKUP
98.66 59.72 us 35.00 us 4772.00 us 10240 WRITE
Duration: 673 seconds
Data Read: 0 bytes
Data Written: 10485760 bytes
Throughput and profile info of one of the 'up' bricks when one brick is down
===========================================================================
0:root at tuxpad fuse_mnt$ dd if=/dev/zero of=FILE bs=1024 count=10240
10240+0 records in
10240+0 records out
10485760 bytes (10 MB) copied, 4.24494 s, 2.5 MB/s
Brick: 127.0.0.2:/home/ravi/bricks/brick1
-----------------------------------------
Cumulative Stats:
Block Size: 1024b+
No. of Reads: 0
No. of Writes: 10240
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 2 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5 RELEASEDIR
0.01 98.00 us 98.00 us 98.00 us 1 OPEN
0.01 57.50 us 43.00 us 72.00 us 2 INODELK
0.01 126.00 us 126.00 us 126.00 us 1 GETXATTR
0.02 184.00 us 184.00 us 184.00 us 1 TRUNCATE
0.02 113.00 us 109.00 us 117.00 us 2 FXATTROP
0.02 122.00 us 16.00 us 228.00 us 2 FLUSH
0.02 132.00 us 38.00 us 226.00 us 2 FINODELK
0.08 418.00 us 283.00 us 553.00 us 2 XATTROP
0.21 763.00 us 122.00 us 1630.00 us 3 LOOKUP
41.23 44.83 us 36.00 us 490.00 us 10240 WRITE
58.38 63.47 us 46.00 us 888.00 us 10240 FSTAT
Duration: 75 seconds
Data Read: 0 bytes
Data Written: 10485760 bytes
--- Additional comment from Worker Ant on 2016-12-30 04:58:52 EST ---
REVIEW: http://review.gluster.org/16309 (afr: Avoid resetting event_gen when
brick is always down) posted (#1) for review on master by Ravishankar N
(ravishankar at redhat.com)
--- Additional comment from Worker Ant on 2017-01-09 12:12:20 EST ---
REVIEW: http://review.gluster.org/16309 (afr: Avoid resetting event_gen when
brick is always down) posted (#2) for review on master by Ravishankar N
(ravishankar at redhat.com)
--- Additional comment from Worker Ant on 2017-01-12 00:14:55 EST ---
REVIEW: http://review.gluster.org/16309 (afr: Avoid resetting event_gen when
brick is always down) posted (#3) for review on master by Ravishankar N
(ravishankar at redhat.com)
--- Additional comment from Worker Ant on 2017-01-12 01:56:57 EST ---
REVIEW: http://review.gluster.org/16309 (afr: Avoid resetting event_gen when
brick is always down) posted (#4) for review on master by Ravishankar N
(ravishankar at redhat.com)
--- Additional comment from Worker Ant on 2017-01-12 21:54:07 EST ---
COMMIT: http://review.gluster.org/16309 committed in master by Jeff Darcy
(jdarcy at redhat.com)
------
commit 522640be476a3f97dac932f7046f0643ec0ec2f2
Author: Ravishankar N <ravishankar at redhat.com>
Date: Fri Dec 30 14:57:17 2016 +0530
afr: Avoid resetting event_gen when brick is always down
Problem:
__afr_set_in_flight_sb_status(), which resets event_gen to zero, is
called if failed_subvols[i] is non-zero for any brick. But
failed_subvols[i]
is true even if the brick was down *before* the transaction started.
Hence say if 1 brick is down in a replica-3, every writev that comes
will trigger an inode refresh because of this resetting, as seen from
the no. of FSTATs in the profile info in the BZ.
Fix:
Reset event gen only if the brick was previously a valid read child and
the FOP failed on it the first time.
Also `s/afr_inode_read_subvol_reset/afr_inode_event_gen_reset` because
the function only resets event gen and not the data/metadata readable.
Change-Id: I603ae646cbde96995c35db77916e2ed80b602a91
BUG: 1409206
Signed-off-by: Ravishankar N <ravishankar at redhat.com>
Reviewed-on: http://review.gluster.org/16309
Smoke: Gluster Build System <jenkins at build.gluster.org>
Reviewed-by: Pranith Kumar Karampuri <pkarampu at redhat.com>
Tested-by: Pranith Kumar Karampuri <pkarampu at redhat.com>
NetBSD-regression: NetBSD Build System <jenkins at build.gluster.org>
CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1409206
[Bug 1409206] Extra lookup/fstats are sent over the network when a brick is
down.
https://bugzilla.redhat.com/show_bug.cgi?id=1410025
[Bug 1410025] Extra lookup/fstats are sent over the network when a brick is
down.
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list