[Gluster-users] Possible SHD stalling
Jaco Kroon
jaco at uls.co.za
Sat May 2 20:42:44 UTC 2020
Hi All,
So we've got a cluster currently running on glusterfs 7.3
Heal performance is absolutely terrible (it starts of great, runs for a
while then completely trips up).
Once the SHD trips up, access speeds to the cluster also becomes poor.
And with poor I mean directory listings that normally takes well under
500ms takes upwards of 10s, but I've seen upwards of 40s (although a
follow up ls immediately after is nice and fast again, temporarily).
With all SHDs killed this particular ls is as per normal (around 30-50ms).
The SHD logs doesn't provide any usable clues, the only variance in logs
is this, just before all daemons across the cluster crawls to a near halt.
[2020-05-02 19:55:58.627036] I [MSGID: 108026]
[afr-self-heal-common.c:1747:afr_log_selfheal]
0-vpbx_shared-replicate-0: Completed entry selfheal on
5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1
[2020-05-02 19:55:58.627058] I [MSGID: 108026]
[afr-self-heal-common.c:1747:afr_log_selfheal]
0-vpbx_shared-replicate-0: Completed entry selfheal on
c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1
[2020-05-02 19:55:59.125001] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]
[2020-05-02 19:55:59.339803] I [MSGID: 108026]
[afr-self-heal-common.c:1747:afr_log_selfheal]
0-vpbx_shared-replicate-0: Completed entry selfheal on
2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1
[2020-05-02 19:55:59.490387] I [MSGID: 108026]
[afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
0-vpbx_shared-replicate-0: performing metadata selfheal on
527a63a3-9acd-4c3f-b95c-5c719325c973
A few more entries will complete overall and then it'll be a file every
other second. Once I saw this I ran a few more tests, and on every test
just prior to slowing to a crawl, that entry is there.
There is a perceived improvement when lowering the thread count, but not
significant enough to confirm a correlation. Even
with**cluster.shd-max-threads=1 this still happens (gets a few tens
thousand files completed instead of few hundred, at 32 it gets to maybe
400 to 500 before it stalls, 16 does maybe a thousand).*
*
There are probably performant bursts hereafter, because from experience
we know this process takes about two months. But this time around it
seems to be much, much worse. Might be because that was healing a
single brick to another brick, so less locking going? At the moment the
replicas are basically trying to populate the newly added arbiters.
Also, we haven't messed as greatly with the health check settings
(queues and threads and locks nearly as much as since the arbiter
addition because performance has simply dropped to non-acceptable levels.
Unfortunately granular entry heal isn't on at the moment (nor can I
switch it on due to a few million outstanding heals). This used to be a
2x2 cluster. We then moved one of the bricks to a newly added server
using replace-brick. What a frightening process this is. We then
proceeded to add the arbiters.
Some of the settings below was absolute shots in the dark, based on the
little information we've been able to track down so far.
Kind Regards,
Jaco
The current gluster volume info (this used to be 2x2
distributed-replicate, aiming to get it to 3x (2 + 1) ... at current
rates we might get there by 2021):
Volume Name: glvol
Type: Distributed-Replicate
Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: zoidberg:/mnt/gluster/a
Brick2: flexo:/mnt/gluster/b
Brick3: bender:/mnt/gluster/c (arbiter)
Brick4: bender:/mnt/gluster/a
Brick5: zoidberg:/mnt/gluster/b
Brick6: flexo:/mnt/gluster/c (arbiter)
Options Reconfigured:
*cluster.heal-timeout: 5 <-- hoping this would at least shorten the
blockages.*
cluster.locking-scheme: granular
cluster.heal-wait-queue-length: 10240
*diagnostics.count-fop-hits: on**
**diagnostics.latency-measurement: on <-- was hoping profiling might
provide some insight. Going to switch this off now.*
cluster.background-self-heal-count: 128
*cluster.shd-wait-qlength: 131072**
**cluster.shd-max-threads: 32 <-- even with this disk IO never goes
over 10% utilization. CPU goes to about 2 cores. So it's lock
contention that's slowing it down we reckon.*
*cluster.metadata-self-heal: off**
**cluster.entry-self-heal: off**
**cluster.data-self-heal: off**<-- these kill performance.
*transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
*cluster.self-heal-daemon: yes <-- disabling/enabling this doesn't help,
have to kill the shd and start ... force to get things moving along again.*
performance.io-thread-count: 16
performance.high-prio-threads: 16
performance.normal-prio-threads: 16
performance.low-prio-threads: 16
performance.least-prio-threads: 4
cluster.self-heal-window-size: 512
cluster.self-heal-readdir-size: 131072
client.event-threads: 8
server.event-threads: 16
network.ping-timeout: 2
*cluster.data-self-heal-algorithm: full <-- most files are small (<1MB),
on dual 1gbit links (so up to 2gbit, but b/w never goes over 5mbit anyway).*
performance info:
Brick: zoidberg:/mnt/gluster/a
-----------------------------------------
Cumulative Stats:
Block Size: 1b+
8b+ 16b+
No. of Reads: 0
0 0
No. of Writes: 117
195 367
Block Size: 32b+
64b+ 128b+
No. of Reads: 306
0 6
No. of Writes: 1016
9 134
Block Size: 256b+ 512b+
1024b+
No. of Reads: 107
58 485
No. of Writes: 731
73 168
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 43
59 383
No. of Writes: 295
475 10562
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 347
380 634
No. of Writes: 1487
6369 3970
Block Size: 131072b+
No. of Reads: 35704
No. of Writes: 12398
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 957293
FORGET
0.00 0.00 us 0.00 us 0.00 us 1455091
RELEASE
0.00 0.00 us 0.00 us 0.00 us 69951
RELEASEDIR
0.00 16.20 us 5.43 us 100.95 us
113 IPC
0.00 687.33 us 80.53 us 15414.03 us 30
REMOVEXATTR
0.00 840.76 us 72.71 us 37648.39 us 52
FTRUNCATE
0.00 138.49 us 36.66 us 25280.24 us
1385 STAT
0.00 364.93 us 63.45 us 15957.66 us 810
FGETXATTR
0.00 2208.82 us 127.23 us 267092.17 us
137 LINK
0.00 37595.33 us 1677.46 us 218313.32 us
12 FSYNC
0.00 675.61 us 106.82 us 114351.01 us 698
TRUNCATE
0.00 3995.66 us 147.00 us 109544.46 us 188
RENAME
0.00 453.11 us 53.37 us 104848.94 us 1680
SETATTR
0.00 6004.32 us 229.33 us 151322.58 us
127 MKNOD
0.00 68.74 us 16.11 us 61050.93 us 11412
FINODELK
0.00 3793.44 us 68.61 us 411523.70 us 221
SETXATTR
0.00 28669.99 us 227.28 us 138988.41 us 30
SYMLINK
0.00 56.18 us 7.31 us 41484.37 us
22640 FLUSH
0.00 105.64 us 19.34 us 351055.72 us 21693
STATFS
0.00 23642.90 us 244.19 us 234912.25 us
178 MKDIR
0.00 537.38 us 31.73 us 296911.67 us
9653 FSTAT
0.00 214.87 us 25.71 us 1712020.99 us
39054 READ
0.01 147.08 us 1.47 us 241611.80 us 69951
OPENDIR
0.01 3899.84 us 117.22 us 503422.21 us
3180 RMDIR
0.01 327.08 us 55.69 us 420376.80 us
38366 WRITE
0.02 5580.36 us 210.11 us 1029702.30 us 6169
CREATE
0.05 92.69 us 29.94 us 687955.86 us 1159936
READLINK
0.40 543.50 us 36.45 us 2430653.54 us
1449307 OPEN
0.48 87643.67 us 70.20 us 661999.51 us 10787
FXATTROP
0.53 2342.72 us 20.91 us 4157542.43 us 447821
UNLINK
0.53 558734.79 us 24.40 us 1399487.63 us
1889 LK
0.78 145786.31 us 19.06 us 37490825.73 us 10687
READDIR
0.99 2426.23 us 12.47 us 24176123.13 us 809430
GETXATTR
1.45 676.83 us 9.30 us 193177067.25 us
4260021 ENTRYLK
1.76 38325.38 us 21.44 us 46693575.93 us 91439
READDIRP
4.44 1200.31 us 9.10 us 2559377.90 us 7352811
INODELK
9.30 1849.24 us 9.28 us 35299050.00 us
9989221 LOOKUP
79.24 47791.46 us 76.77 us 10488824.26 us 3294642
XATTROP
0.00 0.00 us 0.00 us 0.00 us 3194806
UPCALL
0.00 0.00 us 0.00 us 0.00 us 5311
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 54
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 1907
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 3189457
CI_FORGET
Duration: 209524 seconds
Data Read: 4775365531 bytes
Data Written: 2354108664 bytes
Interval 0 Stats:
Block Size: 1b+
8b+ 16b+
No. of Reads: 0
0 0
No. of Writes: 117
195 367
Block Size: 32b+
64b+ 128b+
No. of Reads: 306
0 6
No. of Writes: 1016
9 134
Block Size: 256b+ 512b+
1024b+
No. of Reads: 107
58 485
No. of Writes: 731
73 168
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 43
59 383
No. of Writes: 295
475 10562
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 347
380 634
No. of Writes: 1487
6369 3970
Block Size: 131072b+
No. of Reads: 35704
No. of Writes: 12398
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 957293
FORGET
0.00 0.00 us 0.00 us 0.00 us 1455091
RELEASE
0.00 0.00 us 0.00 us 0.00 us 69951
RELEASEDIR
0.00 16.20 us 5.43 us 100.95 us
113 IPC
0.00 687.33 us 80.53 us 15414.03 us 30
REMOVEXATTR
0.00 840.76 us 72.71 us 37648.39 us 52
FTRUNCATE
0.00 138.49 us 36.66 us 25280.24 us
1385 STAT
0.00 363.20 us 63.45 us 15957.66 us 810
FGETXATTR
0.00 2208.82 us 127.23 us 267092.17 us
137 LINK
0.00 37595.33 us 1677.46 us 218313.32 us
12 FSYNC
0.00 675.61 us 106.82 us 114351.01 us 698
TRUNCATE
0.00 3995.66 us 147.00 us 109544.46 us 188
RENAME
0.00 453.11 us 53.37 us 104848.94 us 1680
SETATTR
0.00 6004.32 us 229.33 us 151322.58 us
127 MKNOD
0.00 68.74 us 16.11 us 61050.93 us 11412
FINODELK
0.00 3793.44 us 68.61 us 411523.70 us 221
SETXATTR
0.00 28669.99 us 227.28 us 138988.41 us 30
SYMLINK
0.00 56.19 us 7.31 us 41484.37 us
22640 FLUSH
0.00 105.61 us 19.34 us 351055.72 us 21693
STATFS
0.00 23642.90 us 244.19 us 234912.25 us
178 MKDIR
0.00 537.38 us 31.73 us 296911.67 us
9653 FSTAT
0.00 214.91 us 25.71 us 1712020.99 us
39054 READ
0.01 147.08 us 1.47 us 241611.80 us 69951
OPENDIR
0.01 3899.84 us 117.22 us 503422.21 us
3180 RMDIR
0.01 327.13 us 55.69 us 420376.80 us
38366 WRITE
0.02 5580.36 us 210.11 us 1029702.30 us 6169
CREATE
0.05 92.69 us 29.94 us 687955.86 us 1159936
READLINK
0.40 543.49 us 36.45 us 2430653.54 us
1449307 OPEN
0.48 87643.67 us 70.20 us 661999.51 us 10787
FXATTROP
0.53 2342.57 us 20.91 us 4157542.43 us 447821
UNLINK
0.53 558416.10 us 24.40 us 1399487.63 us
1889 LK
0.78 145786.31 us 19.06 us 37490825.73 us 10687
READDIR
0.99 2426.24 us 12.47 us 24176123.13 us 809430
GETXATTR
1.45 676.75 us 9.30 us 193177067.25 us
4260021 ENTRYLK
1.76 38324.59 us 21.44 us 46693575.93 us 91439
READDIRP
4.44 1200.37 us 9.10 us 2559377.90 us 7352811
INODELK
9.29 1848.94 us 9.28 us 35299050.00 us
9989221 LOOKUP
79.24 47792.42 us 76.77 us 10488824.26 us 3294642
XATTROP
0.00 0.00 us 0.00 us 0.00 us 3194806
UPCALL
0.00 0.00 us 0.00 us 0.00 us 5311
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 54
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 1907
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 3189457
CI_FORGET
Duration: 209524 seconds
Data Read: 4775365531 bytes
Data Written: 2354108664 bytes
Brick: zoidberg:/mnt/gluster/b
-----------------------------------------
Cumulative Stats:
Block Size: 1b+
2b+ 8b+
No. of Reads: 0
0 0
No. of Writes: 53
1 60
Block Size: 16b+
32b+ 64b+
No. of Reads: 0
213 2
No. of Writes: 14
1057 15
Block Size: 128b+
256b+ 512b+
No. of Reads: 0
41 146
No. of Writes: 59
715 80
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 224
9 15
No. of Writes: 163
282 404
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 52
74 210
No. of Writes: 7415
1430 6860
Block Size: 65536b+ 131072b+
262144b+
No. of Reads: 299
15313 1
No. of Writes: 2458
12378 1
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 686733
FORGET
0.00 0.00 us 0.00 us 0.00 us 837434
RELEASE
0.00 0.00 us 0.00 us 0.00 us 87338
RELEASEDIR
0.00 22.69 us 7.16 us 332.61 us
116 IPC
0.00 119.93 us 70.56 us 272.28 us 57
FTRUNCATE
0.00 152.25 us 86.82 us 340.63 us 65
FGETXATTR
0.00 2336.06 us 90.05 us 20509.81 us 23
REMOVEXATTR
0.00 631.26 us 123.17 us 51803.77 us
121 LINK
0.00 369.27 us 22.64 us 145090.48 us
490 LK
0.00 360.85 us 45.14 us 46846.98 us
540 STAT
0.00 1190.96 us 117.40 us 69781.85 us 200
TRUNCATE
0.00 245.72 us 54.91 us 123985.76 us 1482
SETATTR
0.00 42139.10 us 3313.56 us 312273.21 us
12 FSYNC
0.00 55.83 us 8.26 us 49649.35 us
10047 FLUSH
0.00 57.52 us 14.62 us 820.59 us 9977
FINODELK
0.00 3191.60 us 143.06 us 96214.75 us 188
RENAME
0.00 32862.86 us 215.38 us 83851.13 us 25
SYMLINK
0.00 11392.03 us 230.70 us 105471.90 us
118 MKNOD
0.00 134.63 us 28.08 us 36326.43 us
12587 FSTAT
0.00 112.32 us 19.91 us 381241.34 us 21695
STATFS
0.00 307.86 us 28.00 us 292666.83 us
16841 READ
0.00 29625.60 us 239.57 us 239532.26 us
178 MKDIR
0.00 319.24 us 51.19 us 136353.39 us
33445 WRITE
0.01 175.83 us 1.57 us 513635.22 us 87338
OPENDIR
0.01 249.96 us 58.42 us 594774.86 us 85089
SETXATTR
0.01 11001.25 us 119.87 us 612765.03 us
3180 RMDIR
0.04 14240.66 us 203.89 us 1376758.99 us 6067
CREATE
0.05 89.45 us 30.14 us 687961.60 us 1219542
READLINK
0.39 1114.30 us 44.29 us 2412169.89 us
831371 OPEN
0.51 127078.45 us 78.55 us 694364.09 us 9493
FXATTROP
0.63 399.16 us 8.43 us 3815689.36 us 3764714
ENTRYLK
0.86 91973.00 us 19.89 us 24258016.33 us 22253
READDIR
1.06 6913.71 us 11.58 us 36968532.72 us 364963
GETXATTR
1.29 62130.12 us 36.89 us 8649871.11 us 49472
READDIRP
2.89 1477.53 us 11.10 us 2807637.64 us 4650500
INODELK
4.12 23681.31 us 20.98 us 23291414.50 us
414352 UNLINK
16.09 5000.31 us 9.04 us 23866016.88 us
7655621 LOOKUP
72.04 68953.31 us 68.40 us 27542499.65 us 2486126
XATTROP
0.00 0.00 us 0.00 us 0.00 us 2433405
UPCALL
0.00 0.00 us 0.00 us 0.00 us 2453
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 69
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 1544
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 2430885
CI_FORGET
Duration: 209524 seconds
Data Read: 2051388524 bytes
Data Written: 2208261499 bytes
Interval 0 Stats:
Block Size: 1b+
2b+ 8b+
No. of Reads: 0
0 0
No. of Writes: 53
1 60
Block Size: 16b+
32b+ 64b+
No. of Reads: 0
213 2
No. of Writes: 14
1057 15
Block Size: 128b+
256b+ 512b+
No. of Reads: 0
41 146
No. of Writes: 59
715 80
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 224
9 15
No. of Writes: 163
282 404
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 52
74 210
No. of Writes: 7415
1430 6860
Block Size: 65536b+ 131072b+
262144b+
No. of Reads: 299
15313 1
No. of Writes: 2458
12378 1
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 686733
FORGET
0.00 0.00 us 0.00 us 0.00 us 837434
RELEASE
0.00 0.00 us 0.00 us 0.00 us 87338
RELEASEDIR
0.00 22.69 us 7.16 us 332.61 us
116 IPC
0.00 119.93 us 70.56 us 272.28 us 57
FTRUNCATE
0.00 152.25 us 86.82 us 340.63 us 65
FGETXATTR
0.00 2336.06 us 90.05 us 20509.81 us 23
REMOVEXATTR
0.00 631.26 us 123.17 us 51803.77 us
121 LINK
0.00 369.27 us 22.64 us 145090.48 us
490 LK
0.00 360.85 us 45.14 us 46846.98 us
540 STAT
0.00 1190.96 us 117.40 us 69781.85 us 200
TRUNCATE
0.00 245.72 us 54.91 us 123985.76 us 1482
SETATTR
0.00 42139.10 us 3313.56 us 312273.21 us
12 FSYNC
0.00 55.83 us 8.26 us 49649.35 us
10047 FLUSH
0.00 57.52 us 14.62 us 820.59 us 9977
FINODELK
0.00 3191.60 us 143.06 us 96214.75 us 188
RENAME
0.00 32862.86 us 215.38 us 83851.13 us 25
SYMLINK
0.00 11392.03 us 230.70 us 105471.90 us
118 MKNOD
0.00 134.63 us 28.08 us 36326.43 us
12587 FSTAT
0.00 112.32 us 19.91 us 381241.34 us 21695
STATFS
0.00 307.91 us 28.00 us 292666.83 us
16841 READ
0.00 29625.60 us 239.57 us 239532.26 us
178 MKDIR
0.00 319.22 us 51.19 us 136353.39 us
33445 WRITE
0.01 175.83 us 1.57 us 513635.22 us 87338
OPENDIR
0.01 249.95 us 58.42 us 594774.86 us 85089
SETXATTR
0.01 11001.25 us 119.87 us 612765.03 us
3180 RMDIR
0.04 14240.66 us 203.89 us 1376758.99 us 6067
CREATE
0.05 89.46 us 30.14 us 687961.60 us 1219542
READLINK
0.39 1114.30 us 44.29 us 2412169.89 us
831371 OPEN
0.51 127078.45 us 78.55 us 694364.09 us 9493
FXATTROP
0.63 399.16 us 8.43 us 3815689.36 us 3764714
ENTRYLK
0.86 91973.00 us 19.89 us 24258016.33 us 22253
READDIR
1.06 6913.26 us 11.58 us 36968532.72 us 364963
GETXATTR
1.29 62121.95 us 36.89 us 8649871.11 us 49472
READDIRP
2.89 1477.58 us 11.10 us 2807637.64 us 4650500
INODELK
4.12 23681.29 us 20.98 us 23291414.50 us
414352 UNLINK
16.09 5000.02 us 9.04 us 23866016.88 us
7655621 LOOKUP
72.04 68956.14 us 68.40 us 27542499.65 us 2486126
XATTROP
0.00 0.00 us 0.00 us 0.00 us 2433405
UPCALL
0.00 0.00 us 0.00 us 0.00 us 2453
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 69
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 1544
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 2430885
CI_FORGET
Duration: 209524 seconds
Data Read: 2051388524 bytes
Data Written: 2208261499 bytes
Brick: flexo:/mnt/gluster/b
--------------------------------------
Cumulative Stats:
Block Size: 1b+
8b+ 16b+
No. of Reads: 0
0 0
No. of Writes: 117
195 367
Block Size: 32b+
64b+ 128b+
No. of Reads: 130
0 3
No. of Writes: 1012
9 134
Block Size: 256b+ 512b+
1024b+
No. of Reads: 47
8 257
No. of Writes: 729
73 168
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 10
27 240
No. of Writes: 294
466 10472
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 139
119 135
No. of Writes: 1480
6273 3935
Block Size: 131072b+
No. of Reads: 7892
No. of Writes: 12327
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 955058
FORGET
0.00 0.00 us 0.00 us 0.00 us 1450749
RELEASE
0.00 0.00 us 0.00 us 0.00 us 81377
RELEASEDIR
0.00 15.55 us 5.35 us 31.88 us
108 IPC
0.00 125.85 us 59.77 us 224.15 us 44
FTRUNCATE
0.00 198.38 us 81.68 us 1332.45 us 30
REMOVEXATTR
0.00 4254.16 us 621.52 us 10602.99 us
4 FSYNC
0.00 795.08 us 190.30 us 16518.64 us 30
SYMLINK
0.00 195.90 us 111.02 us 953.39 us
137 LINK
0.00 286.07 us 179.64 us 437.82 us
119 MKNOD
0.00 162.61 us 65.62 us 3864.77 us 221
SETXATTR
0.00 117.29 us 48.50 us 317.17 us
354 STAT
0.00 70.35 us 23.10 us 317.32 us
777 LK
0.00 461.72 us 114.54 us 16013.81 us 188
RENAME
0.00 150.40 us 75.59 us 749.54 us 810
FGETXATTR
0.00 267.80 us 99.67 us 994.58 us 698
TRUNCATE
0.00 1176.00 us 194.89 us 78261.12 us
178 MKDIR
0.00 439.83 us 54.38 us 75270.51 us 1665
SETATTR
0.00 65.89 us 17.39 us 354.77 us 11372
FINODELK
0.00 131.57 us 32.68 us 2107.52 us
7398 FSTAT
0.00 120.71 us 31.43 us 847.97 us
9094 READ
0.00 60.75 us 7.25 us 22520.98 us
22614 FLUSH
0.00 73.03 us 22.41 us 750.17 us 21656
STATFS
0.00 323.63 us 166.56 us 23034.44 us 6145
CREATE
0.00 217.55 us 50.60 us 30734.99 us
38051 WRITE
0.01 146.39 us 1.80 us 318612.45 us 81377
OPENDIR
0.01 5641.37 us 108.84 us 468960.33 us
3177 RMDIR
0.03 100.05 us 30.93 us 49756.93 us 432439
READLINK
0.13 483.47 us 16.81 us 2608013.14 us 448381
UNLINK
0.21 7101.56 us 22.26 us 2554390.49 us 52185
READDIRP
0.35 414.05 us 36.58 us 1038229.93 us
1444606 OPEN
0.49 78518.02 us 73.39 us 373999.31 us 10745
FXATTROP
0.72 776.21 us 11.24 us 4271792.57 us 1605822
GETXATTR
0.77 97829.38 us 18.84 us 9477446.11 us 13532
READDIR
0.79 320.44 us 7.71 us 2369511.06 us 4257297
ENTRYLK
1.97 464.02 us 9.38 us 1038460.80 us 7333722
INODELK
13.73 2372.50 us 8.33 us 6258159.87 us 9979799
LOOKUP
80.79 42372.04 us 61.49 us 2238308.13 us 3289103
XATTROP
0.00 0.00 us 0.00 us 0.00 us 2250268
UPCALL
0.00 0.00 us 0.00 us 0.00 us 3756
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 167
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 1908
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 2246352
CI_FORGET
Duration: 209334 seconds
Data Read: 1059777395 bytes
Data Written: 2336921014 bytes
Interval 0 Stats:
Block Size: 1b+
8b+ 16b+
No. of Reads: 0
0 0
No. of Writes: 117
195 367
Block Size: 32b+
64b+ 128b+
No. of Reads: 130
0 3
No. of Writes: 1012
9 134
Block Size: 256b+ 512b+
1024b+
No. of Reads: 47
8 257
No. of Writes: 729
73 168
Block Size: 2048b+ 4096b+
8192b+
No. of Reads: 10
27 240
No. of Writes: 294
466 10472
Block Size: 16384b+ 32768b+
65536b+
No. of Reads: 139
119 135
No. of Writes: 1480
6273 3935
Block Size: 131072b+
No. of Reads: 7892
No. of Writes: 12327
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 955058
FORGET
0.00 0.00 us 0.00 us 0.00 us 1450749
RELEASE
0.00 0.00 us 0.00 us 0.00 us 81377
RELEASEDIR
0.00 15.55 us 5.35 us 31.88 us
108 IPC
0.00 125.85 us 59.77 us 224.15 us 44
FTRUNCATE
0.00 198.38 us 81.68 us 1332.45 us 30
REMOVEXATTR
0.00 4254.16 us 621.52 us 10602.99 us
4 FSYNC
0.00 795.08 us 190.30 us 16518.64 us 30
SYMLINK
0.00 195.90 us 111.02 us 953.39 us
137 LINK
0.00 286.07 us 179.64 us 437.82 us
119 MKNOD
0.00 162.61 us 65.62 us 3864.77 us 221
SETXATTR
0.00 117.29 us 48.50 us 317.17 us
354 STAT
0.00 70.35 us 23.10 us 317.32 us
777 LK
0.00 461.72 us 114.54 us 16013.81 us 188
RENAME
0.00 150.43 us 75.59 us 749.54 us 810
FGETXATTR
0.00 267.80 us 99.67 us 994.58 us 698
TRUNCATE
0.00 1176.00 us 194.89 us 78261.12 us
178 MKDIR
0.00 439.83 us 54.38 us 75270.51 us 1665
SETATTR
0.00 65.89 us 17.39 us 354.77 us 11372
FINODELK
0.00 131.57 us 32.68 us 2107.52 us
7398 FSTAT
0.00 120.74 us 31.43 us 847.97 us
9094 READ
0.00 60.75 us 7.25 us 22520.98 us
22614 FLUSH
0.00 73.03 us 22.41 us 750.17 us 21656
STATFS
0.00 323.63 us 166.56 us 23034.44 us 6145
CREATE
0.00 217.50 us 50.60 us 30734.99 us
38051 WRITE
0.01 146.39 us 1.80 us 318612.45 us 81377
OPENDIR
0.01 5641.37 us 108.84 us 468960.33 us
3177 RMDIR
0.03 100.06 us 30.93 us 49756.93 us 432439
READLINK
0.13 483.47 us 16.81 us 2608013.14 us 448381
UNLINK
0.21 7101.56 us 22.26 us 2554390.49 us 52185
READDIRP
0.35 414.01 us 36.58 us 1038229.93 us
1444606 OPEN
0.49 78518.02 us 73.39 us 373999.31 us 10745
FXATTROP
0.72 776.24 us 11.24 us 4271792.57 us 1605822
GETXATTR
0.77 97829.38 us 18.84 us 9477446.11 us 13532
READDIR
0.79 320.45 us 7.71 us 2369511.06 us 4257297
ENTRYLK
1.97 464.07 us 9.38 us 1038460.80 us 7333722
INODELK
13.72 2372.42 us 8.33 us 6258159.87 us 9979799
LOOKUP
80.79 42373.50 us 61.49 us 2238308.13 us 3289103
XATTROP
0.00 0.00 us 0.00 us 0.00 us 2250268
UPCALL
0.00 0.00 us 0.00 us 0.00 us 3756
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 167
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 1908
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 2246352
CI_FORGET
Duration: 209334 seconds
Data Read: 1059777395 bytes
Data Written: 2336921014 bytes
Brick: flexo:/mnt/gluster/c
--------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 32843
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 686562
FORGET
0.00 0.00 us 0.00 us 0.00 us 834596
RELEASE
0.00 0.00 us 0.00 us 0.00 us 93757
RELEASEDIR
0.00 18.50 us 8.21 us 50.20 us
107 IPC
0.00 205.83 us 126.30 us 412.06 us
36 LINK
0.00 167.33 us 75.01 us 272.47 us 65
FGETXATTR
0.00 203.58 us 117.35 us 648.28 us 88
RENAME
0.00 203.20 us 66.60 us 809.96 us 153
TRUNCATE
0.00 67.06 us 23.64 us 555.59 us
491 LK
0.01 51.73 us 7.70 us 1174.64 us
9339 FLUSH
0.01 62.41 us 15.13 us 636.35 us 9011
FINODELK
0.03 126.23 us 43.15 us 4760.14 us
10824 FSTAT
0.03 571.94 us 94.85 us 51697.03 us
2594 MKDIR
0.05 264.87 us 81.58 us 53152.06 us 9009
FXATTROP
0.05 423.05 us 38.26 us 548686.55 us 5754
CREATE
0.05 78.14 us 19.17 us 678.32 us
32843 WRITE
0.06 526.39 us 130.84 us 41405.46 us 5798
SYMLINK
0.10 133.73 us 16.25 us 676494.72 us 34904
READDIR
0.12 2054.48 us 99.28 us 88061.83 us
2732 RMDIR
0.21 110.60 us 1.19 us 16833.78 us 93757
OPENDIR
0.72 205.89 us 33.28 us 261023.99 us 170292
UNLINK
0.85 2971.45 us 21.24 us 257839.19 us 13840
READDIRP
1.94 200.88 us 25.49 us 461654.32 us 468842
FTRUNCATE
2.13 212.44 us 54.02 us 461878.34 us 486882
REMOVEXATTR
2.27 223.01 us 52.03 us 178897.40 us 493240
SETXATTR
3.53 155.31 us 7.96 us 67873.00 us 1101338
GETXATTR
3.63 212.32 us 44.42 us 551264.02 us
829002 OPEN
5.77 76.36 us 7.76 us 962634.73 us 3660425
ENTRYLK
6.23 304.93 us 42.51 us 661394.43 us 990644
SETATTR
7.81 382.20 us 107.42 us 846502.78 us 990766
XATTROP
12.39 549.11 us 135.57 us 1659697.83 us
1093387 MKNOD
12.47 139.78 us 9.76 us 552067.56 us 4325281
INODELK
39.53 206.94 us 8.56 us 964823.33 us 9259429
LOOKUP
0.00 0.00 us 0.00 us 0.00 us 1856377
UPCALL
0.00 0.00 us 0.00 us 0.00 us 933300
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 251
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 293
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 923077
CI_FORGET
Duration: 209334 seconds
Data Read: 0 bytes
Data Written: 32843 bytes
Interval 0 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 32843
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 686562
FORGET
0.00 0.00 us 0.00 us 0.00 us 834596
RELEASE
0.00 0.00 us 0.00 us 0.00 us 93757
RELEASEDIR
0.00 18.50 us 8.21 us 50.20 us
107 IPC
0.00 205.83 us 126.30 us 412.06 us
36 LINK
0.00 167.33 us 75.01 us 272.47 us 65
FGETXATTR
0.00 203.58 us 117.35 us 648.28 us 88
RENAME
0.00 203.20 us 66.60 us 809.96 us 153
TRUNCATE
0.00 67.06 us 23.64 us 555.59 us
491 LK
0.01 51.71 us 7.70 us 1174.64 us
9339 FLUSH
0.01 62.41 us 15.13 us 636.35 us 9011
FINODELK
0.03 126.23 us 43.15 us 4760.14 us
10824 FSTAT
0.03 571.94 us 94.85 us 51697.03 us
2594 MKDIR
0.05 264.87 us 81.58 us 53152.06 us 9009
FXATTROP
0.05 423.05 us 38.26 us 548686.55 us 5754
CREATE
0.05 78.14 us 19.17 us 678.32 us
32843 WRITE
0.06 526.39 us 130.84 us 41405.46 us 5798
SYMLINK
0.10 133.73 us 16.25 us 676494.72 us 34904
READDIR
0.12 2054.48 us 99.28 us 88061.83 us
2732 RMDIR
0.21 110.60 us 1.19 us 16833.78 us 93757
OPENDIR
0.72 205.89 us 33.28 us 261023.99 us 170292
UNLINK
0.85 2971.45 us 21.24 us 257839.19 us 13840
READDIRP
1.94 200.87 us 25.49 us 461654.32 us 468842
FTRUNCATE
2.13 212.44 us 54.02 us 461878.34 us 486882
REMOVEXATTR
2.27 223.00 us 52.03 us 178897.40 us 493240
SETXATTR
3.53 155.32 us 7.96 us 67873.00 us 1101338
GETXATTR
3.63 212.32 us 44.42 us 551264.02 us
829002 OPEN
5.77 76.36 us 7.76 us 962634.73 us 3660425
ENTRYLK
6.23 304.88 us 42.51 us 661394.43 us 990644
SETATTR
7.81 382.20 us 107.42 us 846502.78 us 990766
XATTROP
12.39 549.25 us 135.57 us 1659697.83 us
1093387 MKNOD
12.47 139.78 us 9.76 us 552067.56 us 4325281
INODELK
39.53 206.95 us 8.56 us 964823.33 us 9259429
LOOKUP
0.00 0.00 us 0.00 us 0.00 us 1856377
UPCALL
0.00 0.00 us 0.00 us 0.00 us 933300
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 251
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 293
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 923077
CI_FORGET
Duration: 209334 seconds
Data Read: 0 bytes
Data Written: 32843 bytes
Brick: bender:/mnt/gluster/c
---------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 37050
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 928119
FORGET
0.00 0.00 us 0.00 us 0.00 us 1401524
RELEASE
0.00 0.00 us 0.00 us 0.00 us 70092
RELEASEDIR
0.00 19.49 us 8.28 us 180.61 us
100 IPC
0.00 238.88 us 163.97 us 486.93 us
38 LINK
0.00 306.43 us 196.67 us 788.36 us 30
SYMLINK
0.00 2670.18 us 765.93 us 4151.64 us
4 FSYNC
0.00 350.65 us 141.92 us 10573.64 us 122
RENAME
0.00 67.02 us 20.66 us 453.90 us
777 LK
0.00 158.32 us 64.22 us 532.21 us 810
FGETXATTR
0.00 229.87 us 66.47 us 826.31 us 603
TRUNCATE
0.01 1742.85 us 202.81 us 47471.71 us
302 MKDIR
0.01 59.05 us 15.40 us 1813.56 us 10060
FINODELK
0.01 118.54 us 36.65 us 4002.58 us
6409 FSTAT
0.01 49.38 us 8.08 us 1264.41 us
21719 FLUSH
0.03 74.76 us 20.86 us 2008.41 us
37050 WRITE
0.04 553.12 us 40.81 us 386485.29 us 5901
CREATE
0.06 534.01 us 100.86 us 527770.44 us 10057
FXATTROP
0.12 138.96 us 1.00 us 601208.70 us 70092
OPENDIR
0.12 947.76 us 18.54 us 712568.33 us 10779
READDIR
0.15 4099.07 us 124.69 us 421939.47 us
3008 RMDIR
0.61 465.30 us 20.67 us 936959.86 us 109478
UNLINK
1.57 182.41 us 49.54 us 73938.09 us 722669
REMOVEXATTR
1.61 171.41 us 6.98 us 83809.88 us 789294
GETXATTR
1.86 217.88 us 26.40 us 556446.19 us 718835
FTRUNCATE
1.89 219.47 us 49.92 us 506767.78 us 722856
SETXATTR
3.02 181.66 us 44.40 us 344656.69 us
1395957 OPEN
5.56 322.08 us 43.37 us 634069.61 us 1450160
SETATTR
5.84 115.88 us 7.67 us 747387.51 us 4233520
ENTRYLK
6.53 378.33 us 122.84 us 766939.14 us 1449601
XATTROP
9.75 115.48 us 10.70 us 556675.13 us 7097226
INODELK
19.37 1104.98 us 101.28 us 1612346.75 us
1472813 MKNOD
41.85 295.61 us 8.98 us 996805.54 us 11896091
LOOKUP
0.00 0.00 us 0.00 us 0.00 us 3531139
UPCALL
0.00 0.00 us 0.00 us 0.00 us 1862470
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 244
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 632
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 1668669
CI_FORGET
Duration: 208643 seconds
Data Read: 0 bytes
Data Written: 37050 bytes
Interval 0 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 37050
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 928119
FORGET
0.00 0.00 us 0.00 us 0.00 us 1401524
RELEASE
0.00 0.00 us 0.00 us 0.00 us 70092
RELEASEDIR
0.00 19.49 us 8.28 us 180.61 us
100 IPC
0.00 238.88 us 163.97 us 486.93 us
38 LINK
0.00 306.43 us 196.67 us 788.36 us 30
SYMLINK
0.00 2670.18 us 765.93 us 4151.64 us
4 FSYNC
0.00 350.65 us 141.92 us 10573.64 us 122
RENAME
0.00 67.02 us 20.66 us 453.90 us
777 LK
0.00 158.32 us 64.22 us 532.21 us 810
FGETXATTR
0.00 229.87 us 66.47 us 826.31 us 603
TRUNCATE
0.01 1742.85 us 202.81 us 47471.71 us
302 MKDIR
0.01 59.05 us 15.40 us 1813.56 us 10060
FINODELK
0.01 118.54 us 36.65 us 4002.58 us
6409 FSTAT
0.01 49.39 us 8.08 us 1264.41 us
21719 FLUSH
0.03 74.77 us 20.86 us 2008.41 us
37050 WRITE
0.04 553.12 us 40.81 us 386485.29 us 5901
CREATE
0.06 534.01 us 100.86 us 527770.44 us 10057
FXATTROP
0.12 138.96 us 1.00 us 601208.70 us 70092
OPENDIR
0.12 947.76 us 18.54 us 712568.33 us 10779
READDIR
0.15 4099.07 us 124.69 us 421939.47 us
3008 RMDIR
0.61 465.30 us 20.67 us 936959.86 us 109478
UNLINK
1.57 182.41 us 49.54 us 73938.09 us 722669
REMOVEXATTR
1.61 171.41 us 6.98 us 83809.88 us 789294
GETXATTR
1.86 217.88 us 26.40 us 556446.19 us 718835
FTRUNCATE
1.89 219.48 us 49.92 us 506767.78 us 722856
SETXATTR
3.02 181.66 us 44.40 us 344656.69 us
1395957 OPEN
5.56 322.09 us 43.37 us 634069.61 us 1450160
SETATTR
5.84 115.88 us 7.67 us 747387.51 us 4233520
ENTRYLK
6.53 378.33 us 122.84 us 766939.14 us 1449601
XATTROP
9.75 115.48 us 10.70 us 556675.13 us 7097226
INODELK
19.37 1105.06 us 101.28 us 1612346.75 us
1472813 MKNOD
41.84 295.54 us 8.98 us 996805.54 us 11896091
LOOKUP
0.00 0.00 us 0.00 us 0.00 us 3531139
UPCALL
0.00 0.00 us 0.00 us 0.00 us 1862470
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 244
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 632
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 1668669
CI_FORGET
Duration: 208643 seconds
Data Read: 0 bytes
Data Written: 37050 bytes
Brick: bender:/mnt/gluster/a
---------------------------------------
Cumulative Stats:
Block Size: 1b+
2b+ 8b+
No. of Reads: 0
0 0
No. of Writes: 53
1 60
Block Size: 16b+
32b+ 64b+
No. of Reads: 0
235 4
No. of Writes: 14
1033 15
Block Size: 128b+
256b+ 512b+
No. of Reads: 4
115 81
No. of Writes: 58
703 80
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 181
26 74
No. of Writes: 163
278 397
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 131
161 267
No. of Writes: 7264
1398 6742
Block Size: 65536b+ 131072b+
262144b+
No. of Reads: 443
25528 0
No. of Writes: 2426
12240 1
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 684942
FORGET
0.00 0.00 us 0.00 us 0.00 us 837092
RELEASE
0.00 0.00 us 0.00 us 0.00 us 87297
RELEASEDIR
0.00 219.22 us 121.08 us 306.12 us 8
REMOVEXATTR
0.00 21.70 us 7.24 us 196.07 us
97 IPC
0.00 152.77 us 78.90 us 1059.82 us 46
FTRUNCATE
0.00 295.73 us 167.10 us 606.96 us 65
FGETXATTR
0.00 243.66 us 140.50 us 3392.16 us
121 LINK
0.00 1216.47 us 194.37 us 23411.42 us 25
SYMLINK
0.00 327.87 us 234.27 us 551.71 us
107 MKNOD
0.00 36094.61 us 36094.61 us 36094.61 us
1 FSYNC
0.00 219.63 us 147.34 us 783.10 us 188
RENAME
0.00 104.27 us 40.34 us 595.29 us
745 STAT
0.00 531.71 us 113.57 us 53973.08 us 200
TRUNCATE
0.00 289.05 us 63.10 us 70107.41 us 1444
SETATTR
0.00 2430.15 us 232.78 us 68120.54 us
177 MKDIR
0.00 52.68 us 7.28 us 826.01 us
9890 FLUSH
0.00 60.96 us 17.54 us 1393.95 us 9792
FINODELK
0.00 74.26 us 20.00 us 1522.39 us 21521
STATFS
0.00 129.21 us 34.15 us 9068.67 us
13094 FSTAT
0.00 456.99 us 195.50 us 130182.02 us 5967
CREATE
0.00 112.06 us 28.14 us 9545.53 us
27474 READ
0.01 313.76 us 52.98 us 71635.74 us
32926 WRITE
0.01 135.84 us 1.52 us 671373.08 us 87297
OPENDIR
0.01 5225.28 us 131.28 us 951323.69 us
3177 RMDIR
0.01 203.34 us 56.13 us 486756.90 us 85073
SETXATTR
0.03 3521.00 us 145.28 us 3396306.00 us 15090
READDIRP
0.04 113758.50 us 26.72 us 1147074.44 us
564 LK
0.05 92.18 us 29.71 us 39069.99 us 850938
READLINK
0.23 453.79 us 41.84 us 655999.23 us
831212 OPEN
0.41 29425.34 us 19.20 us 1526187.08 us 22482
READDIR
0.50 1954.70 us 21.51 us 1526772.28 us 415226
UNLINK
0.54 1272.83 us 9.66 us 1604368.93 us 686945
GETXATTR
0.55 237.57 us 7.91 us 1277047.91 us 3739718
ENTRYLK
0.60 103888.49 us 89.32 us 761509.35 us 9311
FXATTROP
2.60 909.04 us 9.68 us 2272778.49 us 4649291
INODELK
5.82 1240.20 us 8.78 us 1526868.50 us 7625383
LOOKUP
88.60 58005.98 us 67.31 us 1448810.06 us 2481914
XATTROP
0.00 0.00 us 0.00 us 0.00 us 1396875
UPCALL
0.00 0.00 us 0.00 us 0.00 us 1854
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 28
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 1051
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 1394996
CI_FORGET
Duration: 208643 seconds
Data Read: 3409342848 bytes
Data Written: 2180686507 bytes
Interval 0 Stats:
Block Size: 1b+
2b+ 8b+
No. of Reads: 0
0 0
No. of Writes: 53
1 60
Block Size: 16b+
32b+ 64b+
No. of Reads: 0
235 4
No. of Writes: 14
1033 15
Block Size: 128b+
256b+ 512b+
No. of Reads: 4
115 81
No. of Writes: 58
703 80
Block Size: 1024b+ 2048b+
4096b+
No. of Reads: 181
26 74
No. of Writes: 163
278 397
Block Size: 8192b+ 16384b+
32768b+
No. of Reads: 131
161 267
No. of Writes: 7264
1398 6742
Block Size: 65536b+ 131072b+
262144b+
No. of Reads: 443
25528 0
No. of Writes: 2426
12240 1
%-latency Avg-latency Min-Latency Max-Latency No. of
calls Fop
--------- ----------- ----------- -----------
------------ ----
0.00 0.00 us 0.00 us 0.00 us 684942
FORGET
0.00 0.00 us 0.00 us 0.00 us 837092
RELEASE
0.00 0.00 us 0.00 us 0.00 us 87297
RELEASEDIR
0.00 219.22 us 121.08 us 306.12 us 8
REMOVEXATTR
0.00 21.70 us 7.24 us 196.07 us
97 IPC
0.00 152.77 us 78.90 us 1059.82 us 46
FTRUNCATE
0.00 295.73 us 167.10 us 606.96 us 65
FGETXATTR
0.00 243.66 us 140.50 us 3392.16 us
121 LINK
0.00 1216.47 us 194.37 us 23411.42 us 25
SYMLINK
0.00 327.87 us 234.27 us 551.71 us
107 MKNOD
0.00 36094.61 us 36094.61 us 36094.61 us
1 FSYNC
0.00 219.63 us 147.34 us 783.10 us 188
RENAME
0.00 104.27 us 40.34 us 595.29 us
745 STAT
0.00 531.71 us 113.57 us 53973.08 us 200
TRUNCATE
0.00 289.05 us 63.10 us 70107.41 us 1444
SETATTR
0.00 2430.15 us 232.78 us 68120.54 us
177 MKDIR
0.00 52.69 us 7.28 us 826.01 us
9890 FLUSH
0.00 60.96 us 17.54 us 1393.95 us 9792
FINODELK
0.00 74.25 us 20.00 us 1522.39 us 21521
STATFS
0.00 129.23 us 34.15 us 9068.67 us
13094 FSTAT
0.00 456.99 us 195.50 us 130182.02 us 5967
CREATE
0.00 112.45 us 28.14 us 9545.53 us
27474 READ
0.01 313.77 us 52.98 us 71635.74 us
32926 WRITE
0.01 135.84 us 1.52 us 671373.08 us 87297
OPENDIR
0.01 5225.28 us 131.28 us 951323.69 us
3177 RMDIR
0.01 203.34 us 56.13 us 486756.90 us 85073
SETXATTR
0.03 3521.00 us 145.28 us 3396306.00 us 15090
READDIRP
0.04 113758.50 us 26.72 us 1147074.44 us
564 LK
0.05 92.18 us 29.71 us 39069.99 us 850938
READLINK
0.23 453.79 us 41.84 us 655999.23 us
831212 OPEN
0.41 29425.34 us 19.20 us 1526187.08 us 22482
READDIR
0.50 1954.70 us 21.51 us 1526772.28 us 415226
UNLINK
0.54 1272.79 us 9.66 us 1604368.93 us 686945
GETXATTR
0.55 237.57 us 7.91 us 1277047.91 us 3739718
ENTRYLK
0.60 103888.49 us 89.32 us 761509.35 us 9311
FXATTROP
2.60 909.04 us 9.68 us 2272778.49 us 4649291
INODELK
5.82 1240.10 us 8.78 us 1526868.50 us 7625383
LOOKUP
88.60 58006.02 us 67.31 us 1448810.06 us 2481914
XATTROP
0.00 0.00 us 0.00 us 0.00 us 1396875
UPCALL
0.00 0.00 us 0.00 us 0.00 us 1854
CI_IATT
0.00 0.00 us 0.00 us 0.00 us 28
CI_XATTR
0.00 0.00 us 0.00 us 0.00 us 1051
CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 1394996
CI_FORGET
Duration: 208643 seconds
Data Read: 3409342848 bytes
Data Written: 2180686507 bytes
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200502/5962bb36/attachment.html>
More information about the Gluster-users
mailing list