[Gluster-users] Possible SHD stalling
Artem Russakovskii
archon810 at gmail.com
Mon May 4 19:56:10 UTC 2020
I reported something similar recently to this list, and I highly advise you
to update to 7.5 (in my case it was 5.11 -> 5.13), then see if it keeps
happening. The self heal issues should be fixed there properly at least.
Sincerely,
Artem
--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>
On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za> wrote:
> Hi All,
>
> So we've got a cluster currently running on glusterfs 7.3
>
> Heal performance is absolutely terrible (it starts of great, runs for a
> while then completely trips up).
>
> Once the SHD trips up, access speeds to the cluster also becomes poor.
> And with poor I mean directory listings that normally takes well under
> 500ms takes upwards of 10s, but I've seen upwards of 40s (although a follow
> up ls immediately after is nice and fast again, temporarily). With all
> SHDs killed this particular ls is as per normal (around 30-50ms).
>
> The SHD logs doesn't provide any usable clues, the only variance in logs
> is this, just before all daemons across the cluster crawls to a near halt.
> [2020-05-02 19:55:58.627036] I [MSGID: 108026]
> [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0:
> Completed entry selfheal on 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources=
> sinks=0 1
> [2020-05-02 19:55:58.627058] I [MSGID: 108026]
> [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0:
> Completed entry selfheal on c736c9b6-f116-468b-adbc-f102f5b57d01. sources=
> sinks=0 1
>
> [2020-05-02 19:55:59.125001] W [MSGID: 114031]
> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk] 0-vpbx_shared-client-1:
> remote operation failed [Invalid argument]
>
> [2020-05-02 19:55:59.339803] I [MSGID: 108026]
> [afr-self-heal-common.c:1747:afr_log_selfheal] 0-vpbx_shared-replicate-0:
> Completed entry selfheal on 2e128eeb-e991-4544-8316-4ddfcb4942da. sources=
> sinks=0 1
> [2020-05-02 19:55:59.490387] I [MSGID: 108026]
> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
> 0-vpbx_shared-replicate-0: performing metadata selfheal on
> 527a63a3-9acd-4c3f-b95c-5c719325c973
>
> A few more entries will complete overall and then it'll be a file every
> other second. Once I saw this I ran a few more tests, and on every test
> just prior to slowing to a crawl, that entry is there.
>
> There is a perceived improvement when lowering the thread count, but not
> significant enough to confirm a correlation. Even with cluster.shd-max-threads=1
> this still happens (gets a few tens thousand files completed instead of few
> hundred, at 32 it gets to maybe 400 to 500 before it stalls, 16 does maybe
> a thousand).
>
> There are probably performant bursts hereafter, because from experience we
> know this process takes about two months. But this time around it seems to
> be much, much worse. Might be because that was healing a single brick to
> another brick, so less locking going? At the moment the replicas are
> basically trying to populate the newly added arbiters. Also, we haven't
> messed as greatly with the health check settings (queues and threads and
> locks nearly as much as since the arbiter addition because performance has
> simply dropped to non-acceptable levels.
>
> Unfortunately granular entry heal isn't on at the moment (nor can I switch
> it on due to a few million outstanding heals). This used to be a 2x2
> cluster. We then moved one of the bricks to a newly added server using
> replace-brick. What a frightening process this is. We then proceeded to
> add the arbiters.
>
> Some of the settings below was absolute shots in the dark, based on the
> little information we've been able to track down so far.
>
> Kind Regards,
> Jaco
>
> The current gluster volume info (this used to be 2x2
> distributed-replicate, aiming to get it to 3x (2 + 1) ... at current rates
> we might get there by 2021):
>
> Volume Name: glvol
> Type: Distributed-Replicate
> Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x (2 + 1) = 6
> Transport-type: tcp
> Bricks:
> Brick1: zoidberg:/mnt/gluster/a
> Brick2: flexo:/mnt/gluster/b
> Brick3: bender:/mnt/gluster/c (arbiter)
> Brick4: bender:/mnt/gluster/a
> Brick5: zoidberg:/mnt/gluster/b
> Brick6: flexo:/mnt/gluster/c (arbiter)
> Options Reconfigured:
> *cluster.heal-timeout: 5 <-- hoping this would at least shorten the
> blockages.*
> cluster.locking-scheme: granular
> cluster.heal-wait-queue-length: 10240
> *diagnostics.count-fop-hits: on*
> *diagnostics.latency-measurement: on <-- was hoping profiling might
> provide some insight. Going to switch this off now.*
> cluster.background-self-heal-count: 128
> *cluster.shd-wait-qlength: 131072*
> *cluster.shd-max-threads: 32 <-- even with this disk IO never goes over
> 10% utilization. CPU goes to about 2 cores. So it's lock contention
> that's slowing it down we reckon.*
> *cluster.metadata-self-heal: off*
> *cluster.entry-self-heal: off*
> *cluster.data-self-heal: off*
> * <-- these kill performance. *transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: on
> features.cache-invalidation: on
> features.cache-invalidation-timeout: 600
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> performance.md-cache-timeout: 600
> *cluster.self-heal-daemon: yes <-- disabling/enabling this doesn't help,
> have to kill the shd and start ... force to get things moving along again.*
> performance.io-thread-count: 16
> performance.high-prio-threads: 16
> performance.normal-prio-threads: 16
> performance.low-prio-threads: 16
> performance.least-prio-threads: 4
> cluster.self-heal-window-size: 512
> cluster.self-heal-readdir-size: 131072
> client.event-threads: 8
> server.event-threads: 16
> network.ping-timeout: 2
> *cluster.data-self-heal-algorithm: full <-- most files are small (<1MB),
> on dual 1gbit links (so up to 2gbit, but b/w never goes over 5mbit anyway).*
>
> performance info:
>
> Brick: zoidberg:/mnt/gluster/a
> -----------------------------------------
> Cumulative Stats:
> Block Size: 1b+ 8b+
> 16b+
> No. of Reads: 0
> 0 0
> No. of Writes: 117
> 195 367
>
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 306
> 0 6
> No. of Writes: 1016
> 9 134
>
> Block Size: 256b+ 512b+
> 1024b+
> No. of Reads: 107
> 58 485
> No. of Writes: 731
> 73 168
>
> Block Size: 2048b+ 4096b+
> 8192b+
> No. of Reads: 43
> 59 383
> No. of Writes: 295 475
> 10562
>
> Block Size: 16384b+ 32768b+
> 65536b+
> No. of Reads: 347
> 380 634
> No. of Writes: 1487 6369
> 3970
>
> Block Size: 131072b+
> No. of Reads: 35704
> No. of Writes: 12398
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 957293
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1455091
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 69951
> RELEASEDIR
> 0.00 16.20 us 5.43 us 100.95 us
> 113 IPC
> 0.00 687.33 us 80.53 us 15414.03 us 30
> REMOVEXATTR
> 0.00 840.76 us 72.71 us 37648.39 us 52
> FTRUNCATE
> 0.00 138.49 us 36.66 us 25280.24 us 1385
> STAT
> 0.00 364.93 us 63.45 us 15957.66 us 810
> FGETXATTR
> 0.00 2208.82 us 127.23 us 267092.17 us 137
> LINK
> 0.00 37595.33 us 1677.46 us 218313.32 us 12
> FSYNC
> 0.00 675.61 us 106.82 us 114351.01 us 698
> TRUNCATE
> 0.00 3995.66 us 147.00 us 109544.46 us 188
> RENAME
> 0.00 453.11 us 53.37 us 104848.94 us 1680
> SETATTR
> 0.00 6004.32 us 229.33 us 151322.58 us 127
> MKNOD
> 0.00 68.74 us 16.11 us 61050.93 us 11412
> FINODELK
> 0.00 3793.44 us 68.61 us 411523.70 us 221
> SETXATTR
> 0.00 28669.99 us 227.28 us 138988.41 us 30
> SYMLINK
> 0.00 56.18 us 7.31 us 41484.37 us 22640
> FLUSH
> 0.00 105.64 us 19.34 us 351055.72 us 21693
> STATFS
> 0.00 23642.90 us 244.19 us 234912.25 us 178
> MKDIR
> 0.00 537.38 us 31.73 us 296911.67 us 9653
> FSTAT
> 0.00 214.87 us 25.71 us 1712020.99 us 39054
> READ
> 0.01 147.08 us 1.47 us 241611.80 us 69951
> OPENDIR
> 0.01 3899.84 us 117.22 us 503422.21 us 3180
> RMDIR
> 0.01 327.08 us 55.69 us 420376.80 us 38366
> WRITE
> 0.02 5580.36 us 210.11 us 1029702.30 us 6169
> CREATE
> 0.05 92.69 us 29.94 us 687955.86 us 1159936
> READLINK
> 0.40 543.50 us 36.45 us 2430653.54 us 1449307
> OPEN
> 0.48 87643.67 us 70.20 us 661999.51 us 10787
> FXATTROP
> 0.53 2342.72 us 20.91 us 4157542.43 us 447821
> UNLINK
> 0.53 558734.79 us 24.40 us 1399487.63 us
> 1889 LK
> 0.78 145786.31 us 19.06 us 37490825.73 us 10687
> READDIR
> 0.99 2426.23 us 12.47 us 24176123.13 us 809430
> GETXATTR
> 1.45 676.83 us 9.30 us 193177067.25 us 4260021
> ENTRYLK
> 1.76 38325.38 us 21.44 us 46693575.93 us 91439
> READDIRP
> 4.44 1200.31 us 9.10 us 2559377.90 us 7352811
> INODELK
> 9.30 1849.24 us 9.28 us 35299050.00 us 9989221
> LOOKUP
> 79.24 47791.46 us 76.77 us 10488824.26 us 3294642
> XATTROP
> 0.00 0.00 us 0.00 us 0.00 us 3194806
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 5311
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 54
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 1907
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 3189457
> CI_FORGET
>
> Duration: 209524 seconds
> Data Read: 4775365531 bytes
> Data Written: 2354108664 bytes
>
> Interval 0 Stats:
> Block Size: 1b+ 8b+
> 16b+
> No. of Reads: 0
> 0 0
> No. of Writes: 117
> 195 367
>
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 306
> 0 6
> No. of Writes: 1016
> 9 134
>
> Block Size: 256b+ 512b+
> 1024b+
> No. of Reads: 107
> 58 485
> No. of Writes: 731
> 73 168
>
> Block Size: 2048b+ 4096b+
> 8192b+
> No. of Reads: 43
> 59 383
> No. of Writes: 295 475
> 10562
>
> Block Size: 16384b+ 32768b+
> 65536b+
> No. of Reads: 347
> 380 634
> No. of Writes: 1487 6369
> 3970
>
> Block Size: 131072b+
> No. of Reads: 35704
> No. of Writes: 12398
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 957293
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1455091
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 69951
> RELEASEDIR
> 0.00 16.20 us 5.43 us 100.95 us
> 113 IPC
> 0.00 687.33 us 80.53 us 15414.03 us 30
> REMOVEXATTR
> 0.00 840.76 us 72.71 us 37648.39 us 52
> FTRUNCATE
> 0.00 138.49 us 36.66 us 25280.24 us 1385
> STAT
> 0.00 363.20 us 63.45 us 15957.66 us 810
> FGETXATTR
> 0.00 2208.82 us 127.23 us 267092.17 us 137
> LINK
> 0.00 37595.33 us 1677.46 us 218313.32 us 12
> FSYNC
> 0.00 675.61 us 106.82 us 114351.01 us 698
> TRUNCATE
> 0.00 3995.66 us 147.00 us 109544.46 us 188
> RENAME
> 0.00 453.11 us 53.37 us 104848.94 us 1680
> SETATTR
> 0.00 6004.32 us 229.33 us 151322.58 us 127
> MKNOD
> 0.00 68.74 us 16.11 us 61050.93 us 11412
> FINODELK
> 0.00 3793.44 us 68.61 us 411523.70 us 221
> SETXATTR
> 0.00 28669.99 us 227.28 us 138988.41 us 30
> SYMLINK
> 0.00 56.19 us 7.31 us 41484.37 us 22640
> FLUSH
> 0.00 105.61 us 19.34 us 351055.72 us 21693
> STATFS
> 0.00 23642.90 us 244.19 us 234912.25 us 178
> MKDIR
> 0.00 537.38 us 31.73 us 296911.67 us 9653
> FSTAT
> 0.00 214.91 us 25.71 us 1712020.99 us 39054
> READ
> 0.01 147.08 us 1.47 us 241611.80 us 69951
> OPENDIR
> 0.01 3899.84 us 117.22 us 503422.21 us 3180
> RMDIR
> 0.01 327.13 us 55.69 us 420376.80 us 38366
> WRITE
> 0.02 5580.36 us 210.11 us 1029702.30 us 6169
> CREATE
> 0.05 92.69 us 29.94 us 687955.86 us 1159936
> READLINK
> 0.40 543.49 us 36.45 us 2430653.54 us 1449307
> OPEN
> 0.48 87643.67 us 70.20 us 661999.51 us 10787
> FXATTROP
> 0.53 2342.57 us 20.91 us 4157542.43 us 447821
> UNLINK
> 0.53 558416.10 us 24.40 us 1399487.63 us
> 1889 LK
> 0.78 145786.31 us 19.06 us 37490825.73 us 10687
> READDIR
> 0.99 2426.24 us 12.47 us 24176123.13 us 809430
> GETXATTR
> 1.45 676.75 us 9.30 us 193177067.25 us 4260021
> ENTRYLK
> 1.76 38324.59 us 21.44 us 46693575.93 us 91439
> READDIRP
> 4.44 1200.37 us 9.10 us 2559377.90 us 7352811
> INODELK
> 9.29 1848.94 us 9.28 us 35299050.00 us 9989221
> LOOKUP
> 79.24 47792.42 us 76.77 us 10488824.26 us 3294642
> XATTROP
> 0.00 0.00 us 0.00 us 0.00 us 3194806
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 5311
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 54
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 1907
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 3189457
> CI_FORGET
>
> Duration: 209524 seconds
> Data Read: 4775365531 bytes
> Data Written: 2354108664 bytes
>
> Brick: zoidberg:/mnt/gluster/b
> -----------------------------------------
> Cumulative Stats:
> Block Size: 1b+
> 2b+ 8b+
> No. of Reads: 0
> 0 0
> No. of Writes: 53
> 1 60
>
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0
> 213 2
> No. of Writes: 14
> 1057 15
>
> Block Size: 128b+ 256b+
> 512b+
> No. of Reads: 0
> 41 146
> No. of Writes: 59
> 715 80
>
> Block Size: 1024b+ 2048b+
> 4096b+
> No. of Reads: 224
> 9 15
> No. of Writes: 163
> 282 404
>
> Block Size: 8192b+ 16384b+
> 32768b+
> No. of Reads: 52
> 74 210
> No. of Writes: 7415 1430
> 6860
>
> Block Size: 65536b+ 131072b+
> 262144b+
> No. of Reads: 299
> 15313 1
> No. of Writes: 2458
> 12378 1
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 686733
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 837434
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 87338
> RELEASEDIR
> 0.00 22.69 us 7.16 us 332.61 us
> 116 IPC
> 0.00 119.93 us 70.56 us 272.28 us 57
> FTRUNCATE
> 0.00 152.25 us 86.82 us 340.63 us 65
> FGETXATTR
> 0.00 2336.06 us 90.05 us 20509.81 us 23
> REMOVEXATTR
> 0.00 631.26 us 123.17 us 51803.77 us 121
> LINK
> 0.00 369.27 us 22.64 us 145090.48 us
> 490 LK
> 0.00 360.85 us 45.14 us 46846.98 us 540
> STAT
> 0.00 1190.96 us 117.40 us 69781.85 us 200
> TRUNCATE
> 0.00 245.72 us 54.91 us 123985.76 us 1482
> SETATTR
> 0.00 42139.10 us 3313.56 us 312273.21 us 12
> FSYNC
> 0.00 55.83 us 8.26 us 49649.35 us 10047
> FLUSH
> 0.00 57.52 us 14.62 us 820.59 us 9977
> FINODELK
> 0.00 3191.60 us 143.06 us 96214.75 us 188
> RENAME
> 0.00 32862.86 us 215.38 us 83851.13 us 25
> SYMLINK
> 0.00 11392.03 us 230.70 us 105471.90 us 118
> MKNOD
> 0.00 134.63 us 28.08 us 36326.43 us 12587
> FSTAT
> 0.00 112.32 us 19.91 us 381241.34 us 21695
> STATFS
> 0.00 307.86 us 28.00 us 292666.83 us 16841
> READ
> 0.00 29625.60 us 239.57 us 239532.26 us 178
> MKDIR
> 0.00 319.24 us 51.19 us 136353.39 us 33445
> WRITE
> 0.01 175.83 us 1.57 us 513635.22 us 87338
> OPENDIR
> 0.01 249.96 us 58.42 us 594774.86 us 85089
> SETXATTR
> 0.01 11001.25 us 119.87 us 612765.03 us 3180
> RMDIR
> 0.04 14240.66 us 203.89 us 1376758.99 us 6067
> CREATE
> 0.05 89.45 us 30.14 us 687961.60 us 1219542
> READLINK
> 0.39 1114.30 us 44.29 us 2412169.89 us 831371
> OPEN
> 0.51 127078.45 us 78.55 us 694364.09 us 9493
> FXATTROP
> 0.63 399.16 us 8.43 us 3815689.36 us 3764714
> ENTRYLK
> 0.86 91973.00 us 19.89 us 24258016.33 us 22253
> READDIR
> 1.06 6913.71 us 11.58 us 36968532.72 us 364963
> GETXATTR
> 1.29 62130.12 us 36.89 us 8649871.11 us 49472
> READDIRP
> 2.89 1477.53 us 11.10 us 2807637.64 us 4650500
> INODELK
> 4.12 23681.31 us 20.98 us 23291414.50 us 414352
> UNLINK
> 16.09 5000.31 us 9.04 us 23866016.88 us 7655621
> LOOKUP
> 72.04 68953.31 us 68.40 us 27542499.65 us 2486126
> XATTROP
> 0.00 0.00 us 0.00 us 0.00 us 2433405
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 2453
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 69
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 1544
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 2430885
> CI_FORGET
>
> Duration: 209524 seconds
> Data Read: 2051388524 bytes
> Data Written: 2208261499 bytes
>
> Interval 0 Stats:
> Block Size: 1b+
> 2b+ 8b+
> No. of Reads: 0
> 0 0
> No. of Writes: 53
> 1 60
>
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0
> 213 2
> No. of Writes: 14
> 1057 15
>
> Block Size: 128b+ 256b+
> 512b+
> No. of Reads: 0
> 41 146
> No. of Writes: 59
> 715 80
>
> Block Size: 1024b+ 2048b+
> 4096b+
> No. of Reads: 224
> 9 15
> No. of Writes: 163
> 282 404
>
> Block Size: 8192b+ 16384b+
> 32768b+
> No. of Reads: 52
> 74 210
> No. of Writes: 7415 1430
> 6860
>
> Block Size: 65536b+ 131072b+
> 262144b+
> No. of Reads: 299
> 15313 1
> No. of Writes: 2458
> 12378 1
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 686733
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 837434
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 87338
> RELEASEDIR
> 0.00 22.69 us 7.16 us 332.61 us
> 116 IPC
> 0.00 119.93 us 70.56 us 272.28 us 57
> FTRUNCATE
> 0.00 152.25 us 86.82 us 340.63 us 65
> FGETXATTR
> 0.00 2336.06 us 90.05 us 20509.81 us 23
> REMOVEXATTR
> 0.00 631.26 us 123.17 us 51803.77 us 121
> LINK
> 0.00 369.27 us 22.64 us 145090.48 us
> 490 LK
> 0.00 360.85 us 45.14 us 46846.98 us 540
> STAT
> 0.00 1190.96 us 117.40 us 69781.85 us 200
> TRUNCATE
> 0.00 245.72 us 54.91 us 123985.76 us 1482
> SETATTR
> 0.00 42139.10 us 3313.56 us 312273.21 us 12
> FSYNC
> 0.00 55.83 us 8.26 us 49649.35 us 10047
> FLUSH
> 0.00 57.52 us 14.62 us 820.59 us 9977
> FINODELK
> 0.00 3191.60 us 143.06 us 96214.75 us 188
> RENAME
> 0.00 32862.86 us 215.38 us 83851.13 us 25
> SYMLINK
> 0.00 11392.03 us 230.70 us 105471.90 us 118
> MKNOD
> 0.00 134.63 us 28.08 us 36326.43 us 12587
> FSTAT
> 0.00 112.32 us 19.91 us 381241.34 us 21695
> STATFS
> 0.00 307.91 us 28.00 us 292666.83 us 16841
> READ
> 0.00 29625.60 us 239.57 us 239532.26 us 178
> MKDIR
> 0.00 319.22 us 51.19 us 136353.39 us 33445
> WRITE
> 0.01 175.83 us 1.57 us 513635.22 us 87338
> OPENDIR
> 0.01 249.95 us 58.42 us 594774.86 us 85089
> SETXATTR
> 0.01 11001.25 us 119.87 us 612765.03 us 3180
> RMDIR
> 0.04 14240.66 us 203.89 us 1376758.99 us 6067
> CREATE
> 0.05 89.46 us 30.14 us 687961.60 us 1219542
> READLINK
> 0.39 1114.30 us 44.29 us 2412169.89 us 831371
> OPEN
> 0.51 127078.45 us 78.55 us 694364.09 us 9493
> FXATTROP
> 0.63 399.16 us 8.43 us 3815689.36 us 3764714
> ENTRYLK
> 0.86 91973.00 us 19.89 us 24258016.33 us 22253
> READDIR
> 1.06 6913.26 us 11.58 us 36968532.72 us 364963
> GETXATTR
> 1.29 62121.95 us 36.89 us 8649871.11 us 49472
> READDIRP
> 2.89 1477.58 us 11.10 us 2807637.64 us 4650500
> INODELK
> 4.12 23681.29 us 20.98 us 23291414.50 us 414352
> UNLINK
> 16.09 5000.02 us 9.04 us 23866016.88 us 7655621
> LOOKUP
> 72.04 68956.14 us 68.40 us 27542499.65 us 2486126
> XATTROP
> 0.00 0.00 us 0.00 us 0.00 us 2433405
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 2453
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 69
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 1544
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 2430885
> CI_FORGET
>
> Duration: 209524 seconds
> Data Read: 2051388524 bytes
> Data Written: 2208261499 bytes
>
> Brick: flexo:/mnt/gluster/b
> --------------------------------------
> Cumulative Stats:
> Block Size: 1b+ 8b+
> 16b+
> No. of Reads: 0
> 0 0
> No. of Writes: 117
> 195 367
>
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 130
> 0 3
> No. of Writes: 1012
> 9 134
>
> Block Size: 256b+ 512b+
> 1024b+
> No. of Reads: 47
> 8 257
> No. of Writes: 729
> 73 168
>
> Block Size: 2048b+ 4096b+
> 8192b+
> No. of Reads: 10
> 27 240
> No. of Writes: 294 466
> 10472
>
> Block Size: 16384b+ 32768b+
> 65536b+
> No. of Reads: 139
> 119 135
> No. of Writes: 1480 6273
> 3935
>
> Block Size: 131072b+
> No. of Reads: 7892
> No. of Writes: 12327
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 955058
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1450749
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 81377
> RELEASEDIR
> 0.00 15.55 us 5.35 us 31.88 us
> 108 IPC
> 0.00 125.85 us 59.77 us 224.15 us 44
> FTRUNCATE
> 0.00 198.38 us 81.68 us 1332.45 us 30
> REMOVEXATTR
> 0.00 4254.16 us 621.52 us 10602.99 us 4
> FSYNC
> 0.00 795.08 us 190.30 us 16518.64 us 30
> SYMLINK
> 0.00 195.90 us 111.02 us 953.39 us 137
> LINK
> 0.00 286.07 us 179.64 us 437.82 us 119
> MKNOD
> 0.00 162.61 us 65.62 us 3864.77 us 221
> SETXATTR
> 0.00 117.29 us 48.50 us 317.17 us 354
> STAT
> 0.00 70.35 us 23.10 us 317.32 us
> 777 LK
> 0.00 461.72 us 114.54 us 16013.81 us 188
> RENAME
> 0.00 150.40 us 75.59 us 749.54 us 810
> FGETXATTR
> 0.00 267.80 us 99.67 us 994.58 us 698
> TRUNCATE
> 0.00 1176.00 us 194.89 us 78261.12 us 178
> MKDIR
> 0.00 439.83 us 54.38 us 75270.51 us 1665
> SETATTR
> 0.00 65.89 us 17.39 us 354.77 us 11372
> FINODELK
> 0.00 131.57 us 32.68 us 2107.52 us 7398
> FSTAT
> 0.00 120.71 us 31.43 us 847.97 us 9094
> READ
> 0.00 60.75 us 7.25 us 22520.98 us 22614
> FLUSH
> 0.00 73.03 us 22.41 us 750.17 us 21656
> STATFS
> 0.00 323.63 us 166.56 us 23034.44 us 6145
> CREATE
> 0.00 217.55 us 50.60 us 30734.99 us 38051
> WRITE
> 0.01 146.39 us 1.80 us 318612.45 us 81377
> OPENDIR
> 0.01 5641.37 us 108.84 us 468960.33 us 3177
> RMDIR
> 0.03 100.05 us 30.93 us 49756.93 us 432439
> READLINK
> 0.13 483.47 us 16.81 us 2608013.14 us 448381
> UNLINK
> 0.21 7101.56 us 22.26 us 2554390.49 us 52185
> READDIRP
> 0.35 414.05 us 36.58 us 1038229.93 us 1444606
> OPEN
> 0.49 78518.02 us 73.39 us 373999.31 us 10745
> FXATTROP
> 0.72 776.21 us 11.24 us 4271792.57 us 1605822
> GETXATTR
> 0.77 97829.38 us 18.84 us 9477446.11 us 13532
> READDIR
> 0.79 320.44 us 7.71 us 2369511.06 us 4257297
> ENTRYLK
> 1.97 464.02 us 9.38 us 1038460.80 us 7333722
> INODELK
> 13.73 2372.50 us 8.33 us 6258159.87 us 9979799
> LOOKUP
> 80.79 42372.04 us 61.49 us 2238308.13 us 3289103
> XATTROP
> 0.00 0.00 us 0.00 us 0.00 us 2250268
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 3756
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 167
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 1908
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 2246352
> CI_FORGET
>
> Duration: 209334 seconds
> Data Read: 1059777395 bytes
> Data Written: 2336921014 bytes
>
> Interval 0 Stats:
> Block Size: 1b+ 8b+
> 16b+
> No. of Reads: 0
> 0 0
> No. of Writes: 117
> 195 367
>
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 130
> 0 3
> No. of Writes: 1012
> 9 134
>
> Block Size: 256b+ 512b+
> 1024b+
> No. of Reads: 47
> 8 257
> No. of Writes: 729
> 73 168
>
> Block Size: 2048b+ 4096b+
> 8192b+
> No. of Reads: 10
> 27 240
> No. of Writes: 294 466
> 10472
>
> Block Size: 16384b+ 32768b+
> 65536b+
> No. of Reads: 139
> 119 135
> No. of Writes: 1480 6273
> 3935
>
> Block Size: 131072b+
> No. of Reads: 7892
> No. of Writes: 12327
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 955058
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1450749
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 81377
> RELEASEDIR
> 0.00 15.55 us 5.35 us 31.88 us
> 108 IPC
> 0.00 125.85 us 59.77 us 224.15 us 44
> FTRUNCATE
> 0.00 198.38 us 81.68 us 1332.45 us 30
> REMOVEXATTR
> 0.00 4254.16 us 621.52 us 10602.99 us 4
> FSYNC
> 0.00 795.08 us 190.30 us 16518.64 us 30
> SYMLINK
> 0.00 195.90 us 111.02 us 953.39 us 137
> LINK
> 0.00 286.07 us 179.64 us 437.82 us 119
> MKNOD
> 0.00 162.61 us 65.62 us 3864.77 us 221
> SETXATTR
> 0.00 117.29 us 48.50 us 317.17 us 354
> STAT
> 0.00 70.35 us 23.10 us 317.32 us
> 777 LK
> 0.00 461.72 us 114.54 us 16013.81 us 188
> RENAME
> 0.00 150.43 us 75.59 us 749.54 us 810
> FGETXATTR
> 0.00 267.80 us 99.67 us 994.58 us 698
> TRUNCATE
> 0.00 1176.00 us 194.89 us 78261.12 us 178
> MKDIR
> 0.00 439.83 us 54.38 us 75270.51 us 1665
> SETATTR
> 0.00 65.89 us 17.39 us 354.77 us 11372
> FINODELK
> 0.00 131.57 us 32.68 us 2107.52 us 7398
> FSTAT
> 0.00 120.74 us 31.43 us 847.97 us 9094
> READ
> 0.00 60.75 us 7.25 us 22520.98 us 22614
> FLUSH
> 0.00 73.03 us 22.41 us 750.17 us 21656
> STATFS
> 0.00 323.63 us 166.56 us 23034.44 us 6145
> CREATE
> 0.00 217.50 us 50.60 us 30734.99 us 38051
> WRITE
> 0.01 146.39 us 1.80 us 318612.45 us 81377
> OPENDIR
> 0.01 5641.37 us 108.84 us 468960.33 us 3177
> RMDIR
> 0.03 100.06 us 30.93 us 49756.93 us 432439
> READLINK
> 0.13 483.47 us 16.81 us 2608013.14 us 448381
> UNLINK
> 0.21 7101.56 us 22.26 us 2554390.49 us 52185
> READDIRP
> 0.35 414.01 us 36.58 us 1038229.93 us 1444606
> OPEN
> 0.49 78518.02 us 73.39 us 373999.31 us 10745
> FXATTROP
> 0.72 776.24 us 11.24 us 4271792.57 us 1605822
> GETXATTR
> 0.77 97829.38 us 18.84 us 9477446.11 us 13532
> READDIR
> 0.79 320.45 us 7.71 us 2369511.06 us 4257297
> ENTRYLK
> 1.97 464.07 us 9.38 us 1038460.80 us 7333722
> INODELK
> 13.72 2372.42 us 8.33 us 6258159.87 us 9979799
> LOOKUP
> 80.79 42373.50 us 61.49 us 2238308.13 us 3289103
> XATTROP
> 0.00 0.00 us 0.00 us 0.00 us 2250268
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 3756
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 167
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 1908
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 2246352
> CI_FORGET
>
> Duration: 209334 seconds
> Data Read: 1059777395 bytes
> Data Written: 2336921014 bytes
>
> Brick: flexo:/mnt/gluster/c
> --------------------------------------
> Cumulative Stats:
> Block Size: 1b+
> No. of Reads: 0
> No. of Writes: 32843
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 686562
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 834596
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 93757
> RELEASEDIR
> 0.00 18.50 us 8.21 us 50.20 us
> 107 IPC
> 0.00 205.83 us 126.30 us 412.06 us 36
> LINK
> 0.00 167.33 us 75.01 us 272.47 us 65
> FGETXATTR
> 0.00 203.58 us 117.35 us 648.28 us 88
> RENAME
> 0.00 203.20 us 66.60 us 809.96 us 153
> TRUNCATE
> 0.00 67.06 us 23.64 us 555.59 us
> 491 LK
> 0.01 51.73 us 7.70 us 1174.64 us 9339
> FLUSH
> 0.01 62.41 us 15.13 us 636.35 us 9011
> FINODELK
> 0.03 126.23 us 43.15 us 4760.14 us 10824
> FSTAT
> 0.03 571.94 us 94.85 us 51697.03 us 2594
> MKDIR
> 0.05 264.87 us 81.58 us 53152.06 us 9009
> FXATTROP
> 0.05 423.05 us 38.26 us 548686.55 us 5754
> CREATE
> 0.05 78.14 us 19.17 us 678.32 us 32843
> WRITE
> 0.06 526.39 us 130.84 us 41405.46 us 5798
> SYMLINK
> 0.10 133.73 us 16.25 us 676494.72 us 34904
> READDIR
> 0.12 2054.48 us 99.28 us 88061.83 us 2732
> RMDIR
> 0.21 110.60 us 1.19 us 16833.78 us 93757
> OPENDIR
> 0.72 205.89 us 33.28 us 261023.99 us 170292
> UNLINK
> 0.85 2971.45 us 21.24 us 257839.19 us 13840
> READDIRP
> 1.94 200.88 us 25.49 us 461654.32 us 468842
> FTRUNCATE
> 2.13 212.44 us 54.02 us 461878.34 us 486882
> REMOVEXATTR
> 2.27 223.01 us 52.03 us 178897.40 us 493240
> SETXATTR
> 3.53 155.31 us 7.96 us 67873.00 us 1101338
> GETXATTR
> 3.63 212.32 us 44.42 us 551264.02 us 829002
> OPEN
> 5.77 76.36 us 7.76 us 962634.73 us 3660425
> ENTRYLK
> 6.23 304.93 us 42.51 us 661394.43 us 990644
> SETATTR
> 7.81 382.20 us 107.42 us 846502.78 us 990766
> XATTROP
> 12.39 549.11 us 135.57 us 1659697.83 us 1093387
> MKNOD
> 12.47 139.78 us 9.76 us 552067.56 us 4325281
> INODELK
> 39.53 206.94 us 8.56 us 964823.33 us 9259429
> LOOKUP
> 0.00 0.00 us 0.00 us 0.00 us 1856377
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 933300
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 251
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 293
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 923077
> CI_FORGET
>
> Duration: 209334 seconds
> Data Read: 0 bytes
> Data Written: 32843 bytes
>
> Interval 0 Stats:
> Block Size: 1b+
> No. of Reads: 0
> No. of Writes: 32843
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 686562
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 834596
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 93757
> RELEASEDIR
> 0.00 18.50 us 8.21 us 50.20 us
> 107 IPC
> 0.00 205.83 us 126.30 us 412.06 us 36
> LINK
> 0.00 167.33 us 75.01 us 272.47 us 65
> FGETXATTR
> 0.00 203.58 us 117.35 us 648.28 us 88
> RENAME
> 0.00 203.20 us 66.60 us 809.96 us 153
> TRUNCATE
> 0.00 67.06 us 23.64 us 555.59 us
> 491 LK
> 0.01 51.71 us 7.70 us 1174.64 us 9339
> FLUSH
> 0.01 62.41 us 15.13 us 636.35 us 9011
> FINODELK
> 0.03 126.23 us 43.15 us 4760.14 us 10824
> FSTAT
> 0.03 571.94 us 94.85 us 51697.03 us 2594
> MKDIR
> 0.05 264.87 us 81.58 us 53152.06 us 9009
> FXATTROP
> 0.05 423.05 us 38.26 us 548686.55 us 5754
> CREATE
> 0.05 78.14 us 19.17 us 678.32 us 32843
> WRITE
> 0.06 526.39 us 130.84 us 41405.46 us 5798
> SYMLINK
> 0.10 133.73 us 16.25 us 676494.72 us 34904
> READDIR
> 0.12 2054.48 us 99.28 us 88061.83 us 2732
> RMDIR
> 0.21 110.60 us 1.19 us 16833.78 us 93757
> OPENDIR
> 0.72 205.89 us 33.28 us 261023.99 us 170292
> UNLINK
> 0.85 2971.45 us 21.24 us 257839.19 us 13840
> READDIRP
> 1.94 200.87 us 25.49 us 461654.32 us 468842
> FTRUNCATE
> 2.13 212.44 us 54.02 us 461878.34 us 486882
> REMOVEXATTR
> 2.27 223.00 us 52.03 us 178897.40 us 493240
> SETXATTR
> 3.53 155.32 us 7.96 us 67873.00 us 1101338
> GETXATTR
> 3.63 212.32 us 44.42 us 551264.02 us 829002
> OPEN
> 5.77 76.36 us 7.76 us 962634.73 us 3660425
> ENTRYLK
> 6.23 304.88 us 42.51 us 661394.43 us 990644
> SETATTR
> 7.81 382.20 us 107.42 us 846502.78 us 990766
> XATTROP
> 12.39 549.25 us 135.57 us 1659697.83 us 1093387
> MKNOD
> 12.47 139.78 us 9.76 us 552067.56 us 4325281
> INODELK
> 39.53 206.95 us 8.56 us 964823.33 us 9259429
> LOOKUP
> 0.00 0.00 us 0.00 us 0.00 us 1856377
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 933300
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 251
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 293
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 923077
> CI_FORGET
>
> Duration: 209334 seconds
> Data Read: 0 bytes
> Data Written: 32843 bytes
>
> Brick: bender:/mnt/gluster/c
> ---------------------------------------
> Cumulative Stats:
> Block Size: 1b+
> No. of Reads: 0
> No. of Writes: 37050
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 928119
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1401524
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 70092
> RELEASEDIR
> 0.00 19.49 us 8.28 us 180.61 us
> 100 IPC
> 0.00 238.88 us 163.97 us 486.93 us 38
> LINK
> 0.00 306.43 us 196.67 us 788.36 us 30
> SYMLINK
> 0.00 2670.18 us 765.93 us 4151.64 us 4
> FSYNC
> 0.00 350.65 us 141.92 us 10573.64 us 122
> RENAME
> 0.00 67.02 us 20.66 us 453.90 us
> 777 LK
> 0.00 158.32 us 64.22 us 532.21 us 810
> FGETXATTR
> 0.00 229.87 us 66.47 us 826.31 us 603
> TRUNCATE
> 0.01 1742.85 us 202.81 us 47471.71 us 302
> MKDIR
> 0.01 59.05 us 15.40 us 1813.56 us 10060
> FINODELK
> 0.01 118.54 us 36.65 us 4002.58 us 6409
> FSTAT
> 0.01 49.38 us 8.08 us 1264.41 us 21719
> FLUSH
> 0.03 74.76 us 20.86 us 2008.41 us 37050
> WRITE
> 0.04 553.12 us 40.81 us 386485.29 us 5901
> CREATE
> 0.06 534.01 us 100.86 us 527770.44 us 10057
> FXATTROP
> 0.12 138.96 us 1.00 us 601208.70 us 70092
> OPENDIR
> 0.12 947.76 us 18.54 us 712568.33 us 10779
> READDIR
> 0.15 4099.07 us 124.69 us 421939.47 us 3008
> RMDIR
> 0.61 465.30 us 20.67 us 936959.86 us 109478
> UNLINK
> 1.57 182.41 us 49.54 us 73938.09 us 722669
> REMOVEXATTR
> 1.61 171.41 us 6.98 us 83809.88 us 789294
> GETXATTR
> 1.86 217.88 us 26.40 us 556446.19 us 718835
> FTRUNCATE
> 1.89 219.47 us 49.92 us 506767.78 us 722856
> SETXATTR
> 3.02 181.66 us 44.40 us 344656.69 us 1395957
> OPEN
> 5.56 322.08 us 43.37 us 634069.61 us 1450160
> SETATTR
> 5.84 115.88 us 7.67 us 747387.51 us 4233520
> ENTRYLK
> 6.53 378.33 us 122.84 us 766939.14 us 1449601
> XATTROP
> 9.75 115.48 us 10.70 us 556675.13 us 7097226
> INODELK
> 19.37 1104.98 us 101.28 us 1612346.75 us 1472813
> MKNOD
> 41.85 295.61 us 8.98 us 996805.54 us 11896091
> LOOKUP
> 0.00 0.00 us 0.00 us 0.00 us 3531139
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 1862470
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 244
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 632
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 1668669
> CI_FORGET
>
> Duration: 208643 seconds
> Data Read: 0 bytes
> Data Written: 37050 bytes
>
> Interval 0 Stats:
> Block Size: 1b+
> No. of Reads: 0
> No. of Writes: 37050
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 928119
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1401524
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 70092
> RELEASEDIR
> 0.00 19.49 us 8.28 us 180.61 us
> 100 IPC
> 0.00 238.88 us 163.97 us 486.93 us 38
> LINK
> 0.00 306.43 us 196.67 us 788.36 us 30
> SYMLINK
> 0.00 2670.18 us 765.93 us 4151.64 us 4
> FSYNC
> 0.00 350.65 us 141.92 us 10573.64 us 122
> RENAME
> 0.00 67.02 us 20.66 us 453.90 us
> 777 LK
> 0.00 158.32 us 64.22 us 532.21 us 810
> FGETXATTR
> 0.00 229.87 us 66.47 us 826.31 us 603
> TRUNCATE
> 0.01 1742.85 us 202.81 us 47471.71 us 302
> MKDIR
> 0.01 59.05 us 15.40 us 1813.56 us 10060
> FINODELK
> 0.01 118.54 us 36.65 us 4002.58 us 6409
> FSTAT
> 0.01 49.39 us 8.08 us 1264.41 us 21719
> FLUSH
> 0.03 74.77 us 20.86 us 2008.41 us 37050
> WRITE
> 0.04 553.12 us 40.81 us 386485.29 us 5901
> CREATE
> 0.06 534.01 us 100.86 us 527770.44 us 10057
> FXATTROP
> 0.12 138.96 us 1.00 us 601208.70 us 70092
> OPENDIR
> 0.12 947.76 us 18.54 us 712568.33 us 10779
> READDIR
> 0.15 4099.07 us 124.69 us 421939.47 us 3008
> RMDIR
> 0.61 465.30 us 20.67 us 936959.86 us 109478
> UNLINK
> 1.57 182.41 us 49.54 us 73938.09 us 722669
> REMOVEXATTR
> 1.61 171.41 us 6.98 us 83809.88 us 789294
> GETXATTR
> 1.86 217.88 us 26.40 us 556446.19 us 718835
> FTRUNCATE
> 1.89 219.48 us 49.92 us 506767.78 us 722856
> SETXATTR
> 3.02 181.66 us 44.40 us 344656.69 us 1395957
> OPEN
> 5.56 322.09 us 43.37 us 634069.61 us 1450160
> SETATTR
> 5.84 115.88 us 7.67 us 747387.51 us 4233520
> ENTRYLK
> 6.53 378.33 us 122.84 us 766939.14 us 1449601
> XATTROP
> 9.75 115.48 us 10.70 us 556675.13 us 7097226
> INODELK
> 19.37 1105.06 us 101.28 us 1612346.75 us 1472813
> MKNOD
> 41.84 295.54 us 8.98 us 996805.54 us 11896091
> LOOKUP
> 0.00 0.00 us 0.00 us 0.00 us 3531139
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 1862470
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 244
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 632
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 1668669
> CI_FORGET
>
> Duration: 208643 seconds
> Data Read: 0 bytes
> Data Written: 37050 bytes
>
> Brick: bender:/mnt/gluster/a
> ---------------------------------------
> Cumulative Stats:
> Block Size: 1b+
> 2b+ 8b+
> No. of Reads: 0
> 0 0
> No. of Writes: 53
> 1 60
>
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0
> 235 4
> No. of Writes: 14
> 1033 15
>
> Block Size: 128b+ 256b+
> 512b+
> No. of Reads: 4
> 115 81
> No. of Writes: 58
> 703 80
>
> Block Size: 1024b+ 2048b+
> 4096b+
> No. of Reads: 181
> 26 74
> No. of Writes: 163
> 278 397
>
> Block Size: 8192b+ 16384b+
> 32768b+
> No. of Reads: 131
> 161 267
> No. of Writes: 7264 1398
> 6742
>
> Block Size: 65536b+ 131072b+
> 262144b+
> No. of Reads: 443
> 25528 0
> No. of Writes: 2426
> 12240 1
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 684942
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 837092
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 87297
> RELEASEDIR
> 0.00 219.22 us 121.08 us 306.12 us 8
> REMOVEXATTR
> 0.00 21.70 us 7.24 us 196.07 us
> 97 IPC
> 0.00 152.77 us 78.90 us 1059.82 us 46
> FTRUNCATE
> 0.00 295.73 us 167.10 us 606.96 us 65
> FGETXATTR
> 0.00 243.66 us 140.50 us 3392.16 us 121
> LINK
> 0.00 1216.47 us 194.37 us 23411.42 us 25
> SYMLINK
> 0.00 327.87 us 234.27 us 551.71 us 107
> MKNOD
> 0.00 36094.61 us 36094.61 us 36094.61 us 1
> FSYNC
> 0.00 219.63 us 147.34 us 783.10 us 188
> RENAME
> 0.00 104.27 us 40.34 us 595.29 us 745
> STAT
> 0.00 531.71 us 113.57 us 53973.08 us 200
> TRUNCATE
> 0.00 289.05 us 63.10 us 70107.41 us 1444
> SETATTR
> 0.00 2430.15 us 232.78 us 68120.54 us 177
> MKDIR
> 0.00 52.68 us 7.28 us 826.01 us 9890
> FLUSH
> 0.00 60.96 us 17.54 us 1393.95 us 9792
> FINODELK
> 0.00 74.26 us 20.00 us 1522.39 us 21521
> STATFS
> 0.00 129.21 us 34.15 us 9068.67 us 13094
> FSTAT
> 0.00 456.99 us 195.50 us 130182.02 us 5967
> CREATE
> 0.00 112.06 us 28.14 us 9545.53 us 27474
> READ
> 0.01 313.76 us 52.98 us 71635.74 us 32926
> WRITE
> 0.01 135.84 us 1.52 us 671373.08 us 87297
> OPENDIR
> 0.01 5225.28 us 131.28 us 951323.69 us 3177
> RMDIR
> 0.01 203.34 us 56.13 us 486756.90 us 85073
> SETXATTR
> 0.03 3521.00 us 145.28 us 3396306.00 us 15090
> READDIRP
> 0.04 113758.50 us 26.72 us 1147074.44 us
> 564 LK
> 0.05 92.18 us 29.71 us 39069.99 us 850938
> READLINK
> 0.23 453.79 us 41.84 us 655999.23 us 831212
> OPEN
> 0.41 29425.34 us 19.20 us 1526187.08 us 22482
> READDIR
> 0.50 1954.70 us 21.51 us 1526772.28 us 415226
> UNLINK
> 0.54 1272.83 us 9.66 us 1604368.93 us 686945
> GETXATTR
> 0.55 237.57 us 7.91 us 1277047.91 us 3739718
> ENTRYLK
> 0.60 103888.49 us 89.32 us 761509.35 us 9311
> FXATTROP
> 2.60 909.04 us 9.68 us 2272778.49 us 4649291
> INODELK
> 5.82 1240.20 us 8.78 us 1526868.50 us 7625383
> LOOKUP
> 88.60 58005.98 us 67.31 us 1448810.06 us 2481914
> XATTROP
> 0.00 0.00 us 0.00 us 0.00 us 1396875
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 1854
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 28
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 1051
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 1394996
> CI_FORGET
>
> Duration: 208643 seconds
> Data Read: 3409342848 bytes
> Data Written: 2180686507 bytes
>
> Interval 0 Stats:
> Block Size: 1b+
> 2b+ 8b+
> No. of Reads: 0
> 0 0
> No. of Writes: 53
> 1 60
>
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0
> 235 4
> No. of Writes: 14
> 1033 15
>
> Block Size: 128b+ 256b+
> 512b+
> No. of Reads: 4
> 115 81
> No. of Writes: 58
> 703 80
>
> Block Size: 1024b+ 2048b+
> 4096b+
> No. of Reads: 181
> 26 74
> No. of Writes: 163
> 278 397
>
> Block Size: 8192b+ 16384b+
> 32768b+
> No. of Reads: 131
> 161 267
> No. of Writes: 7264 1398
> 6742
>
> Block Size: 65536b+ 131072b+
> 262144b+
> No. of Reads: 443
> 25528 0
> No. of Writes: 2426
> 12240 1
>
> %-latency Avg-latency Min-Latency Max-Latency No. of
> calls Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 684942
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 837092
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 87297
> RELEASEDIR
> 0.00 219.22 us 121.08 us 306.12 us 8
> REMOVEXATTR
> 0.00 21.70 us 7.24 us 196.07 us
> 97 IPC
> 0.00 152.77 us 78.90 us 1059.82 us 46
> FTRUNCATE
> 0.00 295.73 us 167.10 us 606.96 us 65
> FGETXATTR
> 0.00 243.66 us 140.50 us 3392.16 us 121
> LINK
> 0.00 1216.47 us 194.37 us 23411.42 us 25
> SYMLINK
> 0.00 327.87 us 234.27 us 551.71 us 107
> MKNOD
> 0.00 36094.61 us 36094.61 us 36094.61 us 1
> FSYNC
> 0.00 219.63 us 147.34 us 783.10 us 188
> RENAME
> 0.00 104.27 us 40.34 us 595.29 us 745
> STAT
> 0.00 531.71 us 113.57 us 53973.08 us 200
> TRUNCATE
> 0.00 289.05 us 63.10 us 70107.41 us 1444
> SETATTR
> 0.00 2430.15 us 232.78 us 68120.54 us 177
> MKDIR
> 0.00 52.69 us 7.28 us 826.01 us 9890
> FLUSH
> 0.00 60.96 us 17.54 us 1393.95 us 9792
> FINODELK
> 0.00 74.25 us 20.00 us 1522.39 us 21521
> STATFS
> 0.00 129.23 us 34.15 us 9068.67 us 13094
> FSTAT
> 0.00 456.99 us 195.50 us 130182.02 us 5967
> CREATE
> 0.00 112.45 us 28.14 us 9545.53 us 27474
> READ
> 0.01 313.77 us 52.98 us 71635.74 us 32926
> WRITE
> 0.01 135.84 us 1.52 us 671373.08 us 87297
> OPENDIR
> 0.01 5225.28 us 131.28 us 951323.69 us 3177
> RMDIR
> 0.01 203.34 us 56.13 us 486756.90 us 85073
> SETXATTR
> 0.03 3521.00 us 145.28 us 3396306.00 us 15090
> READDIRP
> 0.04 113758.50 us 26.72 us 1147074.44 us
> 564 LK
> 0.05 92.18 us 29.71 us 39069.99 us 850938
> READLINK
> 0.23 453.79 us 41.84 us 655999.23 us 831212
> OPEN
> 0.41 29425.34 us 19.20 us 1526187.08 us 22482
> READDIR
> 0.50 1954.70 us 21.51 us 1526772.28 us 415226
> UNLINK
> 0.54 1272.79 us 9.66 us 1604368.93 us 686945
> GETXATTR
> 0.55 237.57 us 7.91 us 1277047.91 us 3739718
> ENTRYLK
> 0.60 103888.49 us 89.32 us 761509.35 us 9311
> FXATTROP
> 2.60 909.04 us 9.68 us 2272778.49 us 4649291
> INODELK
> 5.82 1240.10 us 8.78 us 1526868.50 us 7625383
> LOOKUP
> 88.60 58006.02 us 67.31 us 1448810.06 us 2481914
> XATTROP
> 0.00 0.00 us 0.00 us 0.00 us 1396875
> UPCALL
> 0.00 0.00 us 0.00 us 0.00 us 1854
> CI_IATT
> 0.00 0.00 us 0.00 us 0.00 us 28
> CI_XATTR
> 0.00 0.00 us 0.00 us 0.00 us 1051
> CI_UNLINK
> 0.00 0.00 us 0.00 us 0.00 us 1394996
> CI_FORGET
>
> Duration: 208643 seconds
> Data Read: 3409342848 bytes
> Data Written: 2180686507 bytes
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200504/76239ebb/attachment.html>
More information about the Gluster-users
mailing list