[Gluster-users] Possible SHD stalling
Jaco Kroon
jaco at uls.co.za
Thu May 7 19:40:54 UTC 2020
Hi,
So upgrading to 7.5 made a difference in that it takes a bit longer to
cause chaos, but it still does the exact same thing.
To be precise:
I've now had to set:
gluster volume set vpbx_shared cluster.heal-wait-queue-length 1
gluster volume set vpbx_shared cluster.shd-wait-qlength 1
gluster volume set vpbx_shared cluster.shd-max-threads 1
With any of these settings >1 I end up in a situation where simple
directory listings of < 100 entries in a folder takes upwards of a
minute to complete *from time to time*.
It seems that when the logs
With these the maximum stall I can reproduce is usually around 3 seconds
before the directory listing will complete, but about 5s worst case.
I've now reverted these as well, which are known for us to cause issues
under heal required conditions:
cluster.metadata-self-heal
on
cluster.data-self-heal
on
cluster.entry-self-heal
on
Not sure if I need to remount for these to become effective but I've
done so just to be on the safe side.
I'm still seeing these errors:
[2020-05-07 19:20:12.763311] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]
[2020-05-07 19:20:12.995675] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]
[2020-05-07 19:20:59.199776] W [MSGID: 114031]
[client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
0-vpbx_shared-client-1: remote operation failed [Invalid argument]
And the start of my issues still seem to co-incide with that. Sample case:
# time ls
...
real 0m5.140s
user 0m0.009s
sys 0m0.000s
# f=(*); echo "${#f[@]}"
165
This just does not seem right. Note, again, if I set *any* of the above
parameters >1 these delays get far, far worse (I've had over a minute
for this exact same ls).
Any advise would be greatly appreciated. To me, it looks like some lock
is being taken somewhere, and takes a while to release. I don't
understand the glusterfs internals nearly well enough to understand
exactly what's going on here, nor do I know what information to provide
to help drill down on this. Happy to go look at code, if I can get
pointed in a direction.
Kind Regards,
Jaco
On 2020/05/05 08:43, Jaco Kroon wrote:
>
> Hi Artem,
>
> I'll do that. In the meantime consider this (inferred from the
> changelogs).
>
> It seems like glusterfs SHD is queueing into a bounded queue under
> lock conditions, and waiting for space to open up. At least, this is
> what's hinted in the changelogs, I've been trying to grasp the SHD
> code for two days now and am not getting to wrap my head around that.
> If this is true, then the only way to avoid lock-ups are to ensure
> that the queue is larger than the total number of files that can ever
> require healing (effectively making the queue unbounded).
>
> Kind Regards,
> Jaco
>
> On 2020/05/04 21:56, Artem Russakovskii wrote:
>> I reported something similar recently to this list, and I highly
>> advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then
>> see if it keeps happening. The self heal issues should be fixed there
>> properly at least.
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR>
>>
>>
>> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za
>> <mailto:jaco at uls.co.za>> wrote:
>>
>> Hi All,
>>
>> So we've got a cluster currently running on glusterfs 7.3
>>
>> Heal performance is absolutely terrible (it starts of great, runs
>> for a while then completely trips up).
>>
>> Once the SHD trips up, access speeds to the cluster also becomes
>> poor. And with poor I mean directory listings that normally
>> takes well under 500ms takes upwards of 10s, but I've seen
>> upwards of 40s (although a follow up ls immediately after is nice
>> and fast again, temporarily). With all SHDs killed this
>> particular ls is as per normal (around 30-50ms).
>>
>> The SHD logs doesn't provide any usable clues, the only variance
>> in logs is this, just before all daemons across the cluster
>> crawls to a near halt.
>>
>> [2020-05-02 19:55:58.627036] I [MSGID: 108026]
>> [afr-self-heal-common.c:1747:afr_log_selfheal]
>> 0-vpbx_shared-replicate-0: Completed entry selfheal on
>> 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1
>> [2020-05-02 19:55:58.627058] I [MSGID: 108026]
>> [afr-self-heal-common.c:1747:afr_log_selfheal]
>> 0-vpbx_shared-replicate-0: Completed entry selfheal on
>> c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1
>>
>> [2020-05-02 19:55:59.125001] W [MSGID: 114031]
>> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
>> 0-vpbx_shared-client-1: remote operation failed [Invalid argument]
>>
>> [2020-05-02 19:55:59.339803] I [MSGID: 108026]
>> [afr-self-heal-common.c:1747:afr_log_selfheal]
>> 0-vpbx_shared-replicate-0: Completed entry selfheal on
>> 2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1
>> [2020-05-02 19:55:59.490387] I [MSGID: 108026]
>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>> 0-vpbx_shared-replicate-0: performing metadata selfheal on
>> 527a63a3-9acd-4c3f-b95c-5c719325c973
>>
>> A few more entries will complete overall and then it'll be a file
>> every other second. Once I saw this I ran a few more tests, and
>> on every test just prior to slowing to a crawl, that entry is there.
>>
>> There is a perceived improvement when lowering the thread count,
>> but not significant enough to confirm a correlation. Even
>> with**cluster.shd-max-threads=1 this still happens (gets a few
>> tens thousand files completed instead of few hundred, at 32 it
>> gets to maybe 400 to 500 before it stalls, 16 does maybe a
>> thousand).*
>> *
>>
>> There are probably performant bursts hereafter, because from
>> experience we know this process takes about two months. But this
>> time around it seems to be much, much worse. Might be because
>> that was healing a single brick to another brick, so less locking
>> going? At the moment the replicas are basically trying to
>> populate the newly added arbiters. Also, we haven't messed as
>> greatly with the health check settings (queues and threads and
>> locks nearly as much as since the arbiter addition because
>> performance has simply dropped to non-acceptable levels.
>>
>> Unfortunately granular entry heal isn't on at the moment (nor can
>> I switch it on due to a few million outstanding heals). This
>> used to be a 2x2 cluster. We then moved one of the bricks to a
>> newly added server using replace-brick. What a frightening
>> process this is. We then proceeded to add the arbiters.
>>
>> Some of the settings below was absolute shots in the dark, based
>> on the little information we've been able to track down so far.
>>
>> Kind Regards,
>> Jaco
>>
>> The current gluster volume info (this used to be 2x2
>> distributed-replicate, aiming to get it to 3x (2 + 1) ... at
>> current rates we might get there by 2021):
>>
>> Volume Name: glvol
>> Type: Distributed-Replicate
>> Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 2 x (2 + 1) = 6
>> Transport-type: tcp
>> Bricks:
>> Brick1: zoidberg:/mnt/gluster/a
>> Brick2: flexo:/mnt/gluster/b
>> Brick3: bender:/mnt/gluster/c (arbiter)
>> Brick4: bender:/mnt/gluster/a
>> Brick5: zoidberg:/mnt/gluster/b
>> Brick6: flexo:/mnt/gluster/c (arbiter)
>> Options Reconfigured:
>> *cluster.heal-timeout: 5 <-- hoping this would at least shorten
>> the blockages.*
>> cluster.locking-scheme: granular
>> cluster.heal-wait-queue-length: 10240
>> *diagnostics.count-fop-hits: on**
>> **diagnostics.latency-measurement: on <-- was hoping profiling
>> might provide some insight. Going to switch this off now.*
>> cluster.background-self-heal-count: 128
>> *cluster.shd-wait-qlength: 131072**
>> **cluster.shd-max-threads: 32 <-- even with this disk IO never
>> goes over 10% utilization. CPU goes to about 2 cores. So it's
>> lock contention that's slowing it down we reckon.*
>> *cluster.metadata-self-heal: off**
>> **cluster.entry-self-heal: off**
>> **cluster.data-self-heal: off**<-- these kill performance.
>> *transport.address-family: inet
>> nfs.disable: on
>> performance.client-io-threads: on
>> features.cache-invalidation: on
>> features.cache-invalidation-timeout: 600
>> performance.stat-prefetch: on
>> performance.cache-invalidation: on
>> performance.md-cache-timeout: 600
>> *cluster.self-heal-daemon: yes <-- disabling/enabling this
>> doesn't help, have to kill the shd and start ... force to get
>> things moving along again.*
>> performance.io-thread-count: 16
>> performance.high-prio-threads: 16
>> performance.normal-prio-threads: 16
>> performance.low-prio-threads: 16
>> performance.least-prio-threads: 4
>> cluster.self-heal-window-size: 512
>> cluster.self-heal-readdir-size: 131072
>> client.event-threads: 8
>> server.event-threads: 16
>> network.ping-timeout: 2
>> *cluster.data-self-heal-algorithm: full <-- most files are small
>> (<1MB), on dual 1gbit links (so up to 2gbit, but b/w never goes
>> over 5mbit anyway).*
>>
>> performance info:
>>
>> Brick: zoidberg:/mnt/gluster/a
>> -----------------------------------------
>> Cumulative Stats:
>> Block Size: 1b+
>> 8b+ 16b+
>> No. of Reads: 0
>> 0 0
>> No. of Writes: 117
>> 195 367
>>
>> Block Size: 32b+
>> 64b+ 128b+
>> No. of Reads: 306
>> 0 6
>> No. of Writes: 1016
>> 9 134
>>
>> Block Size: 256b+
>> 512b+ 1024b+
>> No. of Reads: 107
>> 58 485
>> No. of Writes: 731
>> 73 168
>>
>> Block Size: 2048b+
>> 4096b+ 8192b+
>> No. of Reads: 43
>> 59 383
>> No. of Writes: 295
>> 475 10562
>>
>> Block Size: 16384b+
>> 32768b+ 65536b+
>> No. of Reads: 347
>> 380 634
>> No. of Writes: 1487
>> 6369 3970
>>
>> Block Size: 131072b+
>> No. of Reads: 35704
>> No. of Writes: 12398
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 957293 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1455091 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 69951 RELEASEDIR
>> 0.00 16.20 us 5.43 us 100.95 us
>> 113 IPC
>> 0.00 687.33 us 80.53 us 15414.03 us
>> 30 REMOVEXATTR
>> 0.00 840.76 us 72.71 us 37648.39 us
>> 52 FTRUNCATE
>> 0.00 138.49 us 36.66 us 25280.24 us
>> 1385 STAT
>> 0.00 364.93 us 63.45 us 15957.66 us
>> 810 FGETXATTR
>> 0.00 2208.82 us 127.23 us 267092.17 us
>> 137 LINK
>> 0.00 37595.33 us 1677.46 us 218313.32 us
>> 12 FSYNC
>> 0.00 675.61 us 106.82 us 114351.01 us
>> 698 TRUNCATE
>> 0.00 3995.66 us 147.00 us 109544.46 us
>> 188 RENAME
>> 0.00 453.11 us 53.37 us 104848.94 us
>> 1680 SETATTR
>> 0.00 6004.32 us 229.33 us 151322.58 us
>> 127 MKNOD
>> 0.00 68.74 us 16.11 us 61050.93 us
>> 11412 FINODELK
>> 0.00 3793.44 us 68.61 us 411523.70 us
>> 221 SETXATTR
>> 0.00 28669.99 us 227.28 us 138988.41 us
>> 30 SYMLINK
>> 0.00 56.18 us 7.31 us 41484.37 us
>> 22640 FLUSH
>> 0.00 105.64 us 19.34 us 351055.72 us
>> 21693 STATFS
>> 0.00 23642.90 us 244.19 us 234912.25 us
>> 178 MKDIR
>> 0.00 537.38 us 31.73 us 296911.67 us
>> 9653 FSTAT
>> 0.00 214.87 us 25.71 us 1712020.99 us
>> 39054 READ
>> 0.01 147.08 us 1.47 us 241611.80 us
>> 69951 OPENDIR
>> 0.01 3899.84 us 117.22 us 503422.21 us
>> 3180 RMDIR
>> 0.01 327.08 us 55.69 us 420376.80 us
>> 38366 WRITE
>> 0.02 5580.36 us 210.11 us 1029702.30 us
>> 6169 CREATE
>> 0.05 92.69 us 29.94 us 687955.86 us
>> 1159936 READLINK
>> 0.40 543.50 us 36.45 us 2430653.54 us
>> 1449307 OPEN
>> 0.48 87643.67 us 70.20 us 661999.51 us
>> 10787 FXATTROP
>> 0.53 2342.72 us 20.91 us 4157542.43 us
>> 447821 UNLINK
>> 0.53 558734.79 us 24.40 us 1399487.63 us
>> 1889 LK
>> 0.78 145786.31 us 19.06 us 37490825.73 us
>> 10687 READDIR
>> 0.99 2426.23 us 12.47 us 24176123.13 us
>> 809430 GETXATTR
>> 1.45 676.83 us 9.30 us 193177067.25 us
>> 4260021 ENTRYLK
>> 1.76 38325.38 us 21.44 us 46693575.93 us
>> 91439 READDIRP
>> 4.44 1200.31 us 9.10 us 2559377.90 us
>> 7352811 INODELK
>> 9.30 1849.24 us 9.28 us 35299050.00 us
>> 9989221 LOOKUP
>> 79.24 47791.46 us 76.77 us 10488824.26 us
>> 3294642 XATTROP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 3194806 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 5311 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 54 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1907 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 3189457 CI_FORGET
>>
>> Duration: 209524 seconds
>> Data Read: 4775365531 bytes
>> Data Written: 2354108664 bytes
>>
>> Interval 0 Stats:
>> Block Size: 1b+
>> 8b+ 16b+
>> No. of Reads: 0
>> 0 0
>> No. of Writes: 117
>> 195 367
>>
>> Block Size: 32b+
>> 64b+ 128b+
>> No. of Reads: 306
>> 0 6
>> No. of Writes: 1016
>> 9 134
>>
>> Block Size: 256b+
>> 512b+ 1024b+
>> No. of Reads: 107
>> 58 485
>> No. of Writes: 731
>> 73 168
>>
>> Block Size: 2048b+
>> 4096b+ 8192b+
>> No. of Reads: 43
>> 59 383
>> No. of Writes: 295
>> 475 10562
>>
>> Block Size: 16384b+
>> 32768b+ 65536b+
>> No. of Reads: 347
>> 380 634
>> No. of Writes: 1487
>> 6369 3970
>>
>> Block Size: 131072b+
>> No. of Reads: 35704
>> No. of Writes: 12398
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 957293 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1455091 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 69951 RELEASEDIR
>> 0.00 16.20 us 5.43 us 100.95 us
>> 113 IPC
>> 0.00 687.33 us 80.53 us 15414.03 us
>> 30 REMOVEXATTR
>> 0.00 840.76 us 72.71 us 37648.39 us
>> 52 FTRUNCATE
>> 0.00 138.49 us 36.66 us 25280.24 us
>> 1385 STAT
>> 0.00 363.20 us 63.45 us 15957.66 us
>> 810 FGETXATTR
>> 0.00 2208.82 us 127.23 us 267092.17 us
>> 137 LINK
>> 0.00 37595.33 us 1677.46 us 218313.32 us
>> 12 FSYNC
>> 0.00 675.61 us 106.82 us 114351.01 us
>> 698 TRUNCATE
>> 0.00 3995.66 us 147.00 us 109544.46 us
>> 188 RENAME
>> 0.00 453.11 us 53.37 us 104848.94 us
>> 1680 SETATTR
>> 0.00 6004.32 us 229.33 us 151322.58 us
>> 127 MKNOD
>> 0.00 68.74 us 16.11 us 61050.93 us
>> 11412 FINODELK
>> 0.00 3793.44 us 68.61 us 411523.70 us
>> 221 SETXATTR
>> 0.00 28669.99 us 227.28 us 138988.41 us
>> 30 SYMLINK
>> 0.00 56.19 us 7.31 us 41484.37 us
>> 22640 FLUSH
>> 0.00 105.61 us 19.34 us 351055.72 us
>> 21693 STATFS
>> 0.00 23642.90 us 244.19 us 234912.25 us
>> 178 MKDIR
>> 0.00 537.38 us 31.73 us 296911.67 us
>> 9653 FSTAT
>> 0.00 214.91 us 25.71 us 1712020.99 us
>> 39054 READ
>> 0.01 147.08 us 1.47 us 241611.80 us
>> 69951 OPENDIR
>> 0.01 3899.84 us 117.22 us 503422.21 us
>> 3180 RMDIR
>> 0.01 327.13 us 55.69 us 420376.80 us
>> 38366 WRITE
>> 0.02 5580.36 us 210.11 us 1029702.30 us
>> 6169 CREATE
>> 0.05 92.69 us 29.94 us 687955.86 us
>> 1159936 READLINK
>> 0.40 543.49 us 36.45 us 2430653.54 us
>> 1449307 OPEN
>> 0.48 87643.67 us 70.20 us 661999.51 us
>> 10787 FXATTROP
>> 0.53 2342.57 us 20.91 us 4157542.43 us
>> 447821 UNLINK
>> 0.53 558416.10 us 24.40 us 1399487.63 us
>> 1889 LK
>> 0.78 145786.31 us 19.06 us 37490825.73 us
>> 10687 READDIR
>> 0.99 2426.24 us 12.47 us 24176123.13 us
>> 809430 GETXATTR
>> 1.45 676.75 us 9.30 us 193177067.25 us
>> 4260021 ENTRYLK
>> 1.76 38324.59 us 21.44 us 46693575.93 us
>> 91439 READDIRP
>> 4.44 1200.37 us 9.10 us 2559377.90 us
>> 7352811 INODELK
>> 9.29 1848.94 us 9.28 us 35299050.00 us
>> 9989221 LOOKUP
>> 79.24 47792.42 us 76.77 us 10488824.26 us
>> 3294642 XATTROP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 3194806 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 5311 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 54 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1907 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 3189457 CI_FORGET
>>
>> Duration: 209524 seconds
>> Data Read: 4775365531 bytes
>> Data Written: 2354108664 bytes
>>
>> Brick: zoidberg:/mnt/gluster/b
>> -----------------------------------------
>> Cumulative Stats:
>> Block Size: 1b+
>> 2b+ 8b+
>> No. of Reads: 0
>> 0 0
>> No. of Writes: 53
>> 1 60
>>
>> Block Size: 16b+
>> 32b+ 64b+
>> No. of Reads: 0
>> 213 2
>> No. of Writes: 14
>> 1057 15
>>
>> Block Size: 128b+
>> 256b+ 512b+
>> No. of Reads: 0
>> 41 146
>> No. of Writes: 59
>> 715 80
>>
>> Block Size: 1024b+
>> 2048b+ 4096b+
>> No. of Reads: 224
>> 9 15
>> No. of Writes: 163
>> 282 404
>>
>> Block Size: 8192b+
>> 16384b+ 32768b+
>> No. of Reads: 52
>> 74 210
>> No. of Writes: 7415
>> 1430 6860
>>
>> Block Size: 65536b+
>> 131072b+ 262144b+
>> No. of Reads: 299
>> 15313 1
>> No. of Writes: 2458
>> 12378 1
>>
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 686733 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 837434 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 87338 RELEASEDIR
>> 0.00 22.69 us 7.16 us 332.61 us
>> 116 IPC
>> 0.00 119.93 us 70.56 us 272.28 us
>> 57 FTRUNCATE
>> 0.00 152.25 us 86.82 us 340.63 us
>> 65 FGETXATTR
>> 0.00 2336.06 us 90.05 us 20509.81 us
>> 23 REMOVEXATTR
>> 0.00 631.26 us 123.17 us 51803.77 us
>> 121 LINK
>> 0.00 369.27 us 22.64 us 145090.48 us
>> 490 LK
>> 0.00 360.85 us 45.14 us 46846.98 us
>> 540 STAT
>> 0.00 1190.96 us 117.40 us 69781.85 us
>> 200 TRUNCATE
>> 0.00 245.72 us 54.91 us 123985.76 us
>> 1482 SETATTR
>> 0.00 42139.10 us 3313.56 us 312273.21 us
>> 12 FSYNC
>> 0.00 55.83 us 8.26 us 49649.35 us
>> 10047 FLUSH
>> 0.00 57.52 us 14.62 us 820.59 us
>> 9977 FINODELK
>> 0.00 3191.60 us 143.06 us 96214.75 us
>> 188 RENAME
>> 0.00 32862.86 us 215.38 us 83851.13 us
>> 25 SYMLINK
>> 0.00 11392.03 us 230.70 us 105471.90 us
>> 118 MKNOD
>> 0.00 134.63 us 28.08 us 36326.43 us
>> 12587 FSTAT
>> 0.00 112.32 us 19.91 us 381241.34 us
>> 21695 STATFS
>> 0.00 307.86 us 28.00 us 292666.83 us
>> 16841 READ
>> 0.00 29625.60 us 239.57 us 239532.26 us
>> 178 MKDIR
>> 0.00 319.24 us 51.19 us 136353.39 us
>> 33445 WRITE
>> 0.01 175.83 us 1.57 us 513635.22 us
>> 87338 OPENDIR
>> 0.01 249.96 us 58.42 us 594774.86 us
>> 85089 SETXATTR
>> 0.01 11001.25 us 119.87 us 612765.03 us
>> 3180 RMDIR
>> 0.04 14240.66 us 203.89 us 1376758.99 us
>> 6067 CREATE
>> 0.05 89.45 us 30.14 us 687961.60 us
>> 1219542 READLINK
>> 0.39 1114.30 us 44.29 us 2412169.89 us
>> 831371 OPEN
>> 0.51 127078.45 us 78.55 us 694364.09 us
>> 9493 FXATTROP
>> 0.63 399.16 us 8.43 us 3815689.36 us
>> 3764714 ENTRYLK
>> 0.86 91973.00 us 19.89 us 24258016.33 us
>> 22253 READDIR
>> 1.06 6913.71 us 11.58 us 36968532.72 us
>> 364963 GETXATTR
>> 1.29 62130.12 us 36.89 us 8649871.11 us
>> 49472 READDIRP
>> 2.89 1477.53 us 11.10 us 2807637.64 us
>> 4650500 INODELK
>> 4.12 23681.31 us 20.98 us 23291414.50 us
>> 414352 UNLINK
>> 16.09 5000.31 us 9.04 us 23866016.88 us
>> 7655621 LOOKUP
>> 72.04 68953.31 us 68.40 us 27542499.65 us
>> 2486126 XATTROP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2433405 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2453 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 69 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1544 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2430885 CI_FORGET
>>
>> Duration: 209524 seconds
>> Data Read: 2051388524 bytes
>> Data Written: 2208261499 bytes
>>
>> Interval 0 Stats:
>> Block Size: 1b+
>> 2b+ 8b+
>> No. of Reads: 0
>> 0 0
>> No. of Writes: 53
>> 1 60
>>
>> Block Size: 16b+
>> 32b+ 64b+
>> No. of Reads: 0
>> 213 2
>> No. of Writes: 14
>> 1057 15
>>
>> Block Size: 128b+
>> 256b+ 512b+
>> No. of Reads: 0
>> 41 146
>> No. of Writes: 59
>> 715 80
>>
>> Block Size: 1024b+
>> 2048b+ 4096b+
>> No. of Reads: 224
>> 9 15
>> No. of Writes: 163
>> 282 404
>>
>> Block Size: 8192b+
>> 16384b+ 32768b+
>> No. of Reads: 52
>> 74 210
>> No. of Writes: 7415
>> 1430 6860
>>
>> Block Size: 65536b+
>> 131072b+ 262144b+
>> No. of Reads: 299
>> 15313 1
>> No. of Writes: 2458
>> 12378 1
>>
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 686733 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 837434 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 87338 RELEASEDIR
>> 0.00 22.69 us 7.16 us 332.61 us
>> 116 IPC
>> 0.00 119.93 us 70.56 us 272.28 us
>> 57 FTRUNCATE
>> 0.00 152.25 us 86.82 us 340.63 us
>> 65 FGETXATTR
>> 0.00 2336.06 us 90.05 us 20509.81 us
>> 23 REMOVEXATTR
>> 0.00 631.26 us 123.17 us 51803.77 us
>> 121 LINK
>> 0.00 369.27 us 22.64 us 145090.48 us
>> 490 LK
>> 0.00 360.85 us 45.14 us 46846.98 us
>> 540 STAT
>> 0.00 1190.96 us 117.40 us 69781.85 us
>> 200 TRUNCATE
>> 0.00 245.72 us 54.91 us 123985.76 us
>> 1482 SETATTR
>> 0.00 42139.10 us 3313.56 us 312273.21 us
>> 12 FSYNC
>> 0.00 55.83 us 8.26 us 49649.35 us
>> 10047 FLUSH
>> 0.00 57.52 us 14.62 us 820.59 us
>> 9977 FINODELK
>> 0.00 3191.60 us 143.06 us 96214.75 us
>> 188 RENAME
>> 0.00 32862.86 us 215.38 us 83851.13 us
>> 25 SYMLINK
>> 0.00 11392.03 us 230.70 us 105471.90 us
>> 118 MKNOD
>> 0.00 134.63 us 28.08 us 36326.43 us
>> 12587 FSTAT
>> 0.00 112.32 us 19.91 us 381241.34 us
>> 21695 STATFS
>> 0.00 307.91 us 28.00 us 292666.83 us
>> 16841 READ
>> 0.00 29625.60 us 239.57 us 239532.26 us
>> 178 MKDIR
>> 0.00 319.22 us 51.19 us 136353.39 us
>> 33445 WRITE
>> 0.01 175.83 us 1.57 us 513635.22 us
>> 87338 OPENDIR
>> 0.01 249.95 us 58.42 us 594774.86 us
>> 85089 SETXATTR
>> 0.01 11001.25 us 119.87 us 612765.03 us
>> 3180 RMDIR
>> 0.04 14240.66 us 203.89 us 1376758.99 us
>> 6067 CREATE
>> 0.05 89.46 us 30.14 us 687961.60 us
>> 1219542 READLINK
>> 0.39 1114.30 us 44.29 us 2412169.89 us
>> 831371 OPEN
>> 0.51 127078.45 us 78.55 us 694364.09 us
>> 9493 FXATTROP
>> 0.63 399.16 us 8.43 us 3815689.36 us
>> 3764714 ENTRYLK
>> 0.86 91973.00 us 19.89 us 24258016.33 us
>> 22253 READDIR
>> 1.06 6913.26 us 11.58 us 36968532.72 us
>> 364963 GETXATTR
>> 1.29 62121.95 us 36.89 us 8649871.11 us
>> 49472 READDIRP
>> 2.89 1477.58 us 11.10 us 2807637.64 us
>> 4650500 INODELK
>> 4.12 23681.29 us 20.98 us 23291414.50 us
>> 414352 UNLINK
>> 16.09 5000.02 us 9.04 us 23866016.88 us
>> 7655621 LOOKUP
>> 72.04 68956.14 us 68.40 us 27542499.65 us
>> 2486126 XATTROP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2433405 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2453 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 69 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1544 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2430885 CI_FORGET
>>
>> Duration: 209524 seconds
>> Data Read: 2051388524 bytes
>> Data Written: 2208261499 bytes
>>
>> Brick: flexo:/mnt/gluster/b
>> --------------------------------------
>> Cumulative Stats:
>> Block Size: 1b+
>> 8b+ 16b+
>> No. of Reads: 0
>> 0 0
>> No. of Writes: 117
>> 195 367
>>
>> Block Size: 32b+
>> 64b+ 128b+
>> No. of Reads: 130
>> 0 3
>> No. of Writes: 1012
>> 9 134
>>
>> Block Size: 256b+
>> 512b+ 1024b+
>> No. of Reads: 47
>> 8 257
>> No. of Writes: 729
>> 73 168
>>
>> Block Size: 2048b+
>> 4096b+ 8192b+
>> No. of Reads: 10
>> 27 240
>> No. of Writes: 294
>> 466 10472
>>
>> Block Size: 16384b+
>> 32768b+ 65536b+
>> No. of Reads: 139
>> 119 135
>> No. of Writes: 1480
>> 6273 3935
>>
>> Block Size: 131072b+
>> No. of Reads: 7892
>> No. of Writes: 12327
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 955058 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1450749 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 81377 RELEASEDIR
>> 0.00 15.55 us 5.35 us 31.88 us
>> 108 IPC
>> 0.00 125.85 us 59.77 us 224.15 us
>> 44 FTRUNCATE
>> 0.00 198.38 us 81.68 us 1332.45 us
>> 30 REMOVEXATTR
>> 0.00 4254.16 us 621.52 us 10602.99 us
>> 4 FSYNC
>> 0.00 795.08 us 190.30 us 16518.64 us
>> 30 SYMLINK
>> 0.00 195.90 us 111.02 us 953.39 us
>> 137 LINK
>> 0.00 286.07 us 179.64 us 437.82 us
>> 119 MKNOD
>> 0.00 162.61 us 65.62 us 3864.77 us
>> 221 SETXATTR
>> 0.00 117.29 us 48.50 us 317.17 us
>> 354 STAT
>> 0.00 70.35 us 23.10 us 317.32 us
>> 777 LK
>> 0.00 461.72 us 114.54 us 16013.81 us
>> 188 RENAME
>> 0.00 150.40 us 75.59 us 749.54 us
>> 810 FGETXATTR
>> 0.00 267.80 us 99.67 us 994.58 us
>> 698 TRUNCATE
>> 0.00 1176.00 us 194.89 us 78261.12 us
>> 178 MKDIR
>> 0.00 439.83 us 54.38 us 75270.51 us
>> 1665 SETATTR
>> 0.00 65.89 us 17.39 us 354.77 us
>> 11372 FINODELK
>> 0.00 131.57 us 32.68 us 2107.52 us
>> 7398 FSTAT
>> 0.00 120.71 us 31.43 us 847.97 us
>> 9094 READ
>> 0.00 60.75 us 7.25 us 22520.98 us
>> 22614 FLUSH
>> 0.00 73.03 us 22.41 us 750.17 us
>> 21656 STATFS
>> 0.00 323.63 us 166.56 us 23034.44 us
>> 6145 CREATE
>> 0.00 217.55 us 50.60 us 30734.99 us
>> 38051 WRITE
>> 0.01 146.39 us 1.80 us 318612.45 us
>> 81377 OPENDIR
>> 0.01 5641.37 us 108.84 us 468960.33 us
>> 3177 RMDIR
>> 0.03 100.05 us 30.93 us 49756.93 us
>> 432439 READLINK
>> 0.13 483.47 us 16.81 us 2608013.14 us
>> 448381 UNLINK
>> 0.21 7101.56 us 22.26 us 2554390.49 us
>> 52185 READDIRP
>> 0.35 414.05 us 36.58 us 1038229.93 us
>> 1444606 OPEN
>> 0.49 78518.02 us 73.39 us 373999.31 us
>> 10745 FXATTROP
>> 0.72 776.21 us 11.24 us 4271792.57 us
>> 1605822 GETXATTR
>> 0.77 97829.38 us 18.84 us 9477446.11 us
>> 13532 READDIR
>> 0.79 320.44 us 7.71 us 2369511.06 us
>> 4257297 ENTRYLK
>> 1.97 464.02 us 9.38 us 1038460.80 us
>> 7333722 INODELK
>> 13.73 2372.50 us 8.33 us 6258159.87 us
>> 9979799 LOOKUP
>> 80.79 42372.04 us 61.49 us 2238308.13 us
>> 3289103 XATTROP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2250268 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 3756 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 167 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1908 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2246352 CI_FORGET
>>
>> Duration: 209334 seconds
>> Data Read: 1059777395 bytes
>> Data Written: 2336921014 bytes
>>
>> Interval 0 Stats:
>> Block Size: 1b+
>> 8b+ 16b+
>> No. of Reads: 0
>> 0 0
>> No. of Writes: 117
>> 195 367
>>
>> Block Size: 32b+
>> 64b+ 128b+
>> No. of Reads: 130
>> 0 3
>> No. of Writes: 1012
>> 9 134
>>
>> Block Size: 256b+
>> 512b+ 1024b+
>> No. of Reads: 47
>> 8 257
>> No. of Writes: 729
>> 73 168
>>
>> Block Size: 2048b+
>> 4096b+ 8192b+
>> No. of Reads: 10
>> 27 240
>> No. of Writes: 294
>> 466 10472
>>
>> Block Size: 16384b+
>> 32768b+ 65536b+
>> No. of Reads: 139
>> 119 135
>> No. of Writes: 1480
>> 6273 3935
>>
>> Block Size: 131072b+
>> No. of Reads: 7892
>> No. of Writes: 12327
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 955058 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1450749 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 81377 RELEASEDIR
>> 0.00 15.55 us 5.35 us 31.88 us
>> 108 IPC
>> 0.00 125.85 us 59.77 us 224.15 us
>> 44 FTRUNCATE
>> 0.00 198.38 us 81.68 us 1332.45 us
>> 30 REMOVEXATTR
>> 0.00 4254.16 us 621.52 us 10602.99 us
>> 4 FSYNC
>> 0.00 795.08 us 190.30 us 16518.64 us
>> 30 SYMLINK
>> 0.00 195.90 us 111.02 us 953.39 us
>> 137 LINK
>> 0.00 286.07 us 179.64 us 437.82 us
>> 119 MKNOD
>> 0.00 162.61 us 65.62 us 3864.77 us
>> 221 SETXATTR
>> 0.00 117.29 us 48.50 us 317.17 us
>> 354 STAT
>> 0.00 70.35 us 23.10 us 317.32 us
>> 777 LK
>> 0.00 461.72 us 114.54 us 16013.81 us
>> 188 RENAME
>> 0.00 150.43 us 75.59 us 749.54 us
>> 810 FGETXATTR
>> 0.00 267.80 us 99.67 us 994.58 us
>> 698 TRUNCATE
>> 0.00 1176.00 us 194.89 us 78261.12 us
>> 178 MKDIR
>> 0.00 439.83 us 54.38 us 75270.51 us
>> 1665 SETATTR
>> 0.00 65.89 us 17.39 us 354.77 us
>> 11372 FINODELK
>> 0.00 131.57 us 32.68 us 2107.52 us
>> 7398 FSTAT
>> 0.00 120.74 us 31.43 us 847.97 us
>> 9094 READ
>> 0.00 60.75 us 7.25 us 22520.98 us
>> 22614 FLUSH
>> 0.00 73.03 us 22.41 us 750.17 us
>> 21656 STATFS
>> 0.00 323.63 us 166.56 us 23034.44 us
>> 6145 CREATE
>> 0.00 217.50 us 50.60 us 30734.99 us
>> 38051 WRITE
>> 0.01 146.39 us 1.80 us 318612.45 us
>> 81377 OPENDIR
>> 0.01 5641.37 us 108.84 us 468960.33 us
>> 3177 RMDIR
>> 0.03 100.06 us 30.93 us 49756.93 us
>> 432439 READLINK
>> 0.13 483.47 us 16.81 us 2608013.14 us
>> 448381 UNLINK
>> 0.21 7101.56 us 22.26 us 2554390.49 us
>> 52185 READDIRP
>> 0.35 414.01 us 36.58 us 1038229.93 us
>> 1444606 OPEN
>> 0.49 78518.02 us 73.39 us 373999.31 us
>> 10745 FXATTROP
>> 0.72 776.24 us 11.24 us 4271792.57 us
>> 1605822 GETXATTR
>> 0.77 97829.38 us 18.84 us 9477446.11 us
>> 13532 READDIR
>> 0.79 320.45 us 7.71 us 2369511.06 us
>> 4257297 ENTRYLK
>> 1.97 464.07 us 9.38 us 1038460.80 us
>> 7333722 INODELK
>> 13.72 2372.42 us 8.33 us 6258159.87 us
>> 9979799 LOOKUP
>> 80.79 42373.50 us 61.49 us 2238308.13 us
>> 3289103 XATTROP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2250268 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 3756 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 167 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1908 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 2246352 CI_FORGET
>>
>> Duration: 209334 seconds
>> Data Read: 1059777395 bytes
>> Data Written: 2336921014 bytes
>>
>> Brick: flexo:/mnt/gluster/c
>> --------------------------------------
>> Cumulative Stats:
>> Block Size: 1b+
>> No. of Reads: 0
>> No. of Writes: 32843
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 686562 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 834596 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 93757 RELEASEDIR
>> 0.00 18.50 us 8.21 us 50.20 us
>> 107 IPC
>> 0.00 205.83 us 126.30 us 412.06 us
>> 36 LINK
>> 0.00 167.33 us 75.01 us 272.47 us
>> 65 FGETXATTR
>> 0.00 203.58 us 117.35 us 648.28 us
>> 88 RENAME
>> 0.00 203.20 us 66.60 us 809.96 us
>> 153 TRUNCATE
>> 0.00 67.06 us 23.64 us 555.59 us
>> 491 LK
>> 0.01 51.73 us 7.70 us 1174.64 us
>> 9339 FLUSH
>> 0.01 62.41 us 15.13 us 636.35 us
>> 9011 FINODELK
>> 0.03 126.23 us 43.15 us 4760.14 us
>> 10824 FSTAT
>> 0.03 571.94 us 94.85 us 51697.03 us
>> 2594 MKDIR
>> 0.05 264.87 us 81.58 us 53152.06 us
>> 9009 FXATTROP
>> 0.05 423.05 us 38.26 us 548686.55 us
>> 5754 CREATE
>> 0.05 78.14 us 19.17 us 678.32 us
>> 32843 WRITE
>> 0.06 526.39 us 130.84 us 41405.46 us
>> 5798 SYMLINK
>> 0.10 133.73 us 16.25 us 676494.72 us
>> 34904 READDIR
>> 0.12 2054.48 us 99.28 us 88061.83 us
>> 2732 RMDIR
>> 0.21 110.60 us 1.19 us 16833.78 us
>> 93757 OPENDIR
>> 0.72 205.89 us 33.28 us 261023.99 us
>> 170292 UNLINK
>> 0.85 2971.45 us 21.24 us 257839.19 us
>> 13840 READDIRP
>> 1.94 200.88 us 25.49 us 461654.32 us
>> 468842 FTRUNCATE
>> 2.13 212.44 us 54.02 us 461878.34 us
>> 486882 REMOVEXATTR
>> 2.27 223.01 us 52.03 us 178897.40 us
>> 493240 SETXATTR
>> 3.53 155.31 us 7.96 us 67873.00 us
>> 1101338 GETXATTR
>> 3.63 212.32 us 44.42 us 551264.02 us
>> 829002 OPEN
>> 5.77 76.36 us 7.76 us 962634.73 us
>> 3660425 ENTRYLK
>> 6.23 304.93 us 42.51 us 661394.43 us
>> 990644 SETATTR
>> 7.81 382.20 us 107.42 us 846502.78 us
>> 990766 XATTROP
>> 12.39 549.11 us 135.57 us 1659697.83 us
>> 1093387 MKNOD
>> 12.47 139.78 us 9.76 us 552067.56 us
>> 4325281 INODELK
>> 39.53 206.94 us 8.56 us 964823.33 us
>> 9259429 LOOKUP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1856377 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 933300 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 251 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 293 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 923077 CI_FORGET
>>
>> Duration: 209334 seconds
>> Data Read: 0 bytes
>> Data Written: 32843 bytes
>>
>> Interval 0 Stats:
>> Block Size: 1b+
>> No. of Reads: 0
>> No. of Writes: 32843
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 686562 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 834596 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 93757 RELEASEDIR
>> 0.00 18.50 us 8.21 us 50.20 us
>> 107 IPC
>> 0.00 205.83 us 126.30 us 412.06 us
>> 36 LINK
>> 0.00 167.33 us 75.01 us 272.47 us
>> 65 FGETXATTR
>> 0.00 203.58 us 117.35 us 648.28 us
>> 88 RENAME
>> 0.00 203.20 us 66.60 us 809.96 us
>> 153 TRUNCATE
>> 0.00 67.06 us 23.64 us 555.59 us
>> 491 LK
>> 0.01 51.71 us 7.70 us 1174.64 us
>> 9339 FLUSH
>> 0.01 62.41 us 15.13 us 636.35 us
>> 9011 FINODELK
>> 0.03 126.23 us 43.15 us 4760.14 us
>> 10824 FSTAT
>> 0.03 571.94 us 94.85 us 51697.03 us
>> 2594 MKDIR
>> 0.05 264.87 us 81.58 us 53152.06 us
>> 9009 FXATTROP
>> 0.05 423.05 us 38.26 us 548686.55 us
>> 5754 CREATE
>> 0.05 78.14 us 19.17 us 678.32 us
>> 32843 WRITE
>> 0.06 526.39 us 130.84 us 41405.46 us
>> 5798 SYMLINK
>> 0.10 133.73 us 16.25 us 676494.72 us
>> 34904 READDIR
>> 0.12 2054.48 us 99.28 us 88061.83 us
>> 2732 RMDIR
>> 0.21 110.60 us 1.19 us 16833.78 us
>> 93757 OPENDIR
>> 0.72 205.89 us 33.28 us 261023.99 us
>> 170292 UNLINK
>> 0.85 2971.45 us 21.24 us 257839.19 us
>> 13840 READDIRP
>> 1.94 200.87 us 25.49 us 461654.32 us
>> 468842 FTRUNCATE
>> 2.13 212.44 us 54.02 us 461878.34 us
>> 486882 REMOVEXATTR
>> 2.27 223.00 us 52.03 us 178897.40 us
>> 493240 SETXATTR
>> 3.53 155.32 us 7.96 us 67873.00 us
>> 1101338 GETXATTR
>> 3.63 212.32 us 44.42 us 551264.02 us
>> 829002 OPEN
>> 5.77 76.36 us 7.76 us 962634.73 us
>> 3660425 ENTRYLK
>> 6.23 304.88 us 42.51 us 661394.43 us
>> 990644 SETATTR
>> 7.81 382.20 us 107.42 us 846502.78 us
>> 990766 XATTROP
>> 12.39 549.25 us 135.57 us 1659697.83 us
>> 1093387 MKNOD
>> 12.47 139.78 us 9.76 us 552067.56 us
>> 4325281 INODELK
>> 39.53 206.95 us 8.56 us 964823.33 us
>> 9259429 LOOKUP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1856377 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 933300 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 251 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 293 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 923077 CI_FORGET
>>
>> Duration: 209334 seconds
>> Data Read: 0 bytes
>> Data Written: 32843 bytes
>>
>> Brick: bender:/mnt/gluster/c
>> ---------------------------------------
>> Cumulative Stats:
>> Block Size: 1b+
>> No. of Reads: 0
>> No. of Writes: 37050
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 928119 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1401524 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 70092 RELEASEDIR
>> 0.00 19.49 us 8.28 us 180.61 us
>> 100 IPC
>> 0.00 238.88 us 163.97 us 486.93 us
>> 38 LINK
>> 0.00 306.43 us 196.67 us 788.36 us
>> 30 SYMLINK
>> 0.00 2670.18 us 765.93 us 4151.64 us
>> 4 FSYNC
>> 0.00 350.65 us 141.92 us 10573.64 us
>> 122 RENAME
>> 0.00 67.02 us 20.66 us 453.90 us
>> 777 LK
>> 0.00 158.32 us 64.22 us 532.21 us
>> 810 FGETXATTR
>> 0.00 229.87 us 66.47 us 826.31 us
>> 603 TRUNCATE
>> 0.01 1742.85 us 202.81 us 47471.71 us
>> 302 MKDIR
>> 0.01 59.05 us 15.40 us 1813.56 us
>> 10060 FINODELK
>> 0.01 118.54 us 36.65 us 4002.58 us
>> 6409 FSTAT
>> 0.01 49.38 us 8.08 us 1264.41 us
>> 21719 FLUSH
>> 0.03 74.76 us 20.86 us 2008.41 us
>> 37050 WRITE
>> 0.04 553.12 us 40.81 us 386485.29 us
>> 5901 CREATE
>> 0.06 534.01 us 100.86 us 527770.44 us
>> 10057 FXATTROP
>> 0.12 138.96 us 1.00 us 601208.70 us
>> 70092 OPENDIR
>> 0.12 947.76 us 18.54 us 712568.33 us
>> 10779 READDIR
>> 0.15 4099.07 us 124.69 us 421939.47 us
>> 3008 RMDIR
>> 0.61 465.30 us 20.67 us 936959.86 us
>> 109478 UNLINK
>> 1.57 182.41 us 49.54 us 73938.09 us
>> 722669 REMOVEXATTR
>> 1.61 171.41 us 6.98 us 83809.88 us
>> 789294 GETXATTR
>> 1.86 217.88 us 26.40 us 556446.19 us
>> 718835 FTRUNCATE
>> 1.89 219.47 us 49.92 us 506767.78 us
>> 722856 SETXATTR
>> 3.02 181.66 us 44.40 us 344656.69 us
>> 1395957 OPEN
>> 5.56 322.08 us 43.37 us 634069.61 us
>> 1450160 SETATTR
>> 5.84 115.88 us 7.67 us 747387.51 us
>> 4233520 ENTRYLK
>> 6.53 378.33 us 122.84 us 766939.14 us
>> 1449601 XATTROP
>> 9.75 115.48 us 10.70 us 556675.13 us
>> 7097226 INODELK
>> 19.37 1104.98 us 101.28 us 1612346.75 us
>> 1472813 MKNOD
>> 41.85 295.61 us 8.98 us 996805.54 us
>> 11896091 LOOKUP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 3531139 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1862470 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 244 CI_XATTR
>> �� 0.00 0.00 us 0.00 us 0.00 us
>> 632 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1668669 CI_FORGET
>>
>> Duration: 208643 seconds
>> Data Read: 0 bytes
>> Data Written: 37050 bytes
>>
>> Interval 0 Stats:
>> Block Size: 1b+
>> No. of Reads: 0
>> No. of Writes: 37050
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 928119 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1401524 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 70092 RELEASEDIR
>> 0.00 19.49 us 8.28 us 180.61 us
>> 100 IPC
>> 0.00 238.88 us 163.97 us 486.93 us
>> 38 LINK
>> 0.00 306.43 us 196.67 us 788.36 us
>> 30 SYMLINK
>> 0.00 2670.18 us 765.93 us 4151.64 us
>> 4 FSYNC
>> 0.00 350.65 us 141.92 us 10573.64 us
>> 122 RENAME
>> 0.00 67.02 us 20.66 us 453.90 us
>> 777 LK
>> 0.00 158.32 us 64.22 us 532.21 us
>> 810 FGETXATTR
>> 0.00 229.87 us 66.47 us 826.31 us
>> 603 TRUNCATE
>> 0.01 1742.85 us 202.81 us 47471.71 us
>> 302 MKDIR
>> 0.01 59.05 us 15.40 us 1813.56 us
>> 10060 FINODELK
>> 0.01 118.54 us 36.65 us 4002.58 us
>> 6409 FSTAT
>> 0.01 49.39 us 8.08 us 1264.41 us
>> 21719 FLUSH
>> 0.03 74.77 us 20.86 us 2008.41 us
>> 37050 WRITE
>> 0.04 553.12 us 40.81 us 386485.29 us
>> 5901 CREATE
>> 0.06 534.01 us 100.86 us 527770.44 us
>> 10057 FXATTROP
>> 0.12 138.96 us 1.00 us 601208.70 us
>> 70092 OPENDIR
>> 0.12 947.76 us 18.54 us 712568.33 us
>> 10779 READDIR
>> 0.15 4099.07 us 124.69 us 421939.47 us
>> 3008 RMDIR
>> 0.61 465.30 us 20.67 us 936959.86 us
>> 109478 UNLINK
>> 1.57 182.41 us 49.54 us 73938.09 us
>> 722669 REMOVEXATTR
>> 1.61 171.41 us 6.98 us 83809.88 us
>> 789294 GETXATTR
>> 1.86 217.88 us 26.40 us 556446.19 us
>> 718835 FTRUNCATE
>> 1.89 219.48 us 49.92 us 506767.78 us
>> 722856 SETXATTR
>> 3.02 181.66 us 44.40 us 344656.69 us
>> 1395957 OPEN
>> 5.56 322.09 us 43.37 us 634069.61 us
>> 1450160 SETATTR
>> 5.84 115.88 us 7.67 us 747387.51 us
>> 4233520 ENTRYLK
>> 6.53 378.33 us 122.84 us 766939.14 us
>> 1449601 XATTROP
>> 9.75 115.48 us 10.70 us 556675.13 us
>> 7097226 INODELK
>> 19.37 1105.06 us 101.28 us 1612346.75 us
>> 1472813 MKNOD
>> 41.84 295.54 us 8.98 us 996805.54 us
>> 11896091 LOOKUP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 3531139 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1862470 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 244 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 632 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1668669 CI_FORGET
>>
>> Duration: 208643 seconds
>> Data Read: 0 bytes
>> Data Written: 37050 bytes
>>
>> Brick: bender:/mnt/gluster/a
>> ---------------------------------------
>> Cumulative Stats:
>> Block Size: 1b+
>> 2b+ 8b+
>> No. of Reads: 0
>> 0 0
>> No. of Writes: 53
>> 1 60
>>
>> Block Size: 16b+
>> 32b+ 64b+
>> No. of Reads: 0
>> 235 4
>> No. of Writes: 14
>> 1033 15
>>
>> Block Size: 128b+
>> 256b+ 512b+
>> No. of Reads: 4
>> 115 81
>> No. of Writes: 58
>> 703 80
>>
>> Block Size: 1024b+
>> 2048b+ 4096b+
>> No. of Reads: 181
>> 26 74
>> No. of Writes: 163
>> 278 397
>>
>> Block Size: 8192b+
>> 16384b+ 32768b+
>> No. of Reads: 131
>> 161 267
>> No. of Writes: 7264
>> 1398 6742
>>
>> Block Size: 65536b+
>> 131072b+ 262144b+
>> No. of Reads: 443
>> 25528 0
>> No. of Writes: 2426
>> 12240 1
>>
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 684942 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 837092 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 87297 RELEASEDIR
>> 0.00 219.22 us 121.08 us 306.12 us
>> 8 REMOVEXATTR
>> 0.00 21.70 us 7.24 us 196.07 us
>> 97 IPC
>> 0.00 152.77 us 78.90 us 1059.82 us
>> 46 FTRUNCATE
>> 0.00 295.73 us 167.10 us 606.96 us
>> 65 FGETXATTR
>> 0.00 243.66 us 140.50 us 3392.16 us
>> 121 LINK
>> 0.00 1216.47 us 194.37 us 23411.42 us
>> 25 SYMLINK
>> 0.00 327.87 us 234.27 us 551.71 us
>> 107 MKNOD
>> 0.00 36094.61 us 36094.61 us 36094.61 us
>> 1 FSYNC
>> 0.00 219.63 us 147.34 us 783.10 us
>> 188 RENAME
>> 0.00 104.27 us 40.34 us 595.29 us
>> 745 STAT
>> 0.00 531.71 us 113.57 us 53973.08 us
>> 200 TRUNCATE
>> 0.00 289.05 us 63.10 us 70107.41 us
>> 1444 SETATTR
>> 0.00 2430.15 us 232.78 us 68120.54 us
>> 177 MKDIR
>> 0.00 52.68 us 7.28 us 826.01 us
>> 9890 FLUSH
>> 0.00 60.96 us 17.54 us 1393.95 us
>> 9792 FINODELK
>> 0.00 74.26 us 20.00 us 1522.39 us
>> 21521 STATFS
>> 0.00 129.21 us 34.15 us 9068.67 us
>> 13094 FSTAT
>> 0.00 456.99 us 195.50 us 130182.02 us
>> 5967 CREATE
>> 0.00 112.06 us 28.14 us 9545.53 us
>> 27474 READ
>> 0.01 313.76 us 52.98 us 71635.74 us
>> 32926 WRITE
>> 0.01 135.84 us 1.52 us 671373.08 us
>> 87297 OPENDIR
>> 0.01 5225.28 us 131.28 us 951323.69 us
>> 3177 RMDIR
>> 0.01 203.34 us 56.13 us 486756.90 us
>> 85073 SETXATTR
>> 0.03 3521.00 us 145.28 us 3396306.00 us
>> 15090 READDIRP
>> 0.04 113758.50 us 26.72 us 1147074.44 us
>> 564 LK
>> 0.05 92.18 us 29.71 us 39069.99 us
>> 850938 READLINK
>> 0.23 453.79 us 41.84 us 655999.23 us
>> 831212 OPEN
>> 0.41 29425.34 us 19.20 us 1526187.08 us
>> 22482 READDIR
>> 0.50 1954.70 us 21.51 us 1526772.28 us
>> 415226 UNLINK
>> 0.54 1272.83 us 9.66 us 1604368.93 us
>> 686945 GETXATTR
>> 0.55 237.57 us 7.91 us 1277047.91 us
>> 3739718 ENTRYLK
>> 0.60 103888.49 us 89.32 us 761509.35 us
>> 9311 FXATTROP
>> 2.60 909.04 us 9.68 us 2272778.49 us
>> 4649291 INODELK
>> 5.82 1240.20 us 8.78 us 1526868.50 us
>> 7625383 LOOKUP
>> 88.60 58005.98 us 67.31 us 1448810.06 us
>> 2481914 XATTROP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1396875 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1854 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 28 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1051 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1394996 CI_FORGET
>>
>> Duration: 208643 seconds
>> Data Read: 3409342848 bytes
>> Data Written: 2180686507 bytes
>>
>> Interval 0 Stats:
>> Block Size: 1b+
>> 2b+ 8b+
>> No. of Reads: 0
>> 0 0
>> No. of Writes: 53
>> 1 60
>>
>> Block Size: 16b+
>> 32b+ 64b+
>> No. of Reads: 0
>> 235 4
>> No. of Writes: 14
>> 1033 15
>>
>> Block Size: 128b+
>> 256b+ 512b+
>> No. of Reads: 4
>> 115 81
>> No. of Writes: 58
>> 703 80
>>
>> Block Size: 1024b+
>> 2048b+ 4096b+
>> No. of Reads: 181
>> 26 74
>> No. of Writes: 163
>> 278 397
>>
>> Block Size: 8192b+
>> 16384b+ 32768b+
>> No. of Reads: 131
>> 161 267
>> No. of Writes: 7264
>> 1398 6742
>>
>> Block Size: 65536b+
>> 131072b+ 262144b+
>> No. of Reads: 443
>> 25528 0
>> No. of Writes: 2426
>> 12240 1
>>
>> %-latency Avg-latency Min-Latency Max-Latency No. of
>> calls Fop
>> --------- ----------- ----------- -----------
>> ------------ ----
>> 0.00 0.00 us 0.00 us 0.00 us
>> 684942 FORGET
>> 0.00 0.00 us 0.00 us 0.00 us
>> 837092 RELEASE
>> 0.00 0.00 us 0.00 us 0.00 us
>> 87297 RELEASEDIR
>> 0.00 219.22 us 121.08 us 306.12 us
>> 8 REMOVEXATTR
>> 0.00 21.70 us 7.24 us 196.07 us
>> 97 IPC
>> 0.00 152.77 us 78.90 us 1059.82 us
>> 46 FTRUNCATE
>> 0.00 295.73 us 167.10 us 606.96 us
>> 65 FGETXATTR
>> 0.00 243.66 us 140.50 us 3392.16 us
>> 121 LINK
>> 0.00 1216.47 us 194.37 us 23411.42 us
>> 25 SYMLINK
>> 0.00 327.87 us 234.27 us 551.71 us
>> 107 MKNOD
>> 0.00 36094.61 us 36094.61 us 36094.61 us
>> 1 FSYNC
>> 0.00 219.63 us 147.34 us 783.10 us
>> 188 RENAME
>> 0.00 104.27 us 40.34 us 595.29 us
>> 745 STAT
>> 0.00 531.71 us 113.57 us 53973.08 us
>> 200 TRUNCATE
>> 0.00 289.05 us 63.10 us 70107.41 us
>> 1444 SETATTR
>> 0.00 2430.15 us 232.78 us 68120.54 us
>> 177 MKDIR
>> 0.00 52.69 us 7.28 us 826.01 us
>> 9890 FLUSH
>> 0.00 60.96 us 17.54 us 1393.95 us
>> 9792 FINODELK
>> 0.00 74.25 us 20.00 us 1522.39 us
>> 21521 STATFS
>> 0.00 129.23 us 34.15 us 9068.67 us
>> 13094 FSTAT
>> 0.00 456.99 us 195.50 us 130182.02 us
>> 5967 CREATE
>> 0.00 112.45 us 28.14 us 9545.53 us
>> 27474 READ
>> 0.01 313.77 us 52.98 us 71635.74 us
>> 32926 WRITE
>> 0.01 135.84 us 1.52 us 671373.08 us
>> 87297 OPENDIR
>> 0.01 5225.28 us 131.28 us 951323.69 us
>> 3177 RMDIR
>> 0.01 203.34 us 56.13 us 486756.90 us
>> 85073 SETXATTR
>> 0.03 3521.00 us 145.28 us 3396306.00 us
>> 15090 READDIRP
>> 0.04 113758.50 us 26.72 us 1147074.44 us
>> 564 LK
>> 0.05 92.18 us 29.71 us 39069.99 us
>> 850938 READLINK
>> 0.23 453.79 us 41.84 us 655999.23 us
>> 831212 OPEN
>> 0.41 29425.34 us 19.20 us 1526187.08 us
>> 22482 READDIR
>> 0.50 1954.70 us 21.51 us 1526772.28 us
>> 415226 UNLINK
>> 0.54 1272.79 us 9.66 us 1604368.93 us
>> 686945 GETXATTR
>> 0.55 237.57 us 7.91 us 1277047.91 us
>> 3739718 ENTRYLK
>> 0.60 103888.49 us 89.32 us 761509.35 us
>> 9311 FXATTROP
>> 2.60 909.04 us 9.68 us 2272778.49 us
>> 4649291 INODELK
>> 5.82 1240.10 us 8.78 us 1526868.50 us
>> 7625383 LOOKUP
>> 88.60 58006.02 us 67.31 us 1448810.06 us
>> 2481914 XATTROP
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1396875 UPCALL
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1854 CI_IATT
>> 0.00 0.00 us 0.00 us 0.00 us
>> 28 CI_XATTR
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1051 CI_UNLINK
>> 0.00 0.00 us 0.00 us 0.00 us
>> 1394996 CI_FORGET
>>
>> Duration: 208643 seconds
>> Data Read: 3409342848 bytes
>> Data Written: 2180686507 bytes
>>
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200507/0ddc505a/attachment.html>
More information about the Gluster-users
mailing list