[Gluster-users] Possible SHD stalling
Ravishankar N
ravishankar at redhat.com
Fri May 8 04:32:33 UTC 2020
On 08/05/20 1:10 am, Jaco Kroon wrote:
>
> Hi,
>
> So upgrading to 7.5 made a difference in that it takes a bit longer to
> cause chaos, but it still does the exact same thing.
>
> To be precise:
>
> I've now had to set:
>
> gluster volume set vpbx_shared cluster.heal-wait-queue-length 1
> gluster volume set vpbx_shared cluster.shd-wait-qlength 1
> gluster volume set vpbx_shared cluster.shd-max-threads 1
>
> With any of these settings >1 I end up in a situation where simple
> directory listings of < 100 entries in a folder takes upwards of a
> minute to complete *from time to time*.
>
Hi Jaco, these options are to be increased only if your hardware is
capable of handling the load.It was contributed initially by facebook as
it worked for them. See
https://bugzilla.redhat.com/show_bug.cgi?id=1221737#c0. So I think this
is expected. Unless you are saying that the `ls` slowness happens even
when there is no self-heal going on, in which case we need to see what
the issue is.
Regards,
Ravi
> It seems that when the logs
>
> With these the maximum stall I can reproduce is usually around 3
> seconds before the directory listing will complete, but about 5s worst
> case. I've now reverted these as well, which are known for us to
> cause issues under heal required conditions:
>
> cluster.metadata-self-heal on
> cluster.data-self-heal on
> cluster.entry-self-heal on
>
> Not sure if I need to remount for these to become effective but I've
> done so just to be on the safe side.
>
> I'm still seeing these errors:
>
> [2020-05-07 19:20:12.763311] W [MSGID: 114031]
> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
> 0-vpbx_shared-client-1: remote operation failed [Invalid argument]
> [2020-05-07 19:20:12.995675] W [MSGID: 114031]
> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
> 0-vpbx_shared-client-1: remote operation failed [Invalid argument]
> [2020-05-07 19:20:59.199776] W [MSGID: 114031]
> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
> 0-vpbx_shared-client-1: remote operation failed [Invalid argument]
>
> And the start of my issues still seem to co-incide with that. Sample case:
>
> # time ls
> ...
> real 0m5.140s
> user 0m0.009s
> sys 0m0.000s
> # f=(*); echo "${#f[@]}"
> 165
>
> This just does not seem right. Note, again, if I set *any* of the
> above parameters >1 these delays get far, far worse (I've had over a
> minute for this exact same ls).
>
> Any advise would be greatly appreciated. To me, it looks like some
> lock is being taken somewhere, and takes a while to release. I don't
> understand the glusterfs internals nearly well enough to understand
> exactly what's going on here, nor do I know what information to
> provide to help drill down on this. Happy to go look at code, if I
> can get pointed in a direction.
>
> Kind Regards,
> Jaco
>
> On 2020/05/05 08:43, Jaco Kroon wrote:
>>
>> Hi Artem,
>>
>> I'll do that. In the meantime consider this (inferred from the
>> changelogs).
>>
>> It seems like glusterfs SHD is queueing into a bounded queue under
>> lock conditions, and waiting for space to open up. At least, this is
>> what's hinted in the changelogs, I've been trying to grasp the SHD
>> code for two days now and am not getting to wrap my head around
>> that. If this is true, then the only way to avoid lock-ups are to
>> ensure that the queue is larger than the total number of files that
>> can ever require healing (effectively making the queue unbounded).
>>
>> Kind Regards,
>> Jaco
>>
>> On 2020/05/04 21:56, Artem Russakovskii wrote:
>>> I reported something similar recently to this list, and I highly
>>> advise you to update to 7.5 (in my case it was 5.11 -> 5.13), then
>>> see if it keeps happening. The self heal issues should be fixed
>>> there properly at least.
>>>
>>> Sincerely,
>>> Artem
>>>
>>> --
>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>> beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR>
>>>
>>>
>>> On Sat, May 2, 2020 at 2:04 PM Jaco Kroon <jaco at uls.co.za
>>> <mailto:jaco at uls.co.za>> wrote:
>>>
>>> Hi All,
>>>
>>> So we've got a cluster currently running on glusterfs 7.3
>>>
>>> Heal performance is absolutely terrible (it starts of great,
>>> runs for a while then completely trips up).
>>>
>>> Once the SHD trips up, access speeds to the cluster also becomes
>>> poor. And with poor I mean directory listings that normally
>>> takes well under 500ms takes upwards of 10s, but I've seen
>>> upwards of 40s (although a follow up ls immediately after is
>>> nice and fast again, temporarily). With all SHDs killed this
>>> particular ls is as per normal (around 30-50ms).
>>>
>>> The SHD logs doesn't provide any usable clues, the only variance
>>> in logs is this, just before all daemons across the cluster
>>> crawls to a near halt.
>>>
>>> [2020-05-02 19:55:58.627036] I [MSGID: 108026]
>>> [afr-self-heal-common.c:1747:afr_log_selfheal]
>>> 0-vpbx_shared-replicate-0: Completed entry selfheal on
>>> 5b9aa3d8-5085-46e2-9500-145b8cbf9a61. sources= sinks=0 1
>>> [2020-05-02 19:55:58.627058] I [MSGID: 108026]
>>> [afr-self-heal-common.c:1747:afr_log_selfheal]
>>> 0-vpbx_shared-replicate-0: Completed entry selfheal on
>>> c736c9b6-f116-468b-adbc-f102f5b57d01. sources= sinks=0 1
>>> [2020-05-02 19:55:59.125001] W [MSGID: 114031]
>>> [client-rpc-fops_v2.c:1983:client4_0_setattr_cbk]
>>> 0-vpbx_shared-client-1: remote operation failed [Invalid argument]
>>> [2020-05-02 19:55:59.339803] I [MSGID: 108026]
>>> [afr-self-heal-common.c:1747:afr_log_selfheal]
>>> 0-vpbx_shared-replicate-0: Completed entry selfheal on
>>> 2e128eeb-e991-4544-8316-4ddfcb4942da. sources= sinks=0 1
>>> [2020-05-02 19:55:59.490387] I [MSGID: 108026]
>>> [afr-self-heal-metadata.c:51:__afr_selfheal_metadata_do]
>>> 0-vpbx_shared-replicate-0: performing metadata selfheal on
>>> 527a63a3-9acd-4c3f-b95c-5c719325c973
>>>
>>> A few more entries will complete overall and then it'll be a
>>> file every other second. Once I saw this I ran a few more
>>> tests, and on every test just prior to slowing to a crawl, that
>>> entry is there.
>>>
>>> There is a perceived improvement when lowering the thread count,
>>> but not significant enough to confirm a correlation. Even
>>> with**cluster.shd-max-threads=1 this still happens (gets a few
>>> tens thousand files completed instead of few hundred, at 32 it
>>> gets to maybe 400 to 500 before it stalls, 16 does maybe a
>>> thousand).*
>>> *
>>>
>>> There are probably performant bursts hereafter, because from
>>> experience we know this process takes about two months. But
>>> this time around it seems to be much, much worse. Might be
>>> because that was healing a single brick to another brick, so
>>> less locking going? At the moment the replicas are basically
>>> trying to populate the newly added arbiters. Also, we haven't
>>> messed as greatly with the health check settings (queues and
>>> threads and locks nearly as much as since the arbiter addition
>>> because performance has simply dropped to non-acceptable levels.
>>>
>>> Unfortunately granular entry heal isn't on at the moment (nor
>>> can I switch it on due to a few million outstanding heals).
>>> This used to be a 2x2 cluster. We then moved one of the bricks
>>> to a newly added server using replace-brick. What a frightening
>>> process this is. We then proceeded to add the arbiters.
>>>
>>> Some of the settings below was absolute shots in the dark, based
>>> on the little information we've been able to track down so far.
>>>
>>> Kind Regards,
>>> Jaco
>>>
>>> The current gluster volume info (this used to be 2x2
>>> distributed-replicate, aiming to get it to 3x (2 + 1) ... at
>>> current rates we might get there by 2021):
>>>
>>> Volume Name: glvol
>>> Type: Distributed-Replicate
>>> Volume ID: 87cfd38f-d72e-4cc2-b644-9e772d2059bb
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 2 x (2 + 1) = 6
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: zoidberg:/mnt/gluster/a
>>> Brick2: flexo:/mnt/gluster/b
>>> Brick3: bender:/mnt/gluster/c (arbiter)
>>> Brick4: bender:/mnt/gluster/a
>>> Brick5: zoidberg:/mnt/gluster/b
>>> Brick6: flexo:/mnt/gluster/c (arbiter)
>>> Options Reconfigured:
>>> *cluster.heal-timeout: 5 <-- hoping this would at least shorten
>>> the blockages.*
>>> cluster.locking-scheme: granular
>>> cluster.heal-wait-queue-length: 10240
>>> *diagnostics.count-fop-hits: on**
>>> **diagnostics.latency-measurement: on <-- was hoping profiling
>>> might provide some insight. Going to switch this off now.*
>>> cluster.background-self-heal-count: 128
>>> *cluster.shd-wait-qlength: 131072**
>>> **cluster.shd-max-threads: 32 <-- even with this disk IO never
>>> goes over 10% utilization. CPU goes to about 2 cores. So it's
>>> lock contention that's slowing it down we reckon.*
>>> *cluster.metadata-self-heal: off**
>>> **cluster.entry-self-heal: off**
>>> **cluster.data-self-heal: off**<-- these kill performance.
>>> *transport.address-family: inet
>>> nfs.disable: on
>>> performance.client-io-threads: on
>>> features.cache-invalidation: on
>>> features.cache-invalidation-timeout: 600
>>> performance.stat-prefetch: on
>>> performance.cache-invalidation: on
>>> performance.md-cache-timeout: 600
>>> *cluster.self-heal-daemon: yes <-- disabling/enabling this
>>> doesn't help, have to kill the shd and start ... force to get
>>> things moving along again.*
>>> performance.io-thread-count: 16
>>> performance.high-prio-threads: 16
>>> performance.normal-prio-threads: 16
>>> performance.low-prio-threads: 16
>>> performance.least-prio-threads: 4
>>> cluster.self-heal-window-size: 512
>>> cluster.self-heal-readdir-size: 131072
>>> client.event-threads: 8
>>> server.event-threads: 16
>>> network.ping-timeout: 2
>>> *cluster.data-self-heal-algorithm: full <-- most files are small
>>> (<1MB), on dual 1gbit links (so up to 2gbit, but b/w never goes
>>> over 5mbit anyway).*
>>>
>>> performance info:
>>>
>>> Brick: zoidberg:/mnt/gluster/a
>>> -----------------------------------------
>>> Cumulative Stats:
>>> Block Size: 1b+ 8b+ 16b+
>>> No. of Reads: 0 0 0
>>> No. of Writes: 117 195 367
>>>
>>> Block Size: 32b+ 64b+ 128b+
>>> No. of Reads: 306 0 6
>>> No. of Writes: 1016 9 134
>>>
>>> Block Size: 256b+ 512b+ 1024b+
>>> No. of Reads: 107 58 485
>>> No. of Writes: 731 73 168
>>>
>>> Block Size: 2048b+ 4096b+ 8192b+
>>> No. of Reads: 43 59 383
>>> No. of Writes: 295 475 10562
>>>
>>> Block Size: 16384b+ 32768b+ 65536b+
>>> No. of Reads: 347 380 634
>>> No. of Writes: 1487 6369 3970
>>>
>>> Block Size: 131072b+
>>> No. of Reads: 35704
>>> No. of Writes: 12398
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 957293 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1455091 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 69951
>>> RELEASEDIR
>>> 0.00 16.20 us 5.43 us 100.95 us
>>> 113 IPC
>>> 0.00 687.33 us 80.53 us 15414.03 us
>>> 30 REMOVEXATTR
>>> 0.00 840.76 us 72.71 us 37648.39 us
>>> 52 FTRUNCATE
>>> 0.00 138.49 us 36.66 us 25280.24 us
>>> 1385 STAT
>>> 0.00 364.93 us 63.45 us 15957.66 us
>>> 810 FGETXATTR
>>> 0.00 2208.82 us 127.23 us 267092.17 us
>>> 137 LINK
>>> 0.00 37595.33 us 1677.46 us 218313.32 us
>>> 12 FSYNC
>>> 0.00 675.61 us 106.82 us 114351.01 us
>>> 698 TRUNCATE
>>> 0.00 3995.66 us 147.00 us 109544.46 us
>>> 188 RENAME
>>> 0.00 453.11 us 53.37 us 104848.94 us
>>> 1680 SETATTR
>>> 0.00 6004.32 us 229.33 us 151322.58 us
>>> 127 MKNOD
>>> 0.00 68.74 us 16.11 us 61050.93 us
>>> 11412 FINODELK
>>> 0.00 3793.44 us 68.61 us 411523.70 us
>>> 221 SETXATTR
>>> 0.00 28669.99 us 227.28 us 138988.41 us
>>> 30 SYMLINK
>>> 0.00 56.18 us 7.31 us 41484.37 us
>>> 22640 FLUSH
>>> 0.00 105.64 us 19.34 us 351055.72 us
>>> 21693 STATFS
>>> 0.00 23642.90 us 244.19 us 234912.25 us
>>> 178 MKDIR
>>> 0.00 537.38 us 31.73 us 296911.67 us
>>> 9653 FSTAT
>>> 0.00 214.87 us 25.71 us 1712020.99 us
>>> 39054 READ
>>> 0.01 147.08 us 1.47 us 241611.80 us
>>> 69951 OPENDIR
>>> 0.01 3899.84 us 117.22 us 503422.21 us
>>> 3180 RMDIR
>>> 0.01 327.08 us 55.69 us 420376.80 us
>>> 38366 WRITE
>>> 0.02 5580.36 us 210.11 us 1029702.30 us
>>> 6169 CREATE
>>> 0.05 92.69 us 29.94 us 687955.86 us
>>> 1159936 READLINK
>>> 0.40 543.50 us 36.45 us 2430653.54 us
>>> 1449307 OPEN
>>> 0.48 87643.67 us 70.20 us 661999.51 us
>>> 10787 FXATTROP
>>> 0.53 2342.72 us 20.91 us 4157542.43 us
>>> 447821 UNLINK
>>> 0.53 558734.79 us 24.40 us 1399487.63 us
>>> 1889 LK
>>> 0.78 145786.31 us 19.06 us 37490825.73 us
>>> 10687 READDIR
>>> 0.99 2426.23 us 12.47 us 24176123.13 us
>>> 809430 GETXATTR
>>> 1.45 676.83 us 9.30 us 193177067.25 us
>>> 4260021 ENTRYLK
>>> 1.76 38325.38 us 21.44 us 46693575.93 us
>>> 91439 READDIRP
>>> 4.44 1200.31 us 9.10 us 2559377.90 us
>>> 7352811 INODELK
>>> 9.30 1849.24 us 9.28 us 35299050.00 us
>>> 9989221 LOOKUP
>>> 79.24 47791.46 us 76.77 us 10488824.26 us
>>> 3294642 XATTROP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 3194806 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 5311 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 54
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 1907
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 3189457
>>> CI_FORGET
>>>
>>> Duration: 209524 seconds
>>> Data Read: 4775365531 bytes
>>> Data Written: 2354108664 bytes
>>>
>>> Interval 0 Stats:
>>> Block Size: 1b+ 8b+ 16b+
>>> No. of Reads: 0 0 0
>>> No. of Writes: 117 195 367
>>>
>>> Block Size: 32b+ 64b+ 128b+
>>> No. of Reads: 306 0 6
>>> No. of Writes: 1016 9 134
>>>
>>> Block Size: 256b+ 512b+ 1024b+
>>> No. of Reads: 107 58 485
>>> No. of Writes: 731 73 168
>>>
>>> Block Size: 2048b+ 4096b+ 8192b+
>>> No. of Reads: 43 59 383
>>> No. of Writes: 295 475 10562
>>>
>>> Block Size: 16384b+ 32768b+ 65536b+
>>> No. of Reads: 347 380 634
>>> No. of Writes: 1487 6369 3970
>>>
>>> Block Size: 131072b+
>>> No. of Reads: 35704
>>> No. of Writes: 12398
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 957293 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1455091 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 69951
>>> RELEASEDIR
>>> 0.00 16.20 us 5.43 us 100.95 us
>>> 113 IPC
>>> 0.00 687.33 us 80.53 us 15414.03 us
>>> 30 REMOVEXATTR
>>> 0.00 840.76 us 72.71 us 37648.39 us
>>> 52 FTRUNCATE
>>> 0.00 138.49 us 36.66 us 25280.24 us
>>> 1385 STAT
>>> 0.00 363.20 us 63.45 us 15957.66 us
>>> 810 FGETXATTR
>>> 0.00 2208.82 us 127.23 us 267092.17 us
>>> 137 LINK
>>> 0.00 37595.33 us 1677.46 us 218313.32 us
>>> 12 FSYNC
>>> 0.00 675.61 us 106.82 us 114351.01 us
>>> 698 TRUNCATE
>>> 0.00 3995.66 us 147.00 us 109544.46 us
>>> 188 RENAME
>>> 0.00 453.11 us 53.37 us 104848.94 us
>>> 1680 SETATTR
>>> 0.00 6004.32 us 229.33 us 151322.58 us
>>> 127 MKNOD
>>> 0.00 68.74 us 16.11 us 61050.93 us
>>> 11412 FINODELK
>>> 0.00 3793.44 us 68.61 us 411523.70 us
>>> 221 SETXATTR
>>> 0.00 28669.99 us 227.28 us 138988.41 us
>>> 30 SYMLINK
>>> 0.00 56.19 us 7.31 us 41484.37 us
>>> 22640 FLUSH
>>> 0.00 105.61 us 19.34 us 351055.72 us
>>> 21693 STATFS
>>> 0.00 23642.90 us 244.19 us 234912.25 us
>>> 178 MKDIR
>>> 0.00 537.38 us 31.73 us 296911.67 us
>>> 9653 FSTAT
>>> 0.00 214.91 us 25.71 us 1712020.99 us
>>> 39054 READ
>>> 0.01 147.08 us 1.47 us 241611.80 us
>>> 69951 OPENDIR
>>> 0.01 3899.84 us 117.22 us 503422.21 us
>>> 3180 RMDIR
>>> 0.01 327.13 us 55.69 us 420376.80 us
>>> 38366 WRITE
>>> 0.02 5580.36 us 210.11 us 1029702.30 us
>>> 6169 CREATE
>>> 0.05 92.69 us 29.94 us 687955.86 us
>>> 1159936 READLINK
>>> 0.40 543.49 us 36.45 us 2430653.54 us
>>> 1449307 OPEN
>>> 0.48 87643.67 us 70.20 us 661999.51 us
>>> 10787 FXATTROP
>>> 0.53 2342.57 us 20.91 us 4157542.43 us
>>> 447821 UNLINK
>>> 0.53 558416.10 us 24.40 us 1399487.63 us
>>> 1889 LK
>>> 0.78 145786.31 us 19.06 us 37490825.73 us
>>> 10687 READDIR
>>> 0.99 2426.24 us 12.47 us 24176123.13 us
>>> 809430 GETXATTR
>>> 1.45 676.75 us 9.30 us 193177067.25 us
>>> 4260021 ENTRYLK
>>> 1.76 38324.59 us 21.44 us 46693575.93 us
>>> 91439 READDIRP
>>> 4.44 1200.37 us 9.10 us 2559377.90 us
>>> 7352811 INODELK
>>> 9.29 1848.94 us 9.28 us 35299050.00 us
>>> 9989221 LOOKUP
>>> 79.24 47792.42 us 76.77 us 10488824.26 us
>>> 3294642 XATTROP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 3194806 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 5311 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 54
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 1907
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 3189457
>>> CI_FORGET
>>>
>>> Duration: 209524 seconds
>>> Data Read: 4775365531 bytes
>>> Data Written: 2354108664 bytes
>>>
>>> Brick: zoidberg:/mnt/gluster/b
>>> -----------------------------------------
>>> Cumulative Stats:
>>> Block Size: 1b+ 2b+ 8b+
>>> No. of Reads: 0 0 0
>>> No. of Writes: 53 1 60
>>>
>>> Block Size: 16b+ 32b+ 64b+
>>> No. of Reads: 0 213 2
>>> No. of Writes: 14 1057 15
>>>
>>> Block Size: 128b+ 256b+ 512b+
>>> No. of Reads: 0 41 146
>>> No. of Writes: 59 715 80
>>>
>>> Block Size: 1024b+ 2048b+ 4096b+
>>> No. of Reads: 224 9 15
>>> No. of Writes: 163 282 404
>>>
>>> Block Size: 8192b+ 16384b+ 32768b+
>>> No. of Reads: 52 74 210
>>> No. of Writes: 7415 1430 6860
>>>
>>> Block Size: 65536b+ 131072b+ 262144b+
>>> No. of Reads: 299 15313 1
>>> No. of Writes: 2458 12378 1
>>>
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 686733 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 837434 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 87338
>>> RELEASEDIR
>>> 0.00 22.69 us 7.16 us 332.61 us
>>> 116 IPC
>>> 0.00 119.93 us 70.56 us 272.28 us
>>> 57 FTRUNCATE
>>> 0.00 152.25 us 86.82 us 340.63 us
>>> 65 FGETXATTR
>>> 0.00 2336.06 us 90.05 us 20509.81 us
>>> 23 REMOVEXATTR
>>> 0.00 631.26 us 123.17 us 51803.77 us
>>> 121 LINK
>>> 0.00 369.27 us 22.64 us 145090.48 us
>>> 490 LK
>>> 0.00 360.85 us 45.14 us 46846.98 us
>>> 540 STAT
>>> 0.00 1190.96 us 117.40 us 69781.85 us
>>> 200 TRUNCATE
>>> 0.00 245.72 us 54.91 us 123985.76 us
>>> 1482 SETATTR
>>> 0.00 42139.10 us 3313.56 us 312273.21 us
>>> 12 FSYNC
>>> 0.00 55.83 us 8.26 us 49649.35 us
>>> 10047 FLUSH
>>> 0.00 57.52 us 14.62 us 820.59 us
>>> 9977 FINODELK
>>> 0.00 3191.60 us 143.06 us 96214.75 us
>>> 188 RENAME
>>> 0.00 32862.86 us 215.38 us 83851.13 us
>>> 25 SYMLINK
>>> 0.00 11392.03 us 230.70 us 105471.90 us
>>> 118 MKNOD
>>> 0.00 134.63 us 28.08 us 36326.43 us
>>> 12587 FSTAT
>>> 0.00 112.32 us 19.91 us 381241.34 us
>>> 21695 STATFS
>>> 0.00 307.86 us 28.00 us 292666.83 us
>>> 16841 READ
>>> 0.00 29625.60 us 239.57 us 239532.26 us
>>> 178 MKDIR
>>> 0.00 319.24 us 51.19 us 136353.39 us
>>> 33445 WRITE
>>> 0.01 175.83 us 1.57 us 513635.22 us
>>> 87338 OPENDIR
>>> 0.01 249.96 us 58.42 us 594774.86 us
>>> 85089 SETXATTR
>>> 0.01 11001.25 us 119.87 us 612765.03 us
>>> 3180 RMDIR
>>> 0.04 14240.66 us 203.89 us 1376758.99 us
>>> 6067 CREATE
>>> 0.05 89.45 us 30.14 us 687961.60 us
>>> 1219542 READLINK
>>> 0.39 1114.30 us 44.29 us 2412169.89 us
>>> 831371 OPEN
>>> 0.51 127078.45 us 78.55 us 694364.09 us
>>> 9493 FXATTROP
>>> 0.63 399.16 us 8.43 us 3815689.36 us
>>> 3764714 ENTRYLK
>>> 0.86 91973.00 us 19.89 us 24258016.33 us
>>> 22253 READDIR
>>> 1.06 6913.71 us 11.58 us 36968532.72 us
>>> 364963 GETXATTR
>>> 1.29 62130.12 us 36.89 us 8649871.11 us
>>> 49472 READDIRP
>>> 2.89 1477.53 us 11.10 us 2807637.64 us
>>> 4650500 INODELK
>>> 4.12 23681.31 us 20.98 us 23291414.50 us
>>> 414352 UNLINK
>>> 16.09 5000.31 us 9.04 us 23866016.88 us
>>> 7655621 LOOKUP
>>> 72.04 68953.31 us 68.40 us 27542499.65 us
>>> 2486126 XATTROP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 2433405 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 2453 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 69
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 1544
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 2430885
>>> CI_FORGET
>>>
>>> Duration: 209524 seconds
>>> Data Read: 2051388524 bytes
>>> Data Written: 2208261499 bytes
>>>
>>> Interval 0 Stats:
>>> Block Size: 1b+ 2b+ 8b+
>>> No. of Reads: 0 0 0
>>> No. of Writes: 53 1 60
>>>
>>> Block Size: 16b+ 32b+ 64b+
>>> No. of Reads: 0 213 2
>>> No. of Writes: 14 1057 15
>>>
>>> Block Size: 128b+ 256b+ 512b+
>>> No. of Reads: 0 41 146
>>> No. of Writes: 59 715 80
>>>
>>> Block Size: 1024b+ 2048b+ 4096b+
>>> No. of Reads: 224 9 15
>>> No. of Writes: 163 282 404
>>>
>>> Block Size: 8192b+ 16384b+ 32768b+
>>> No. of Reads: 52 74 210
>>> No. of Writes: 7415 1430 6860
>>>
>>> Block Size: 65536b+ 131072b+ 262144b+
>>> No. of Reads: 299 15313 1
>>> No. of Writes: 2458 12378 1
>>>
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 686733 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 837434 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 87338
>>> RELEASEDIR
>>> 0.00 22.69 us 7.16 us 332.61 us
>>> 116 IPC
>>> 0.00 119.93 us 70.56 us 272.28 us
>>> 57 FTRUNCATE
>>> 0.00 152.25 us 86.82 us 340.63 us
>>> 65 FGETXATTR
>>> 0.00 2336.06 us 90.05 us 20509.81 us
>>> 23 REMOVEXATTR
>>> 0.00 631.26 us 123.17 us 51803.77 us
>>> 121 LINK
>>> 0.00 369.27 us 22.64 us 145090.48 us
>>> 490 LK
>>> 0.00 360.85 us 45.14 us 46846.98 us
>>> 540 STAT
>>> 0.00 1190.96 us 117.40 us 69781.85 us
>>> 200 TRUNCATE
>>> 0.00 245.72 us 54.91 us 123985.76 us
>>> 1482 SETATTR
>>> 0.00 42139.10 us 3313.56 us 312273.21 us
>>> 12 FSYNC
>>> 0.00 55.83 us 8.26 us 49649.35 us
>>> 10047 FLUSH
>>> 0.00 57.52 us 14.62 us 820.59 us
>>> 9977 FINODELK
>>> 0.00 3191.60 us 143.06 us 96214.75 us
>>> 188 RENAME
>>> 0.00 32862.86 us 215.38 us 83851.13 us
>>> 25 SYMLINK
>>> 0.00 11392.03 us 230.70 us 105471.90 us
>>> 118 MKNOD
>>> 0.00 134.63 us 28.08 us 36326.43 us
>>> 12587 FSTAT
>>> 0.00 112.32 us 19.91 us 381241.34 us
>>> 21695 STATFS
>>> 0.00 307.91 us 28.00 us 292666.83 us
>>> 16841 READ
>>> 0.00 29625.60 us 239.57 us 239532.26 us
>>> 178 MKDIR
>>> 0.00 319.22 us 51.19 us 136353.39 us
>>> 33445 WRITE
>>> 0.01 175.83 us 1.57 us 513635.22 us
>>> 87338 OPENDIR
>>> 0.01 249.95 us 58.42 us 594774.86 us
>>> 85089 SETXATTR
>>> 0.01 11001.25 us 119.87 us 612765.03 us
>>> 3180 RMDIR
>>> 0.04 14240.66 us 203.89 us 1376758.99 us
>>> 6067 CREATE
>>> 0.05 89.46 us 30.14 us 687961.60 us
>>> 1219542 READLINK
>>> 0.39 1114.30 us 44.29 us 2412169.89 us
>>> 831371 OPEN
>>> 0.51 127078.45 us 78.55 us 694364.09 us
>>> 9493 FXATTROP
>>> 0.63 399.16 us 8.43 us 3815689.36 us
>>> 3764714 ENTRYLK
>>> 0.86 91973.00 us 19.89 us 24258016.33 us
>>> 22253 READDIR
>>> 1.06 6913.26 us 11.58 us 36968532.72 us
>>> 364963 GETXATTR
>>> 1.29 62121.95 us 36.89 us 8649871.11 us
>>> 49472 READDIRP
>>> 2.89 1477.58 us 11.10 us 2807637.64 us
>>> 4650500 INODELK
>>> 4.12 23681.29 us 20.98 us 23291414.50 us
>>> 414352 UNLINK
>>> 16.09 5000.02 us 9.04 us 23866016.88 us
>>> 7655621 LOOKUP
>>> 72.04 68956.14 us 68.40 us 27542499.65 us
>>> 2486126 XATTROP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 2433405 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 2453 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 69
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 1544
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 2430885
>>> CI_FORGET
>>>
>>> Duration: 209524 seconds
>>> Data Read: 2051388524 bytes
>>> Data Written: 2208261499 bytes
>>>
>>> Brick: flexo:/mnt/gluster/b
>>> --------------------------------------
>>> Cumulative Stats:
>>> Block Size: 1b+ 8b+ 16b+
>>> No. of Reads: 0 0 0
>>> No. of Writes: 117 195 367
>>>
>>> Block Size: 32b+ 64b+ 128b+
>>> No. of Reads: 130 0 3
>>> No. of Writes: 1012 9 134
>>>
>>> Block Size: 256b+ 512b+ 1024b+
>>> No. of Reads: 47 8 257
>>> No. of Writes: 729 73 168
>>>
>>> Block Size: 2048b+ 4096b+ 8192b+
>>> No. of Reads: 10 27 240
>>> No. of Writes: 294 466 10472
>>>
>>> Block Size: 16384b+ 32768b+ 65536b+
>>> No. of Reads: 139 119 135
>>> No. of Writes: 1480 6273 3935
>>>
>>> Block Size: 131072b+
>>> No. of Reads: 7892
>>> No. of Writes: 12327
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 955058 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1450749 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 81377
>>> RELEASEDIR
>>> 0.00 15.55 us 5.35 us 31.88 us
>>> 108 IPC
>>> 0.00 125.85 us 59.77 us 224.15 us
>>> 44 FTRUNCATE
>>> 0.00 198.38 us 81.68 us 1332.45 us 30
>>> REMOVEXATTR
>>> 0.00 4254.16 us 621.52 us 10602.99 us
>>> 4 FSYNC
>>> 0.00 795.08 us 190.30 us 16518.64 us
>>> 30 SYMLINK
>>> 0.00 195.90 us 111.02 us 953.39 us
>>> 137 LINK
>>> 0.00 286.07 us 179.64 us 437.82 us
>>> 119 MKNOD
>>> 0.00 162.61 us 65.62 us 3864.77 us
>>> 221 SETXATTR
>>> 0.00 117.29 us 48.50 us 317.17 us
>>> 354 STAT
>>> 0.00 70.35 us 23.10 us 317.32 us
>>> 777 LK
>>> 0.00 461.72 us 114.54 us 16013.81 us
>>> 188 RENAME
>>> 0.00 150.40 us 75.59 us 749.54 us
>>> 810 FGETXATTR
>>> 0.00 267.80 us 99.67 us 994.58 us
>>> 698 TRUNCATE
>>> 0.00 1176.00 us 194.89 us 78261.12 us
>>> 178 MKDIR
>>> 0.00 439.83 us 54.38 us 75270.51 us
>>> 1665 SETATTR
>>> 0.00 65.89 us 17.39 us 354.77 us
>>> 11372 FINODELK
>>> 0.00 131.57 us 32.68 us 2107.52 us
>>> 7398 FSTAT
>>> 0.00 120.71 us 31.43 us 847.97 us
>>> 9094 READ
>>> 0.00 60.75 us 7.25 us 22520.98 us
>>> 22614 FLUSH
>>> 0.00 73.03 us 22.41 us 750.17 us
>>> 21656 STATFS
>>> 0.00 323.63 us 166.56 us 23034.44 us
>>> 6145 CREATE
>>> 0.00 217.55 us 50.60 us 30734.99 us
>>> 38051 WRITE
>>> 0.01 146.39 us 1.80 us 318612.45 us
>>> 81377 OPENDIR
>>> 0.01 5641.37 us 108.84 us 468960.33 us
>>> 3177 RMDIR
>>> 0.03 100.05 us 30.93 us 49756.93 us
>>> 432439 READLINK
>>> 0.13 483.47 us 16.81 us 2608013.14 us
>>> 448381 UNLINK
>>> 0.21 7101.56 us 22.26 us 2554390.49 us
>>> 52185 READDIRP
>>> 0.35 414.05 us 36.58 us 1038229.93 us
>>> 1444606 OPEN
>>> 0.49 78518.02 us 73.39 us 373999.31 us
>>> 10745 FXATTROP
>>> 0.72 776.21 us 11.24 us 4271792.57 us
>>> 1605822 GETXATTR
>>> 0.77 97829.38 us 18.84 us 9477446.11 us
>>> 13532 READDIR
>>> 0.79 320.44 us 7.71 us 2369511.06 us
>>> 4257297 ENTRYLK
>>> 1.97 464.02 us 9.38 us 1038460.80 us
>>> 7333722 INODELK
>>> 13.73 2372.50 us 8.33 us 6258159.87 us
>>> 9979799 LOOKUP
>>> 80.79 42372.04 us 61.49 us 2238308.13 us
>>> 3289103 XATTROP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 2250268 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 3756 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 167
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 1908
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 2246352
>>> CI_FORGET
>>>
>>> Duration: 209334 seconds
>>> Data Read: 1059777395 bytes
>>> Data Written: 2336921014 bytes
>>>
>>> Interval 0 Stats:
>>> Block Size: 1b+ 8b+ 16b+
>>> No. of Reads: 0 0 0
>>> No. of Writes: 117 195 367
>>>
>>> Block Size: 32b+ 64b+ 128b+
>>> No. of Reads: 130 0 3
>>> No. of Writes: 1012 9 134
>>>
>>> Block Size: 256b+ 512b+ 1024b+
>>> No. of Reads: 47 8 257
>>> No. of Writes: 729 73 168
>>>
>>> Block Size: 2048b+ 4096b+ 8192b+
>>> No. of Reads: 10 27 240
>>> No. of Writes: 294 466 10472
>>>
>>> Block Size: 16384b+ 32768b+ 65536b+
>>> No. of Reads: 139 119 135
>>> No. of Writes: 1480 6273 3935
>>>
>>> Block Size: 131072b+
>>> No. of Reads: 7892
>>> No. of Writes: 12327
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 955058 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1450749 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 81377
>>> RELEASEDIR
>>> 0.00 15.55 us 5.35 us 31.88 us
>>> 108 IPC
>>> 0.00 125.85 us 59.77 us 224.15 us
>>> 44 FTRUNCATE
>>> 0.00 198.38 us 81.68 us 1332.45 us 30
>>> REMOVEXATTR
>>> 0.00 4254.16 us 621.52 us 10602.99 us
>>> 4 FSYNC
>>> 0.00 795.08 us 190.30 us 16518.64 us
>>> 30 SYMLINK
>>> 0.00 195.90 us 111.02 us 953.39 us
>>> 137 LINK
>>> 0.00 286.07 us 179.64 us 437.82 us
>>> 119 MKNOD
>>> 0.00 162.61 us 65.62 us 3864.77 us
>>> 221 SETXATTR
>>> 0.00 117.29 us 48.50 us 317.17 us
>>> 354 STAT
>>> 0.00 70.35 us 23.10 us 317.32 us
>>> 777 LK
>>> 0.00 461.72 us 114.54 us 16013.81 us
>>> 188 RENAME
>>> 0.00 150.43 us 75.59 us 749.54 us
>>> 810 FGETXATTR
>>> 0.00 267.80 us 99.67 us 994.58 us
>>> 698 TRUNCATE
>>> 0.00 1176.00 us 194.89 us 78261.12 us
>>> 178 MKDIR
>>> 0.00 439.83 us 54.38 us 75270.51 us
>>> 1665 SETATTR
>>> 0.00 65.89 us 17.39 us 354.77 us
>>> 11372 FINODELK
>>> 0.00 131.57 us 32.68 us 2107.52 us
>>> 7398 FSTAT
>>> 0.00 120.74 us 31.43 us 847.97 us
>>> 9094 READ
>>> 0.00 60.75 us 7.25 us 22520.98 us
>>> 22614 FLUSH
>>> 0.00 73.03 us 22.41 us 750.17 us
>>> 21656 STATFS
>>> 0.00 323.63 us 166.56 us 23034.44 us
>>> 6145 CREATE
>>> 0.00 217.50 us 50.60 us 30734.99 us
>>> 38051 WRITE
>>> 0.01 146.39 us 1.80 us 318612.45 us
>>> 81377 OPENDIR
>>> 0.01 5641.37 us 108.84 us 468960.33 us
>>> 3177 RMDIR
>>> 0.03 100.06 us 30.93 us 49756.93 us
>>> 432439 READLINK
>>> 0.13 483.47 us 16.81 us 2608013.14 us
>>> 448381 UNLINK
>>> 0.21 7101.56 us 22.26 us 2554390.49 us
>>> 52185 READDIRP
>>> 0.35 414.01 us 36.58 us 1038229.93 us
>>> 1444606 OPEN
>>> 0.49 78518.02 us 73.39 us 373999.31 us
>>> 10745 FXATTROP
>>> 0.72 776.24 us 11.24 us 4271792.57 us
>>> 1605822 GETXATTR
>>> 0.77 97829.38 us 18.84 us 9477446.11 us
>>> 13532 READDIR
>>> 0.79 320.45 us 7.71 us 2369511.06 us
>>> 4257297 ENTRYLK
>>> 1.97 464.07 us 9.38 us 1038460.80 us
>>> 7333722 INODELK
>>> 13.72 2372.42 us 8.33 us 6258159.87 us
>>> 9979799 LOOKUP
>>> 80.79 42373.50 us 61.49 us 2238308.13 us
>>> 3289103 XATTROP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 2250268 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 3756 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 167
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 1908
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 2246352
>>> CI_FORGET
>>>
>>> Duration: 209334 seconds
>>> Data Read: 1059777395 bytes
>>> Data Written: 2336921014 bytes
>>>
>>> Brick: flexo:/mnt/gluster/c
>>> --------------------------------------
>>> Cumulative Stats:
>>> Block Size: 1b+
>>> No. of Reads: 0
>>> No. of Writes: 32843
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 686562 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 834596 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 93757
>>> RELEASEDIR
>>> 0.00 18.50 us 8.21 us 50.20 us
>>> 107 IPC
>>> 0.00 205.83 us 126.30 us 412.06 us
>>> 36 LINK
>>> 0.00 167.33 us 75.01 us 272.47 us
>>> 65 FGETXATTR
>>> 0.00 203.58 us 117.35 us 648.28 us
>>> 88 RENAME
>>> 0.00 203.20 us 66.60 us 809.96 us
>>> 153 TRUNCATE
>>> 0.00 67.06 us 23.64 us 555.59 us
>>> 491 LK
>>> 0.01 51.73 us 7.70 us 1174.64 us
>>> 9339 FLUSH
>>> 0.01 62.41 us 15.13 us 636.35 us
>>> 9011 FINODELK
>>> 0.03 126.23 us 43.15 us 4760.14 us
>>> 10824 FSTAT
>>> 0.03 571.94 us 94.85 us 51697.03 us
>>> 2594 MKDIR
>>> 0.05 264.87 us 81.58 us 53152.06 us
>>> 9009 FXATTROP
>>> 0.05 423.05 us 38.26 us 548686.55 us
>>> 5754 CREATE
>>> 0.05 78.14 us 19.17 us 678.32 us
>>> 32843 WRITE
>>> 0.06 526.39 us 130.84 us 41405.46 us
>>> 5798 SYMLINK
>>> 0.10 133.73 us 16.25 us 676494.72 us
>>> 34904 READDIR
>>> 0.12 2054.48 us 99.28 us 88061.83 us
>>> 2732 RMDIR
>>> 0.21 110.60 us 1.19 us 16833.78 us
>>> 93757 OPENDIR
>>> 0.72 205.89 us 33.28 us 261023.99 us
>>> 170292 UNLINK
>>> 0.85 2971.45 us 21.24 us 257839.19 us
>>> 13840 READDIRP
>>> 1.94 200.88 us 25.49 us 461654.32 us
>>> 468842 FTRUNCATE
>>> 2.13 212.44 us 54.02 us 461878.34 us
>>> 486882 REMOVEXATTR
>>> 2.27 223.01 us 52.03 us 178897.40 us
>>> 493240 SETXATTR
>>> 3.53 155.31 us 7.96 us 67873.00 us
>>> 1101338 GETXATTR
>>> 3.63 212.32 us 44.42 us 551264.02 us
>>> 829002 OPEN
>>> 5.77 76.36 us 7.76 us 962634.73 us
>>> 3660425 ENTRYLK
>>> 6.23 304.93 us 42.51 us 661394.43 us
>>> 990644 SETATTR
>>> 7.81 382.20 us 107.42 us 846502.78 us
>>> 990766 XATTROP
>>> 12.39 549.11 us 135.57 us 1659697.83 us
>>> 1093387 MKNOD
>>> 12.47 139.78 us 9.76 us 552067.56 us
>>> 4325281 INODELK
>>> 39.53 206.94 us 8.56 us 964823.33 us
>>> 9259429 LOOKUP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1856377 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 933300 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 251
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 293
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 923077
>>> CI_FORGET
>>>
>>> Duration: 209334 seconds
>>> Data Read: 0 bytes
>>> Data Written: 32843 bytes
>>>
>>> Interval 0 Stats:
>>> Block Size: 1b+
>>> No. of Reads: 0
>>> No. of Writes: 32843
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 686562 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 834596 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 93757
>>> RELEASEDIR
>>> 0.00 18.50 us 8.21 us 50.20 us
>>> 107 IPC
>>> 0.00 205.83 us 126.30 us 412.06 us
>>> 36 LINK
>>> 0.00 167.33 us 75.01 us 272.47 us
>>> 65 FGETXATTR
>>> 0.00 203.58 us 117.35 us 648.28 us
>>> 88 RENAME
>>> 0.00 203.20 us 66.60 us 809.96 us
>>> 153 TRUNCATE
>>> 0.00 67.06 us 23.64 us 555.59 us
>>> 491 LK
>>> 0.01 51.71 us 7.70 us 1174.64 us
>>> 9339 FLUSH
>>> 0.01 62.41 us 15.13 us 636.35 us
>>> 9011 FINODELK
>>> 0.03 126.23 us 43.15 us 4760.14 us
>>> 10824 FSTAT
>>> 0.03 571.94 us 94.85 us 51697.03 us
>>> 2594 MKDIR
>>> 0.05 264.87 us 81.58 us 53152.06 us
>>> 9009 FXATTROP
>>> 0.05 423.05 us 38.26 us 548686.55 us
>>> 5754 CREATE
>>> 0.05 78.14 us 19.17 us 678.32 us
>>> 32843 WRITE
>>> 0.06 526.39 us 130.84 us 41405.46 us
>>> 5798 SYMLINK
>>> 0.10 133.73 us 16.25 us 676494.72 us
>>> 34904 READDIR
>>> 0.12 2054.48 us 99.28 us 88061.83 us
>>> 2732 RMDIR
>>> 0.21 110.60 us 1.19 us 16833.78 us
>>> 93757 OPENDIR
>>> 0.72 205.89 us 33.28 us 261023.99 us
>>> 170292 UNLINK
>>> 0.85 2971.45 us 21.24 us 257839.19 us
>>> 13840 READDIRP
>>> 1.94 200.87 us 25.49 us 461654.32 us
>>> 468842 FTRUNCATE
>>> 2.13 212.44 us 54.02 us 461878.34 us
>>> 486882 REMOVEXATTR
>>> 2.27 223.00 us 52.03 us 178897.40 us
>>> 493240 SETXATTR
>>> 3.53 155.32 us 7.96 us 67873.00 us
>>> 1101338 GETXATTR
>>> 3.63 212.32 us 44.42 us 551264.02 us
>>> 829002 OPEN
>>> 5.77 76.36 us 7.76 us 962634.73 us
>>> 3660425 ENTRYLK
>>> 6.23 304.88 us 42.51 us 661394.43 us
>>> 990644 SETATTR
>>> 7.81 382.20 us 107.42 us 846502.78 us
>>> 990766 XATTROP
>>> 12.39 549.25 us 135.57 us 1659697.83 us
>>> 1093387 MKNOD
>>> 12.47 139.78 us 9.76 us 552067.56 us
>>> 4325281 INODELK
>>> 39.53 206.95 us 8.56 us 964823.33 us
>>> 9259429 LOOKUP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1856377 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 933300 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 251
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 293
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 923077
>>> CI_FORGET
>>>
>>> Duration: 209334 seconds
>>> Data Read: 0 bytes
>>> Data Written: 32843 bytes
>>>
>>> Brick: bender:/mnt/gluster/c
>>> ---------------------------------------
>>> Cumulative Stats:
>>> Block Size: 1b+
>>> No. of Reads: 0
>>> No. of Writes: 37050
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> �� 0.00 0.00 us 0.00 us 0.00 us
>>> 928119 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1401524 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 70092
>>> RELEASEDIR
>>> 0.00 19.49 us 8.28 us 180.61 us
>>> 100 IPC
>>> 0.00 238.88 us 163.97 us 486.93 us
>>> 38 LINK
>>> 0.00 306.43 us 196.67 us 788.36 us
>>> 30 SYMLINK
>>> 0.00 2670.18 us 765.93 us 4151.64 us
>>> 4 FSYNC
>>> 0.00 350.65 us 141.92 us 10573.64 us
>>> 122 RENAME
>>> 0.00 67.02 us 20.66 us 453.90 us
>>> 777 LK
>>> 0.00 158.32 us 64.22 us 532.21 us
>>> 810 FGETXATTR
>>> 0.00 229.87 us 66.47 us 826.31 us
>>> 603 TRUNCATE
>>> 0.01 1742.85 us 202.81 us 47471.71 us
>>> 302 MKDIR
>>> 0.01 59.05 us 15.40 us 1813.56 us
>>> 10060 FINODELK
>>> 0.01 118.54 us 36.65 us 4002.58 us
>>> 6409 FSTAT
>>> 0.01 49.38 us 8.08 us 1264.41 us
>>> 21719 FLUSH
>>> 0.03 74.76 us 20.86 us 2008.41 us
>>> 37050 WRITE
>>> 0.04 553.12 us 40.81 us 386485.29 us
>>> 5901 CREATE
>>> 0.06 534.01 us 100.86 us 527770.44 us
>>> 10057 FXATTROP
>>> 0.12 138.96 us 1.00 us 601208.70 us
>>> 70092 OPENDIR
>>> 0.12 947.76 us 18.54 us 712568.33 us
>>> 10779 READDIR
>>> 0.15 4099.07 us 124.69 us 421939.47 us
>>> 3008 RMDIR
>>> 0.61 465.30 us 20.67 us 936959.86 us
>>> 109478 UNLINK
>>> 1.57 182.41 us 49.54 us 73938.09 us
>>> 722669 REMOVEXATTR
>>> 1.61 171.41 us 6.98 us 83809.88 us
>>> 789294 GETXATTR
>>> 1.86 217.88 us 26.40 us 556446.19 us
>>> 718835 FTRUNCATE
>>> 1.89 219.47 us 49.92 us 506767.78 us
>>> 722856 SETXATTR
>>> 3.02 181.66 us 44.40 us 344656.69 us
>>> 1395957 OPEN
>>> 5.56 322.08 us 43.37 us 634069.61 us
>>> 1450160 SETATTR
>>> 5.84 115.88 us 7.67 us 747387.51 us
>>> 4233520 ENTRYLK
>>> 6.53 378.33 us 122.84 us 766939.14 us
>>> 1449601 XATTROP
>>> 9.75 115.48 us 10.70 us 556675.13 us
>>> 7097226 INODELK
>>> 19.37 1104.98 us 101.28 us 1612346.75 us
>>> 1472813 MKNOD
>>> 41.85 295.61 us 8.98 us 996805.54 us
>>> 11896091 LOOKUP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 3531139 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1862470 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 244
>>> CI_XATTR
>>> �� 0.00 0.00 us 0.00 us 0.00 us 632
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 1668669
>>> CI_FORGET
>>>
>>> Duration: 208643 seconds
>>> Data Read: 0 bytes
>>> Data Written: 37050 bytes
>>>
>>> Interval 0 Stats:
>>> Block Size: 1b+
>>> No. of Reads: 0
>>> No. of Writes: 37050
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 928119 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1401524 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 70092
>>> RELEASEDIR
>>> 0.00 19.49 us 8.28 us 180.61 us
>>> 100 IPC
>>> 0.00 238.88 us 163.97 us 486.93 us
>>> 38 LINK
>>> 0.00 306.43 us 196.67 us 788.36 us
>>> 30 SYMLINK
>>> 0.00 2670.18 us 765.93 us 4151.64 us
>>> 4 FSYNC
>>> 0.00 350.65 us 141.92 us 10573.64 us
>>> 122 RENAME
>>> 0.00 67.02 us 20.66 us 453.90 us
>>> 777 LK
>>> 0.00 158.32 us 64.22 us 532.21 us
>>> 810 FGETXATTR
>>> 0.00 229.87 us 66.47 us 826.31 us
>>> 603 TRUNCATE
>>> 0.01 1742.85 us 202.81 us 47471.71 us
>>> 302 MKDIR
>>> 0.01 59.05 us 15.40 us 1813.56 us
>>> 10060 FINODELK
>>> 0.01 118.54 us 36.65 us 4002.58 us
>>> 6409 FSTAT
>>> 0.01 49.39 us 8.08 us 1264.41 us
>>> 21719 FLUSH
>>> 0.03 74.77 us 20.86 us 2008.41 us
>>> 37050 WRITE
>>> 0.04 553.12 us 40.81 us 386485.29 us
>>> 5901 CREATE
>>> 0.06 534.01 us 100.86 us 527770.44 us
>>> 10057 FXATTROP
>>> 0.12 138.96 us 1.00 us 601208.70 us
>>> 70092 OPENDIR
>>> 0.12 947.76 us 18.54 us 712568.33 us
>>> 10779 READDIR
>>> 0.15 4099.07 us 124.69 us 421939.47 us
>>> 3008 RMDIR
>>> 0.61 465.30 us 20.67 us 936959.86 us
>>> 109478 UNLINK
>>> 1.57 182.41 us 49.54 us 73938.09 us
>>> 722669 REMOVEXATTR
>>> 1.61 171.41 us 6.98 us 83809.88 us
>>> 789294 GETXATTR
>>> 1.86 217.88 us 26.40 us 556446.19 us
>>> 718835 FTRUNCATE
>>> 1.89 219.48 us 49.92 us 506767.78 us
>>> 722856 SETXATTR
>>> 3.02 181.66 us 44.40 us 344656.69 us
>>> 1395957 OPEN
>>> 5.56 322.09 us 43.37 us 634069.61 us
>>> 1450160 SETATTR
>>> 5.84 115.88 us 7.67 us 747387.51 us
>>> 4233520 ENTRYLK
>>> 6.53 378.33 us 122.84 us 766939.14 us
>>> 1449601 XATTROP
>>> 9.75 115.48 us 10.70 us 556675.13 us
>>> 7097226 INODELK
>>> 19.37 1105.06 us 101.28 us 1612346.75 us
>>> 1472813 MKNOD
>>> 41.84 295.54 us 8.98 us 996805.54 us
>>> 11896091 LOOKUP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 3531139 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1862470 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 244
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 632
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 1668669
>>> CI_FORGET
>>>
>>> Duration: 208643 seconds
>>> Data Read: 0 bytes
>>> Data Written: 37050 bytes
>>>
>>> Brick: bender:/mnt/gluster/a
>>> ---------------------------------------
>>> Cumulative Stats:
>>> Block Size: 1b+ 2b+ 8b+
>>> No. of Reads: 0 0 0
>>> No. of Writes: 53 1 60
>>>
>>> Block Size: 16b+ 32b+ 64b+
>>> No. of Reads: 0 235 4
>>> No. of Writes: 14 1033 15
>>>
>>> Block Size: 128b+ 256b+ 512b+
>>> No. of Reads: 4 115 81
>>> No. of Writes: 58 703 80
>>>
>>> Block Size: 1024b+ 2048b+ 4096b+
>>> No. of Reads: 181 26 74
>>> No. of Writes: 163 278 397
>>>
>>> Block Size: 8192b+ 16384b+ 32768b+
>>> No. of Reads: 131 161 267
>>> No. of Writes: 7264 1398 6742
>>>
>>> Block Size: 65536b+ 131072b+ 262144b+
>>> No. of Reads: 443 25528 0
>>> No. of Writes: 2426 12240 1
>>>
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 684942 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 837092 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 87297
>>> RELEASEDIR
>>> 0.00 219.22 us 121.08 us 306.12 us 8
>>> REMOVEXATTR
>>> 0.00 21.70 us 7.24 us 196.07 us
>>> 97 IPC
>>> 0.00 152.77 us 78.90 us 1059.82 us
>>> 46 FTRUNCATE
>>> 0.00 295.73 us 167.10 us 606.96 us
>>> 65 FGETXATTR
>>> 0.00 243.66 us 140.50 us 3392.16 us
>>> 121 LINK
>>> 0.00 1216.47 us 194.37 us 23411.42 us
>>> 25 SYMLINK
>>> 0.00 327.87 us 234.27 us 551.71 us
>>> 107 MKNOD
>>> 0.00 36094.61 us 36094.61 us 36094.61 us
>>> 1 FSYNC
>>> 0.00 219.63 us 147.34 us 783.10 us
>>> 188 RENAME
>>> 0.00 104.27 us 40.34 us 595.29 us
>>> 745 STAT
>>> 0.00 531.71 us 113.57 us 53973.08 us
>>> 200 TRUNCATE
>>> 0.00 289.05 us 63.10 us 70107.41 us
>>> 1444 SETATTR
>>> 0.00 2430.15 us 232.78 us 68120.54 us
>>> 177 MKDIR
>>> 0.00 52.68 us 7.28 us 826.01 us
>>> 9890 FLUSH
>>> 0.00 60.96 us 17.54 us 1393.95 us
>>> 9792 FINODELK
>>> 0.00 74.26 us 20.00 us 1522.39 us
>>> 21521 STATFS
>>> 0.00 129.21 us 34.15 us 9068.67 us
>>> 13094 FSTAT
>>> 0.00 456.99 us 195.50 us 130182.02 us
>>> 5967 CREATE
>>> 0.00 112.06 us 28.14 us 9545.53 us
>>> 27474 READ
>>> 0.01 313.76 us 52.98 us 71635.74 us
>>> 32926 WRITE
>>> 0.01 135.84 us 1.52 us 671373.08 us
>>> 87297 OPENDIR
>>> 0.01 5225.28 us 131.28 us 951323.69 us
>>> 3177 RMDIR
>>> 0.01 203.34 us 56.13 us 486756.90 us
>>> 85073 SETXATTR
>>> 0.03 3521.00 us 145.28 us 3396306.00 us
>>> 15090 READDIRP
>>> 0.04 113758.50 us 26.72 us 1147074.44 us
>>> 564 LK
>>> 0.05 92.18 us 29.71 us 39069.99 us
>>> 850938 READLINK
>>> 0.23 453.79 us 41.84 us 655999.23 us
>>> 831212 OPEN
>>> 0.41 29425.34 us 19.20 us 1526187.08 us
>>> 22482 READDIR
>>> 0.50 1954.70 us 21.51 us 1526772.28 us
>>> 415226 UNLINK
>>> 0.54 1272.83 us 9.66 us 1604368.93 us
>>> 686945 GETXATTR
>>> 0.55 237.57 us 7.91 us 1277047.91 us
>>> 3739718 ENTRYLK
>>> 0.60 103888.49 us 89.32 us 761509.35 us
>>> 9311 FXATTROP
>>> 2.60 909.04 us 9.68 us 2272778.49 us
>>> 4649291 INODELK
>>> 5.82 1240.20 us 8.78 us 1526868.50 us
>>> 7625383 LOOKUP
>>> 88.60 58005.98 us 67.31 us 1448810.06 us
>>> 2481914 XATTROP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1396875 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1854 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 28
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 1051
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 1394996
>>> CI_FORGET
>>>
>>> Duration: 208643 seconds
>>> Data Read: 3409342848 bytes
>>> Data Written: 2180686507 bytes
>>>
>>> Interval 0 Stats:
>>> Block Size: 1b+ 2b+ 8b+
>>> No. of Reads: 0 0 0
>>> No. of Writes: 53 1 60
>>>
>>> Block Size: 16b+ 32b+ 64b+
>>> No. of Reads: 0 235 4
>>> No. of Writes: 14 1033 15
>>>
>>> Block Size: 128b+ 256b+ 512b+
>>> No. of Reads: 4 115 81
>>> No. of Writes: 58 703 80
>>>
>>> Block Size: 1024b+ 2048b+ 4096b+
>>> No. of Reads: 181 26 74
>>> No. of Writes: 163 278 397
>>>
>>> Block Size: 8192b+ 16384b+ 32768b+
>>> No. of Reads: 131 161 267
>>> No. of Writes: 7264 1398 6742
>>>
>>> Block Size: 65536b+ 131072b+ 262144b+
>>> No. of Reads: 443 25528 0
>>> No. of Writes: 2426 12240 1
>>>
>>> %-latency Avg-latency Min-Latency Max-Latency No. of
>>> calls Fop
>>> --------- ----------- ----------- -----------
>>> ------------ ----
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 684942 FORGET
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 837092 RELEASE
>>> 0.00 0.00 us 0.00 us 0.00 us 87297
>>> RELEASEDIR
>>> 0.00 219.22 us 121.08 us 306.12 us 8
>>> REMOVEXATTR
>>> 0.00 21.70 us 7.24 us 196.07 us
>>> 97 IPC
>>> 0.00 152.77 us 78.90 us 1059.82 us
>>> 46 FTRUNCATE
>>> 0.00 295.73 us 167.10 us 606.96 us
>>> 65 FGETXATTR
>>> 0.00 243.66 us 140.50 us 3392.16 us
>>> 121 LINK
>>> 0.00 1216.47 us 194.37 us 23411.42 us
>>> 25 SYMLINK
>>> 0.00 327.87 us 234.27 us 551.71 us
>>> 107 MKNOD
>>> 0.00 36094.61 us 36094.61 us 36094.61 us
>>> 1 FSYNC
>>> 0.00 219.63 us 147.34 us 783.10 us
>>> 188 RENAME
>>> 0.00 104.27 us 40.34 us 595.29 us
>>> 745 STAT
>>> 0.00 531.71 us 113.57 us 53973.08 us
>>> 200 TRUNCATE
>>> 0.00 289.05 us 63.10 us 70107.41 us
>>> 1444 SETATTR
>>> 0.00 2430.15 us 232.78 us 68120.54 us
>>> 177 MKDIR
>>> 0.00 52.69 us 7.28 us 826.01 us
>>> 9890 FLUSH
>>> 0.00 60.96 us 17.54 us 1393.95 us
>>> 9792 FINODELK
>>> 0.00 74.25 us 20.00 us 1522.39 us
>>> 21521 STATFS
>>> 0.00 129.23 us 34.15 us 9068.67 us
>>> 13094 FSTAT
>>> 0.00 456.99 us 195.50 us 130182.02 us
>>> 5967 CREATE
>>> 0.00 112.45 us 28.14 us 9545.53 us
>>> 27474 READ
>>> 0.01 313.77 us 52.98 us 71635.74 us
>>> 32926 WRITE
>>> 0.01 135.84 us 1.52 us 671373.08 us
>>> 87297 OPENDIR
>>> 0.01 5225.28 us 131.28 us 951323.69 us
>>> 3177 RMDIR
>>> 0.01 203.34 us 56.13 us 486756.90 us
>>> 85073 SETXATTR
>>> 0.03 3521.00 us 145.28 us 3396306.00 us
>>> 15090 READDIRP
>>> 0.04 113758.50 us 26.72 us 1147074.44 us
>>> 564 LK
>>> 0.05 92.18 us 29.71 us 39069.99 us
>>> 850938 READLINK
>>> 0.23 453.79 us 41.84 us 655999.23 us
>>> 831212 OPEN
>>> 0.41 29425.34 us 19.20 us 1526187.08 us
>>> 22482 READDIR
>>> 0.50 1954.70 us 21.51 us 1526772.28 us
>>> 415226 UNLINK
>>> 0.54 1272.79 us 9.66 us 1604368.93 us
>>> 686945 GETXATTR
>>> 0.55 237.57 us 7.91 us 1277047.91 us
>>> 3739718 ENTRYLK
>>> 0.60 103888.49 us 89.32 us 761509.35 us
>>> 9311 FXATTROP
>>> 2.60 909.04 us 9.68 us 2272778.49 us
>>> 4649291 INODELK
>>> 5.82 1240.10 us 8.78 us 1526868.50 us
>>> 7625383 LOOKUP
>>> 88.60 58006.02 us 67.31 us 1448810.06 us
>>> 2481914 XATTROP
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1396875 UPCALL
>>> 0.00 0.00 us 0.00 us 0.00 us
>>> 1854 CI_IATT
>>> 0.00 0.00 us 0.00 us 0.00 us 28
>>> CI_XATTR
>>> 0.00 0.00 us 0.00 us 0.00 us 1051
>>> CI_UNLINK
>>> 0.00 0.00 us 0.00 us 0.00 us 1394996
>>> CI_FORGET
>>>
>>> Duration: 208643 seconds
>>> Data Read: 3409342848 bytes
>>> Data Written: 2180686507 bytes
>>>
>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200508/8dc8090e/attachment-0001.html>
More information about the Gluster-users
mailing list