[Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0
David Spisla
spisla80 at gmail.com
Tue Nov 5 15:25:10 UTC 2019
Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC <rkavunga at redhat.com>:
>
> On 11/4/19 8:46 PM, David Spisla wrote:
>
> Dear Gluster Community,
>
> I also have a issue concerning performance. The last days I updated our
> test cluster from GlusterFS v5.5 to v7.0 . The setup in general:
>
> 2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with 2
> Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did
> several tests to ensure that Samba don't causes the fall.
> The setup ist completely the same except the Gluster Version
> Here are my results:
> 64KiB 1MiB 10MiB (Filesize)
> 3,49 47,41 300,50 (Values in MiB/s with
> GlusterFS v5.5)
> 0,16 2,61 76,63 (Values in MiB/s with
> GlusterFS v7.0)
>
>
> Can you please share the profile information [1] for both versions? Also
> it would be really helpful if you can mention the io patterns that used for
> this tests.
>
> [1] :
> https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/
>
Hello Rafi,
thank you for your help.
* First more information about the io patterns: As a client we use a DL360
Windws Server 2017 machine with 10Gbit NIC connected to the storage
machines. The share will be mounted via SMB and the tests writes with fio.
We use this job files (see attachment). Each job file will be executed
separetely and there is a sleep about 60s between each test run to calm
down the system before starting a new test.
* Attached below you find the profile output from the tests with v5.5
(ctime enabled), v7.0 (ctime enabled).
* Beside of the tests with Samba I did also some fio tests directly on the
FUSE Mounts (locally on one of the storage nodes). The results show that
there is only a small decrease of performance between v5.5 and v7.0
(All values in MiB/s)
64KiB 1MiB 10MiB
50,09 679,96 1023,02 (v5.5)
47,00 656,46 977,60 (v7.0)
It seems to be that the combination of samba + gluster7.0 has a lot of
problems, or not?
>
> We use this volume options (GlusterFS 7.0):
>
> Volume Name: archive1
> Type: Distributed-Replicate
> Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x 2 = 4
> Transport-type: tcp
> Bricks:
> Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
> Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
> Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
> Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
> Options Reconfigured:
> performance.client-io-threads: off
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> user.smb: disable
> features.read-only: off
> features.worm: off
> features.worm-file-level: on
> features.retention-mode: enterprise
> features.default-retention-period: 120
> network.ping-timeout: 10
> features.cache-invalidation: on
> features.cache-invalidation-timeout: 600
> performance.nl-cache: on
> performance.nl-cache-timeout: 600
> client.event-threads: 32
> server.event-threads: 32
> cluster.lookup-optimize: on
> performance.stat-prefetch: on
> performance.cache-invalidation: on
> performance.md-cache-timeout: 600
> performance.cache-samba-metadata: on
> performance.cache-ima-xattrs: on
> performance.io-thread-count: 64
> cluster.use-compound-fops: on
> performance.cache-size: 512MB
> performance.cache-refresh-timeout: 10
> performance.read-ahead: off
> performance.write-behind-window-size: 4MB
> performance.write-behind: on
> storage.build-pgfid: on
> features.ctime: on
> cluster.quorum-type: fixed
> cluster.quorum-count: 1
> features.bitrot: on
> features.scrub: Active
> features.scrub-freq: daily
>
> For GlusterFS 5.5 its nearly the same except the fact that there were 2
> options to enable ctime feature.
>
>
>
> Ctime stores additional metadata information as an extended attributes
> which sometimes exceeds the default inode size. In such scenarios the
> additional xattrs won't fit into the default size. This will result in
> additional blocks to be used to store xattrs in the inide, which will
> effect the latency. This is purely based on the i/o operations and the
> total xattrs size stored in the inode.
>
> Is it possible for you to repeat the test by disabling ctime or increasing
> the inode size to a higher value say 1024KB?
>
I will do so but for today I could not finish tests with ctime disabled (or
higher inode value) because it takes a lot of time with v7.0 due to the low
performance and I will perform it tomorrow. As soon as possible I give you
the results.
By the way: You really mean inode size on xfs layer 1024KB? Or do you mean
1024Bytes? We use per default 512Bytes, because this is the recommended
size until now . But it seems to be that there is a need for a new
recommendation when using ctime feature as a default. I can not image that
this is the real cause for the low performance because in v5.5 we also use
ctime feature with inode size 512Bytes.
Regards
David
>
> Our optimization for Samba looks like this (for every version):
>
> [global]
> workgroup = SAMBA
> netbios name = CLUSTER
> kernel share modes = no
> aio read size = 1
> aio write size = 1
> kernel oplocks = no
> max open files = 100000
> nt acl support = no
> security = user
> server min protocol = SMB2
> store dos attributes = no
> strict locking = no
> full_audit:failure = pwrite_send pwrite_recv pwrite offload_write_send
> offload_write_recv create_file open unlink connect disconnect rename chown
> fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
> full_audit:success = pwrite_send pwrite_recv pwrite offload_write_send
> offload_write_recv create_file open unlink connect disconnect rename chown
> fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
> full_audit:facility = local5
> durable handles = yes
> posix locking = no
> log level = 2
> max log size = 100000
> debug pid = yes
>
> What can be the cause for this rapid falling of the performance for small
> files? Are some of our vol options not recommended anymore?
> There were some patches concerning performance for small files in v6.0 und
> v7.0 :
>
> #1670031 <https://bugzilla.redhat.com/1670031>: performance regression
> seen with smallfile workload tests
>
> #1659327 <https://bugzilla.redhat.com/1659327>: 43% regression in
> small-file sequential read performance
>
> And one patch for the io-cache:
>
> #1659869 <https://bugzilla.redhat.com/1659869>: improvements to io-cache
>
> Regards
>
> David Spisla
>
>
> ________
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/118564314
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/118564314
>
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191105/26cfa138/attachment.html>
-------------- next part --------------
Gluster5.5 (Ctime enabled)
==========================
fs-dl380-c1-n1:~ # gluster vo profile repo1 info
Brick: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 13722 125700 15237
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 3236 746 30
Block Size: 4096b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 2 9051 9050
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 7 163 0
No. of Writes: 3034 214018 99
Block Size: 524288b+ 1048576b+ 2097152b+
No. of Reads: 0 0 0
No. of Writes: 431 4898 1732
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 19937 FORGET
0.00 0.00 us 0.00 us 0.00 us 28745 RELEASE
0.00 0.00 us 0.00 us 0.00 us 27033 RELEASEDIR
0.00 372.52 us 372.52 us 372.52 us 1 XATTROP
0.00 155.97 us 148.40 us 163.74 us 3 SETXATTR
0.00 355.85 us 319.36 us 392.34 us 3 MKDIR
0.00 35.10 us 14.17 us 58.11 us 39 IPC
0.00 2092.77 us 759.52 us 2785.95 us 3 RMDIR
0.00 57.69 us 13.35 us 514.30 us 135 READDIR
0.01 894.26 us 34.03 us 56491.66 us 182 READ
0.01 374.15 us 22.21 us 19577.39 us 827 STATFS
0.03 69.78 us 21.13 us 352.94 us 7778 FSTAT
0.03 282.54 us 17.73 us 23688.04 us 2268 FSETXATTR
0.04 60.61 us 14.01 us 10210.17 us 13348 FINODELK
0.04 355.90 us 31.44 us 56158.91 us 2325 OPEN
0.05 306.29 us 109.78 us 674.82 us 3665 SETATTR
0.10 276.25 us 21.09 us 1058896.03 us 7498 INODELK
0.16 507.07 us 181.63 us 5291.74 us 6607 CREATE
0.18 297.21 us 90.56 us 11128.32 us 13316 FXATTROP
0.24 773.47 us 179.92 us 27742.67 us 6606 UNLINK
0.33 265.41 us 12.03 us 62361.64 us 26444 ENTRYLK
0.88 234.65 us 18.24 us 15480.79 us 80510 LOOKUP
0.93 750.11 us 12.26 us 14390.35 us 26600 GETXATTR
1.04 1127.56 us 11.10 us 227020.48 us 19784 FLUSH
1.09 881.86 us 1.22 us 33956.54 us 26502 OPENDIR
4.70 595.48 us 68.68 us 230191.32 us 168825 WRITE
90.14 33977.47 us 84.81 us 129099.85 us 56766 READDIRP
0.00 0.00 us 0.00 us 0.00 us 1958138 UPCALL
0.00 0.00 us 0.00 us 0.00 us 1958010 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 2261 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 128 CI_FORGET
Duration: 6513 seconds
Data Read: 21823488 bytes
Data Written: 38141710505 bytes
Interval 1 Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 13722 125700 15237
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 3236 746 30
Block Size: 4096b+ 65536b+ 131072b+
No. of Reads: 0 7 163
No. of Writes: 2 3034 2
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 55 431 4898
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1732
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 126 FORGET
0.00 0.00 us 0.00 us 0.00 us 8927 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26500 RELEASEDIR
0.00 372.52 us 372.52 us 372.52 us 1 XATTROP
0.00 155.97 us 148.40 us 163.74 us 3 SETXATTR
0.00 355.85 us 319.36 us 392.34 us 3 MKDIR
0.00 36.18 us 14.17 us 58.11 us 34 IPC
0.00 2092.77 us 759.52 us 2785.95 us 3 RMDIR
0.00 57.02 us 13.35 us 514.30 us 127 READDIR
0.01 894.26 us 34.03 us 56491.66 us 182 READ
0.01 380.67 us 22.21 us 19577.39 us 806 STATFS
0.03 69.78 us 21.13 us 352.94 us 7778 FSTAT
0.03 282.54 us 17.73 us 23688.04 us 2268 FSETXATTR
0.04 60.61 us 14.01 us 10210.17 us 13348 FINODELK
0.04 355.90 us 31.44 us 56158.91 us 2325 OPEN
0.05 306.29 us 109.78 us 674.82 us 3665 SETATTR
0.10 276.25 us 21.09 us 1058896.03 us 7498 INODELK
0.16 507.07 us 181.63 us 5291.74 us 6607 CREATE
0.18 297.21 us 90.56 us 11128.32 us 13316 FXATTROP
0.24 773.47 us 179.92 us 27742.67 us 6606 UNLINK
0.33 265.41 us 12.03 us 62361.64 us 26444 ENTRYLK
0.88 234.65 us 18.24 us 15480.79 us 80496 LOOKUP
0.93 750.43 us 12.26 us 14390.35 us 26587 GETXATTR
1.04 1127.56 us 11.10 us 227020.48 us 19784 FLUSH
1.09 882.00 us 1.22 us 33956.54 us 26498 OPENDIR
4.70 595.47 us 68.68 us 230191.32 us 168825 WRITE
90.14 33978.50 us 84.81 us 129099.85 us 56766 READDIRP
0.00 0.00 us 0.00 us 0.00 us 1958138 UPCALL
0.00 0.00 us 0.00 us 0.00 us 1958010 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 2261 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 128 CI_FORGET
Duration: 789 seconds
Data Read: 21823488 bytes
Data Written: 9485570217 bytes
Brick: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 8952 8950 2966
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 0 0 0
No. of Writes: 217821 88 484
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4982 1846
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 20194 FORGET
0.00 0.00 us 0.00 us 0.00 us 26431 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26795 RELEASEDIR
0.00 164.43 us 161.90 us 167.62 us 3 SETXATTR
0.00 30.57 us 17.79 us 44.45 us 21 IPC
0.00 390.62 us 383.31 us 397.93 us 2 XATTROP
0.00 56.79 us 26.70 us 152.48 us 23 FSETXATTR
0.00 447.69 us 400.71 us 510.30 us 3 MKDIR
0.00 129.81 us 55.26 us 289.15 us 28 OPEN
0.00 75.71 us 14.13 us 816.87 us 75 READDIR
0.00 2188.47 us 796.44 us 2993.06 us 3 RMDIR
0.01 160.12 us 21.94 us 6000.03 us 813 STATFS
0.06 70.99 us 24.27 us 29788.04 us 7349 INODELK
0.07 73.26 us 24.65 us 677.95 us 7784 FSTAT
0.09 60.50 us 17.42 us 521.01 us 13194 FINODELK
0.13 312.52 us 117.48 us 565.10 us 3652 SETATTR
0.39 508.69 us 179.00 us 1052.82 us 6594 CREATE
0.44 292.42 us 90.47 us 4546.87 us 13194 FXATTROP
0.77 337.08 us 12.36 us 56472.78 us 19800 FLUSH
0.81 1067.61 us 218.87 us 50116.71 us 6594 UNLINK
1.08 356.53 us 1.54 us 22107.25 us 26472 OPENDIR
1.12 368.18 us 13.26 us 68738.68 us 26406 ENTRYLK
1.19 1003.64 us 166.98 us 6085.26 us 10330 WRITE
2.29 753.23 us 12.07 us 5793.53 us 26507 GETXATTR
2.44 253.59 us 16.41 us 13225.37 us 83900 LOOKUP
89.11 12640.00 us 20.25 us 145809.04 us 61411 READDIRP
0.00 0.00 us 0.00 us 0.00 us 425 UPCALL
0.00 0.00 us 0.00 us 0.00 us 35 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 27 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 390 CI_FORGET
Duration: 6513 seconds
Data Read: 0 bytes
Data Written: 38958530560 bytes
Interval 1 Stats:
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 0 0 0
No. of Writes: 2966 3 49
Block Size: 524288b+ 1048576b+ 2097152b+
No. of Reads: 0 0 0
No. of Writes: 484 4982 1846
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 387 FORGET
0.00 0.00 us 0.00 us 0.00 us 6622 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26472 RELEASEDIR
0.00 164.43 us 161.90 us 167.62 us 3 SETXATTR
0.00 30.40 us 17.79 us 44.45 us 19 IPC
0.00 390.62 us 383.31 us 397.93 us 2 XATTROP
0.00 56.79 us 26.70 us 152.48 us 23 FSETXATTR
0.00 447.69 us 400.71 us 510.30 us 3 MKDIR
0.00 129.81 us 55.26 us 289.15 us 28 OPEN
0.00 77.61 us 14.13 us 816.87 us 71 READDIR
0.00 2188.47 us 796.44 us 2993.06 us 3 RMDIR
0.01 161.55 us 21.94 us 6000.03 us 792 STATFS
0.06 70.99 us 24.27 us 29788.04 us 7349 INODELK
0.07 73.26 us 24.65 us 677.95 us 7784 FSTAT
0.09 60.50 us 17.42 us 521.01 us 13194 FINODELK
0.13 312.52 us 117.48 us 565.10 us 3652 SETATTR
0.39 508.69 us 179.00 us 1052.82 us 6594 CREATE
0.44 292.42 us 90.47 us 4546.87 us 13194 FXATTROP
0.77 337.08 us 12.36 us 56472.78 us 19800 FLUSH
0.81 1067.61 us 218.87 us 50116.71 us 6594 UNLINK
1.08 356.55 us 1.54 us 22107.25 us 26470 OPENDIR
1.12 368.18 us 13.26 us 68738.68 us 26406 ENTRYLK
1.19 1003.64 us 166.98 us 6085.26 us 10330 WRITE
2.29 753.43 us 12.07 us 5793.53 us 26499 GETXATTR
2.44 253.59 us 16.41 us 13225.37 us 83890 LOOKUP
89.11 12640.71 us 20.25 us 145809.04 us 61411 READDIRP
0.00 0.00 us 0.00 us 0.00 us 425 UPCALL
0.00 0.00 us 0.00 us 0.00 us 35 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 27 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 390 CI_FORGET
Duration: 789 seconds
Data Read: 0 bytes
Data Written: 9811918848 bytes
Brick: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 8952 8950 2966
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 24 0 0
No. of Writes: 217821 88 484
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4982 1846
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 26404 FORGET
0.00 0.00 us 0.00 us 0.00 us 26426 RELEASE
0.00 0.00 us 0.00 us 0.00 us 27037 RELEASEDIR
0.00 242.57 us 242.57 us 242.57 us 1 READDIRP
0.00 270.43 us 224.72 us 316.13 us 2 REMOVEXATTR
0.00 376.30 us 357.25 us 395.35 us 2 XATTROP
0.00 411.18 us 363.84 us 462.52 us 3 MKDIR
0.00 36.88 us 25.54 us 54.42 us 36 IPC
0.00 176.85 us 32.37 us 492.36 us 15 FSETXATTR
0.01 206.12 us 108.73 us 412.08 us 21 OPEN
0.01 197.53 us 85.44 us 451.06 us 27 READ
0.01 1995.17 us 760.64 us 2693.43 us 3 RMDIR
0.01 1366.94 us 127.39 us 3611.57 us 5 SETXATTR
0.02 88.63 us 25.71 us 863.15 us 135 READDIR
0.13 116.18 us 34.05 us 2167.66 us 828 STATFS
0.78 78.60 us 26.05 us 52960.99 us 7352 INODELK
1.19 67.15 us 20.02 us 2608.91 us 13194 FINODELK
1.58 59.32 us 18.09 us 2981.27 us 19800 FLUSH
1.64 334.42 us 118.08 us 855.90 us 3654 SETATTR
2.57 72.45 us 19.16 us 732.36 us 26406 ENTRYLK
3.84 108.04 us 2.38 us 677.07 us 26502 OPENDIR
4.56 515.76 us 191.84 us 883.80 us 6594 CREATE
5.32 300.53 us 96.16 us 6531.99 us 13194 FXATTROP
7.63 862.82 us 248.89 us 4093.19 us 6594 UNLINK
14.59 1052.78 us 173.50 us 3653.35 us 10330 WRITE
26.87 753.92 us 21.93 us 5703.54 us 26567 GETXATTR
29.25 276.32 us 32.87 us 3104.49 us 78907 LOOKUP
0.00 0.00 us 0.00 us 0.00 us 6646 UPCALL
0.00 0.00 us 0.00 us 0.00 us 30 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 17 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6616 CI_FORGET
Duration: 6596 seconds
Data Read: 3145728 bytes
Data Written: 38958530560 bytes
Interval 1 Stats:
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 0 24 0
No. of Writes: 2966 3 49
Block Size: 524288b+ 1048576b+ 2097152b+
No. of Reads: 0 0 0
No. of Writes: 484 4982 1846
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6597 FORGET
0.00 0.00 us 0.00 us 0.00 us 6615 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26500 RELEASEDIR
0.00 242.57 us 242.57 us 242.57 us 1 READDIRP
0.00 270.43 us 224.72 us 316.13 us 2 REMOVEXATTR
0.00 376.30 us 357.25 us 395.35 us 2 XATTROP
0.00 411.18 us 363.84 us 462.52 us 3 MKDIR
0.00 37.24 us 25.54 us 54.42 us 34 IPC
0.00 176.85 us 32.37 us 492.36 us 15 FSETXATTR
0.01 206.12 us 108.73 us 412.08 us 21 OPEN
0.01 197.53 us 85.44 us 451.06 us 27 READ
0.01 1995.17 us 760.64 us 2693.43 us 3 RMDIR
0.01 1366.94 us 127.39 us 3611.57 us 5 SETXATTR
0.02 88.84 us 25.71 us 863.15 us 127 READDIR
0.13 116.49 us 34.05 us 2167.66 us 806 STATFS
0.78 78.60 us 26.05 us 52960.99 us 7352 INODELK
1.19 67.15 us 20.02 us 2608.91 us 13194 FINODELK
1.58 59.32 us 18.09 us 2981.27 us 19800 FLUSH
1.64 334.42 us 118.08 us 855.90 us 3654 SETATTR
2.57 72.45 us 19.16 us 732.36 us 26406 ENTRYLK
3.84 108.06 us 2.38 us 677.07 us 26498 OPENDIR
4.56 515.76 us 191.84 us 883.80 us 6594 CREATE
5.32 300.53 us 96.16 us 6531.99 us 13194 FXATTROP
7.63 862.82 us 248.89 us 4093.19 us 6594 UNLINK
14.59 1052.78 us 173.50 us 3653.35 us 10330 WRITE
26.87 754.21 us 21.93 us 5703.54 us 26555 GETXATTR
29.25 276.33 us 32.87 us 3104.49 us 78893 LOOKUP
0.00 0.00 us 0.00 us 0.00 us 6646 UPCALL
0.00 0.00 us 0.00 us 0.00 us 30 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 17 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6616 CI_FORGET
Duration: 789 seconds
Data Read: 3145728 bytes
Data Written: 9811918848 bytes
Brick: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 13722 125700 15237
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 3236 746 30
Block Size: 4096b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 2 9051 9050
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 7 139 0
No. of Writes: 3034 214018 99
Block Size: 524288b+ 1048576b+ 2097152b+
No. of Reads: 0 0 0
No. of Writes: 431 4898 1732
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 26420 FORGET
0.00 0.00 us 0.00 us 0.00 us 28806 RELEASE
0.00 0.00 us 0.00 us 0.00 us 27035 RELEASEDIR
0.00 373.31 us 373.31 us 373.31 us 1 READDIRP
0.00 410.11 us 410.11 us 410.11 us 1 XATTROP
0.00 172.89 us 140.25 us 229.22 us 3 SETXATTR
0.00 348.41 us 319.63 us 404.97 us 3 MKDIR
0.00 40.45 us 31.64 us 58.41 us 36 IPC
0.00 2020.28 us 690.06 us 2754.73 us 3 RMDIR
0.01 74.65 us 18.21 us 868.57 us 135 READDIR
0.02 170.34 us 80.49 us 293.60 us 155 READ
0.07 107.76 us 30.67 us 5006.30 us 827 STATFS
0.14 77.87 us 27.17 us 17142.15 us 2329 FSETXATTR
0.25 136.56 us 42.42 us 1019.27 us 2386 OPEN
0.36 61.10 us 16.56 us 3013.27 us 7498 INODELK
0.60 57.67 us 18.69 us 337.84 us 13355 FINODELK
0.78 50.74 us 13.71 us 556.16 us 19784 FLUSH
0.90 315.86 us 109.57 us 1634.45 us 3665 SETATTR
1.26 61.32 us 17.83 us 759.14 us 26444 ENTRYLK
2.03 98.28 us 2.27 us 4248.68 us 26502 OPENDIR
2.58 500.80 us 174.65 us 1841.95 us 6607 CREATE
3.16 304.42 us 93.33 us 46617.14 us 13316 FXATTROP
4.43 859.31 us 203.35 us 3433.33 us 6606 UNLINK
13.58 230.56 us 25.82 us 1441.09 us 75517 LOOKUP
15.45 745.15 us 20.75 us 6030.94 us 26595 GETXATTR
54.36 412.94 us 74.85 us 5222.73 us 168825 WRITE
0.00 0.00 us 0.00 us 0.00 us 1966855 UPCALL
0.00 0.00 us 0.00 us 0.00 us 1957919 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 2325 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 8936 CI_FORGET
Duration: 6596 seconds
Data Read: 18677760 bytes
Data Written: 38141710505 bytes
Interval 1 Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 13722 125700 15237
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 3236 746 30
Block Size: 4096b+ 65536b+ 131072b+
No. of Reads: 0 7 139
No. of Writes: 2 3034 2
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 55 431 4898
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1732
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6609 FORGET
0.00 0.00 us 0.00 us 0.00 us 8988 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26500 RELEASEDIR
0.00 373.31 us 373.31 us 373.31 us 1 READDIRP
0.00 410.11 us 410.11 us 410.11 us 1 XATTROP
0.00 172.89 us 140.25 us 229.22 us 3 SETXATTR
0.00 348.41 us 319.63 us 404.97 us 3 MKDIR
0.00 40.85 us 31.64 us 58.41 us 34 IPC
0.00 2020.28 us 690.06 us 2754.73 us 3 RMDIR
0.01 74.66 us 18.21 us 868.57 us 127 READDIR
0.02 170.34 us 80.49 us 293.60 us 155 READ
0.07 107.62 us 30.67 us 5006.30 us 806 STATFS
0.14 77.87 us 27.17 us 17142.15 us 2329 FSETXATTR
0.25 136.56 us 42.42 us 1019.27 us 2386 OPEN
0.36 61.10 us 16.56 us 3013.27 us 7498 INODELK
0.60 57.67 us 18.69 us 337.84 us 13355 FINODELK
0.78 50.74 us 13.71 us 556.16 us 19784 FLUSH
0.90 315.86 us 109.57 us 1634.45 us 3665 SETATTR
1.26 61.32 us 17.83 us 759.14 us 26444 ENTRYLK
2.03 98.29 us 2.27 us 4248.68 us 26498 OPENDIR
2.58 500.80 us 174.65 us 1841.95 us 6607 CREATE
3.16 304.42 us 93.33 us 46617.14 us 13316 FXATTROP
4.43 859.31 us 203.35 us 3433.33 us 6606 UNLINK
13.58 230.56 us 25.82 us 1441.09 us 75503 LOOKUP
15.45 745.45 us 20.75 us 6030.94 us 26582 GETXATTR
54.37 412.94 us 74.85 us 5222.73 us 168825 WRITE
0.00 0.00 us 0.00 us 0.00 us 1966855 UPCALL
0.00 0.00 us 0.00 us 0.00 us 1957919 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 2325 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 8936 CI_FORGET
Duration: 789 seconds
Data Read: 18677760 bytes
Data Written: 9485570217 bytes
Gluster7.0 (Ctime enabled)
==========================
fs-dl380-c1-n1:~ # gluster vo profile repo1 info
Brick: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 23977 143222 10220
Block Size: 512b+ 1024b+ 8192b+
No. of Reads: 0 0 1
No. of Writes: 2067 51 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 8 171
No. of Writes: 26 2954 0
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 50 480 5002
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1841
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6590 FORGET
0.00 0.00 us 0.00 us 0.00 us 7432 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 129.79 us 114.92 us 140.65 us 3 SETXATTR
0.00 386.13 us 331.50 us 440.75 us 2 READDIRP
0.00 292.58 us 249.60 us 360.18 us 3 XATTROP
0.00 576.10 us 446.41 us 775.50 us 3 MKDIR
0.00 1857.67 us 820.07 us 2405.74 us 3 RMDIR
0.00 33.78 us 14.96 us 120.77 us 216 IPC
0.02 73.47 us 11.54 us 315.66 us 412 FSETXATTR
0.02 169.11 us 43.46 us 405.47 us 188 READ
0.04 69.85 us 9.31 us 2344.93 us 906 READDIR
0.05 79.66 us 7.34 us 267.55 us 1151 GETXATTR
0.06 123.14 us 28.68 us 517.29 us 846 OPEN
0.32 85.70 us 25.91 us 399.66 us 6594 FGETXATTR
0.48 107.57 us 25.52 us 14326.25 us 7828 STATFS
0.49 43.59 us 11.85 us 2791.32 us 19801 FLUSH
0.50 62.56 us 13.76 us 1445.20 us 14181 FINODELK
0.67 177.44 us 56.71 us 687.67 us 6594 FSTAT
0.92 61.48 us 11.24 us 2399.86 us 26408 ENTRYLK
1.31 315.70 us 96.90 us 915.84 us 7305 SETATTR
1.97 525.68 us 173.64 us 3103.01 us 6595 CREATE
2.20 277.03 us 68.90 us 13473.77 us 13942 FXATTROP
2.70 214.72 us 9.56 us 2757.47 us 22114 LOOKUP
3.01 802.37 us 182.07 us 5259.34 us 6594 UNLINK
3.08 101.50 us 0.65 us 4039.32 us 53271 OPENDIR
36.60 4043.67 us 15.93 us 994013.08 us 15909 INODELK
45.55 421.68 us 64.99 us 9380.82 us 189890 WRITE
0.00 0.00 us 0.00 us 0.00 us 3677977 UPCALL
0.00 0.00 us 0.00 us 0.00 us 3671345 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 36 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6632 CI_FORGET
Duration: 5698 seconds
Data Read: 22977840 bytes
Data Written: 9841829137 bytes
Interval 0 Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 23977 143222 10220
Block Size: 512b+ 1024b+ 8192b+
No. of Reads: 0 0 1
No. of Writes: 2067 51 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 8 171
No. of Writes: 26 2954 0
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 50 480 5002
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1841
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6590 FORGET
0.00 0.00 us 0.00 us 0.00 us 7432 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 129.79 us 114.92 us 140.65 us 3 SETXATTR
0.00 386.13 us 331.50 us 440.75 us 2 READDIRP
0.00 292.58 us 249.60 us 360.18 us 3 XATTROP
0.00 576.10 us 446.41 us 775.50 us 3 MKDIR
0.00 1857.67 us 820.07 us 2405.74 us 3 RMDIR
0.00 33.78 us 14.96 us 120.77 us 216 IPC
0.02 73.47 us 11.54 us 315.66 us 412 FSETXATTR
0.02 169.11 us 43.46 us 405.47 us 188 READ
0.04 69.85 us 9.31 us 2344.93 us 906 READDIR
0.05 79.66 us 7.34 us 267.55 us 1151 GETXATTR
0.06 123.14 us 28.68 us 517.29 us 846 OPEN
0.32 85.70 us 25.91 us 399.66 us 6594 FGETXATTR
0.48 107.70 us 25.52 us 14326.25 us 7828 STATFS
0.49 43.59 us 11.85 us 2791.32 us 19801 FLUSH
0.50 62.56 us 13.76 us 1445.20 us 14181 FINODELK
0.67 177.44 us 56.71 us 687.67 us 6594 FSTAT
0.92 61.48 us 11.24 us 2399.86 us 26408 ENTRYLK
1.31 315.70 us 96.90 us 915.84 us 7305 SETATTR
1.97 525.68 us 173.64 us 3103.01 us 6595 CREATE
2.20 277.03 us 68.90 us 13473.77 us 13942 FXATTROP
2.70 214.72 us 9.56 us 2757.47 us 22114 LOOKUP
3.01 802.37 us 182.07 us 5259.34 us 6594 UNLINK
3.08 101.50 us 0.65 us 4039.32 us 53271 OPENDIR
36.60 4043.67 us 15.93 us 994013.08 us 15909 INODELK
45.55 421.68 us 64.99 us 9380.82 us 189890 WRITE
0.00 0.00 us 0.00 us 0.00 us 3677977 UPCALL
0.00 0.00 us 0.00 us 0.00 us 3671345 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 36 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6632 CI_FORGET
Duration: 5698 seconds
Data Read: 22977840 bytes
Data Written: 9841829137 bytes
Brick: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 4096b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 1 24 3023
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 320 0 0
No. of Writes: 0 41 436
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4856 1768
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6605 FORGET
0.00 0.00 us 0.00 us 0.00 us 7075 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 138.84 us 112.92 us 160.96 us 3 SETXATTR
0.00 399.64 us 344.42 us 454.87 us 2 READDIRP
0.00 571.87 us 398.94 us 844.47 us 3 MKDIR
0.01 2045.86 us 777.17 us 2709.66 us 3 RMDIR
0.01 35.10 us 12.94 us 93.54 us 216 IPC
0.06 72.26 us 12.14 us 334.52 us 432 FSETXATTR
0.10 113.49 us 26.00 us 484.88 us 473 OPEN
0.10 174.90 us 87.44 us 453.04 us 324 READ
0.11 65.07 us 11.03 us 4413.29 us 907 READDIR
0.11 67.32 us 13.89 us 1223.39 us 918 GETXATTR
1.06 87.26 us 26.59 us 194.28 us 6606 FGETXATTR
1.34 93.17 us 20.71 us 8258.97 us 7828 STATFS
1.37 56.19 us 14.38 us 588.75 us 13240 FINODELK
1.74 47.75 us 9.44 us 1288.06 us 19775 FLUSH
2.27 186.16 us 58.73 us 1957.66 us 6606 FSTAT
2.90 59.56 us 15.95 us 2550.84 us 26442 ENTRYLK
4.20 310.72 us 92.26 us 920.97 us 7338 SETATTR
6.31 517.93 us 194.94 us 3140.66 us 6606 CREATE
6.84 279.99 us 71.20 us 2480.15 us 13240 FXATTROP
8.44 214.97 us 15.99 us 3525.82 us 21284 LOOKUP
9.83 807.18 us 187.73 us 3510.05 us 6606 UNLINK
9.91 100.86 us 1.15 us 2942.26 us 53271 OPENDIR
19.45 1039.19 us 126.76 us 5976.65 us 10149 WRITE
23.82 873.49 us 13.82 us 996890.14 us 14792 INODELK
0.00 0.00 us 0.00 us 0.00 us 6707 UPCALL
0.00 0.00 us 0.00 us 0.00 us 82 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 21 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6625 CI_FORGET
Duration: 5698 seconds
Data Read: 41943040 bytes
Data Written: 9455665152 bytes
Interval 0 Stats:
Block Size: 4096b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 1 24 3023
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 320 0 0
No. of Writes: 0 41 436
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4856 1768
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6605 FORGET
0.00 0.00 us 0.00 us 0.00 us 7075 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 138.84 us 112.92 us 160.96 us 3 SETXATTR
0.00 399.64 us 344.42 us 454.87 us 2 READDIRP
0.00 571.87 us 398.94 us 844.47 us 3 MKDIR
0.01 2045.86 us 777.17 us 2709.66 us 3 RMDIR
0.01 35.10 us 12.94 us 93.54 us 216 IPC
0.06 72.26 us 12.14 us 334.52 us 432 FSETXATTR
0.10 113.49 us 26.00 us 484.88 us 473 OPEN
0.10 174.90 us 87.44 us 453.04 us 324 READ
0.11 65.07 us 11.03 us 4413.29 us 907 READDIR
0.11 67.32 us 13.89 us 1223.39 us 918 GETXATTR
1.06 87.26 us 26.59 us 194.28 us 6606 FGETXATTR
1.34 93.17 us 20.71 us 8258.97 us 7828 STATFS
1.37 56.19 us 14.38 us 588.75 us 13240 FINODELK
1.74 47.75 us 9.44 us 1288.06 us 19775 FLUSH
2.27 186.16 us 58.73 us 1957.66 us 6606 FSTAT
2.90 59.56 us 15.95 us 2550.84 us 26442 ENTRYLK
4.20 310.72 us 92.26 us 920.97 us 7338 SETATTR
6.31 517.93 us 194.94 us 3140.66 us 6606 CREATE
6.84 279.99 us 71.20 us 2480.15 us 13240 FXATTROP
8.44 214.97 us 15.99 us 3525.82 us 21284 LOOKUP
9.83 807.18 us 187.73 us 3510.05 us 6606 UNLINK
9.91 100.86 us 1.15 us 2942.26 us 53271 OPENDIR
19.45 1039.20 us 126.76 us 5976.65 us 10149 WRITE
23.82 873.49 us 13.82 us 996890.14 us 14792 INODELK
0.00 0.00 us 0.00 us 0.00 us 6707 UPCALL
0.00 0.00 us 0.00 us 0.00 us 82 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 21 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 6625 CI_FORGET
Duration: 5698 seconds
Data Read: 41943040 bytes
Data Written: 9455665152 bytes
Brick: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 23977 143222 10220
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 1
No. of Writes: 2067 51 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 1 170
No. of Writes: 26 2954 0
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 50 480 5002
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1841
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 944 FORGET
0.00 0.00 us 0.00 us 0.00 us 7430 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 134.44 us 75.05 us 193.89 us 3 SETXATTR
0.00 268.64 us 259.90 us 278.55 us 3 XATTROP
0.00 439.49 us 203.80 us 682.65 us 3 MKDIR
0.00 1853.60 us 823.20 us 2393.82 us 3 RMDIR
0.00 34.05 us 13.77 us 101.66 us 215 IPC
0.00 50.12 us 19.28 us 294.14 us 410 FSETXATTR
0.00 176.78 us 47.52 us 268.32 us 173 READ
0.00 67.16 us 9.59 us 4569.92 us 906 READDIR
0.00 135.83 us 38.87 us 18143.17 us 837 OPEN
0.02 76.76 us 23.00 us 192.51 us 6594 FGETXATTR
0.04 65.77 us 10.26 us 19823.69 us 14227 FINODELK
0.04 68.49 us 11.99 us 18024.04 us 15909 INODELK
0.05 155.32 us 16.76 us 21625.26 us 7828 STATFS
0.08 142.52 us 37.26 us 4159.63 us 14358 FSTAT
0.08 296.31 us 85.27 us 2190.78 us 7305 SETATTR
0.13 509.05 us 179.26 us 6280.47 us 6595 CREATE
0.15 274.40 us 79.84 us 13303.23 us 13942 FXATTROP
0.18 233.24 us 8.82 us 53288.69 us 19801 FLUSH
0.19 738.29 us 191.82 us 26804.67 us 6594 UNLINK
0.44 416.08 us 11.09 us 31352.67 us 27762 LOOKUP
0.55 551.61 us 10.11 us 46117.06 us 26408 ENTRYLK
0.74 367.23 us 0.64 us 34196.75 us 53271 OPENDIR
2.73 372.74 us 8.60 us 61916.08 us 192531 GETXATTR
4.06 560.96 us 62.01 us 62168.04 us 189890 WRITE
90.53 13985.76 us 15.94 us 60188.98 us 170026 READDIRP
0.00 0.00 us 0.00 us 0.00 us 3672463 UPCALL
0.00 0.00 us 0.00 us 0.00 us 3671509 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 37 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 954 CI_FORGET
Duration: 5691 seconds
Data Read: 22379229 bytes
Data Written: 9841829137 bytes
Interval 0 Stats:
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 23977 143222 10220
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 1
No. of Writes: 2067 51 0
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 1 170
No. of Writes: 26 2954 0
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 50 480 5002
Block Size: 2097152b+
No. of Reads: 0
No. of Writes: 1841
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 944 FORGET
0.00 0.00 us 0.00 us 0.00 us 7430 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 134.44 us 75.05 us 193.89 us 3 SETXATTR
0.00 268.64 us 259.90 us 278.55 us 3 XATTROP
0.00 439.49 us 203.80 us 682.65 us 3 MKDIR
0.00 1853.60 us 823.20 us 2393.82 us 3 RMDIR
0.00 34.05 us 13.77 us 101.66 us 215 IPC
0.00 50.12 us 19.28 us 294.14 us 410 FSETXATTR
0.00 176.78 us 47.52 us 268.32 us 173 READ
0.00 67.16 us 9.59 us 4569.92 us 906 READDIR
0.00 135.83 us 38.87 us 18143.17 us 837 OPEN
0.02 76.76 us 23.00 us 192.51 us 6594 FGETXATTR
0.04 65.77 us 10.26 us 19823.69 us 14227 FINODELK
0.04 68.49 us 11.99 us 18024.04 us 15909 INODELK
0.05 155.32 us 16.76 us 21625.26 us 7828 STATFS
0.08 142.52 us 37.26 us 4159.63 us 14358 FSTAT
0.08 296.31 us 85.27 us 2190.78 us 7305 SETATTR
0.13 509.05 us 179.26 us 6280.47 us 6595 CREATE
0.15 274.40 us 79.84 us 13303.23 us 13942 FXATTROP
0.18 233.24 us 8.82 us 53288.69 us 19801 FLUSH
0.19 738.29 us 191.82 us 26804.67 us 6594 UNLINK
0.44 416.08 us 11.09 us 31352.67 us 27762 LOOKUP
0.55 551.61 us 10.11 us 46117.06 us 26408 ENTRYLK
0.74 367.23 us 0.64 us 34196.75 us 53271 OPENDIR
2.73 372.74 us 8.60 us 61916.08 us 192531 GETXATTR
4.06 560.96 us 62.01 us 62168.04 us 189890 WRITE
90.53 13985.90 us 15.94 us 60188.98 us 170026 READDIRP
0.00 0.00 us 0.00 us 0.00 us 3672463 UPCALL
0.00 0.00 us 0.00 us 0.00 us 3671509 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 37 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 954 CI_FORGET
Duration: 5691 seconds
Data Read: 22379229 bytes
Data Written: 9841829137 bytes
Brick: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
Block Size: 4096b+ 32768b+ 65536b+
No. of Reads: 0 0 7
No. of Writes: 1 24 3023
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 320 0 0
No. of Writes: 0 41 436
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4856 1768
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 249 FORGET
0.00 0.00 us 0.00 us 0.00 us 7069 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 163.77 us 143.80 us 187.63 us 3 SETXATTR
0.00 539.28 us 199.35 us 909.49 us 3 MKDIR
0.00 1861.11 us 734.86 us 2433.60 us 3 RMDIR
0.00 34.80 us 14.08 us 120.65 us 215 IPC
0.00 47.22 us 16.75 us 265.51 us 426 FSETXATTR
0.00 105.54 us 34.69 us 438.87 us 474 OPEN
0.00 58.10 us 9.99 us 2784.86 us 906 READDIR
0.00 193.84 us 78.00 us 1070.84 us 338 READ
0.01 78.11 us 27.12 us 267.42 us 6606 FGETXATTR
0.02 62.29 us 15.49 us 2094.30 us 13240 FINODELK
0.03 71.48 us 13.41 us 2627.65 us 14792 INODELK
0.03 146.99 us 13.72 us 18406.23 us 7828 STATFS
0.06 144.89 us 37.17 us 1728.42 us 14262 FSTAT
0.06 294.14 us 95.08 us 1298.49 us 7338 SETATTR
0.09 501.65 us 171.12 us 3673.36 us 6606 CREATE
0.10 277.18 us 83.58 us 2255.78 us 13240 FXATTROP
0.11 618.23 us 182.22 us 31669.04 us 6606 UNLINK
0.19 247.20 us 12.45 us 56440.26 us 27624 LOOKUP
0.19 269.37 us 9.01 us 29850.52 us 26442 ENTRYLK
0.26 942.72 us 125.40 us 11143.54 us 10149 WRITE
0.52 569.83 us 8.92 us 3526.39 us 33889 GETXATTR
1.14 2122.15 us 8.75 us 170951.32 us 19775 FLUSH
1.33 920.75 us 0.81 us 27584.76 us 53271 OPENDIR
95.86 23866.45 us 76.91 us 66659.49 us 148246 READDIRP
0.00 0.00 us 0.00 us 0.00 us 341 UPCALL
0.00 0.00 us 0.00 us 0.00 us 91 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 30 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 250 CI_FORGET
Duration: 5691 seconds
Data Read: 42401792 bytes
Data Written: 9455665152 bytes
Interval 0 Stats:
Block Size: 4096b+ 32768b+ 65536b+
No. of Reads: 0 0 7
No. of Writes: 1 24 3023
Block Size: 131072b+ 262144b+ 524288b+
No. of Reads: 320 0 0
No. of Writes: 0 41 436
Block Size: 1048576b+ 2097152b+
No. of Reads: 0 0
No. of Writes: 4856 1768
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 249 FORGET
0.00 0.00 us 0.00 us 0.00 us 7069 RELEASE
0.00 0.00 us 0.00 us 0.00 us 53292 RELEASEDIR
0.00 163.77 us 143.80 us 187.63 us 3 SETXATTR
0.00 539.28 us 199.35 us 909.49 us 3 MKDIR
0.00 1861.11 us 734.86 us 2433.60 us 3 RMDIR
0.00 34.80 us 14.08 us 120.65 us 215 IPC
0.00 47.22 us 16.75 us 265.51 us 426 FSETXATTR
0.00 105.54 us 34.69 us 438.87 us 474 OPEN
0.00 58.10 us 9.99 us 2784.86 us 906 READDIR
0.00 193.84 us 78.00 us 1070.84 us 338 READ
0.01 78.11 us 27.12 us 267.42 us 6606 FGETXATTR
0.02 62.29 us 15.49 us 2094.30 us 13240 FINODELK
0.03 71.48 us 13.41 us 2627.65 us 14792 INODELK
0.03 146.99 us 13.72 us 18406.23 us 7828 STATFS
0.06 144.89 us 37.17 us 1728.42 us 14262 FSTAT
0.06 294.14 us 95.08 us 1298.49 us 7338 SETATTR
0.09 501.65 us 171.12 us 3673.36 us 6606 CREATE
0.10 277.18 us 83.58 us 2255.78 us 13240 FXATTROP
0.11 618.23 us 182.22 us 31669.04 us 6606 UNLINK
0.19 247.20 us 12.45 us 56440.26 us 27624 LOOKUP
0.19 269.37 us 9.01 us 29850.52 us 26442 ENTRYLK
0.26 942.72 us 125.40 us 11143.54 us 10149 WRITE
0.52 569.83 us 8.92 us 3526.39 us 33889 GETXATTR
1.14 2122.15 us 8.75 us 170951.32 us 19775 FLUSH
1.33 920.75 us 0.81 us 27584.76 us 53271 OPENDIR
95.86 23866.52 us 76.91 us 66659.49 us 148246 READDIRP
0.00 0.00 us 0.00 us 0.00 us 341 UPCALL
0.00 0.00 us 0.00 us 0.00 us 91 CI_IATT
0.00 0.00 us 0.00 us 0.00 us 30 CI_UNLINK
0.00 0.00 us 0.00 us 0.00 us 250 CI_FORGET
Duration: 5691 seconds
Data Read: 42401792 bytes
Data Written: 9455665152 bytes
Gluster7.0 (Ctime disabled)
==========================
ToDo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 64KiB_x_6000_8T.fio
Type: application/octet-stream
Size: 194 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191105/26cfa138/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 10MiB_x_1200_8T.fio
Type: application/octet-stream
Size: 195 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191105/26cfa138/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 1MiB_x_6000_8T.fio
Type: application/octet-stream
Size: 193 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191105/26cfa138/attachment-0002.obj>
More information about the Gluster-users
mailing list