[Gluster-users] Performance is falling rapidly when updating from v5.5 to v7.0

David Spisla spisla80 at gmail.com
Tue Nov 5 17:35:08 UTC 2019


I did the test with Gluster 7.0 ctime disabled. But it had no effect:
(All values in MiB/s)
64KiB    1MiB     10MiB
0,16       2,60       54,74

Attached there is now the complete profile file also with the results from
the last test. I will not repeat it with an higher inode size because I
don't think this will have an effect.
There must be another cause for the low performance

Regards
David Spisla

Am Di., 5. Nov. 2019 um 16:25 Uhr schrieb David Spisla <spisla80 at gmail.com>:

>
>
> Am Di., 5. Nov. 2019 um 12:06 Uhr schrieb RAFI KC <rkavunga at redhat.com>:
>
>>
>> On 11/4/19 8:46 PM, David Spisla wrote:
>>
>> Dear Gluster Community,
>>
>> I also have a issue concerning performance. The last days I updated our
>> test cluster from GlusterFS v5.5 to v7.0 . The setup in general:
>>
>> 2 HP DL380 Servers with 10Gbit NICs, 1 Distribute-Replica 2 Volume with 2
>> Replica Pairs. Client is SMB Samba (access via vfs_glusterfs) . I did
>> several tests to ensure that Samba don't causes the fall.
>> The setup ist completely the same except the Gluster Version
>> Here are my results:
>> 64KiB           1MiB             10MiB            (Filesize)
>> 3,49             47,41            300,50          (Values in MiB/s with
>> GlusterFS v5.5)
>> 0,16              2,61             76,63            (Values in MiB/s with
>> GlusterFS v7.0)
>>
>>
>> Can you please share the profile information [1] for both versions?  Also
>> it would be really helpful if you can mention the io patterns that used for
>> this tests.
>>
>> [1] :
>> https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/
>>
> Hello Rafi,
> thank you for your help.
>
> * First more information about the io patterns: As a client we use a DL360
> Windws Server 2017 machine with 10Gbit NIC connected to the storage
> machines. The share will be mounted via SMB and the tests writes with fio.
> We use this job files (see attachment). Each job file will be executed
> separetely and there is a sleep about 60s between each test run to calm
> down the system before starting a new test.
>
> * Attached below you find the profile output from the tests with v5.5
> (ctime enabled), v7.0 (ctime enabled).
>
> * Beside of the tests with Samba I did also some fio tests directly on the
> FUSE Mounts (locally on one of the storage nodes). The results show that
> there is only a small decrease of performance between v5.5 and v7.0
> (All values in MiB/s)
> 64KiB    1MiB     10MiB
> 50,09     679,96   1023,02 (v5.5)
> 47,00     656,46    977,60 (v7.0)
>
> It seems to be that the combination of samba + gluster7.0 has a lot of
> problems, or not?
>
>
>>
>> We use this volume options (GlusterFS 7.0):
>>
>> Volume Name: archive1
>> Type: Distributed-Replicate
>> Volume ID: 44c17844-0bd4-4ca2-98d8-a1474add790c
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 2 x 2 = 4
>> Transport-type: tcp
>> Bricks:
>> Brick1: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
>> Brick2: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
>> Brick3: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
>> Brick4: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
>> Options Reconfigured:
>> performance.client-io-threads: off
>> nfs.disable: on
>> storage.fips-mode-rchecksum: on
>> transport.address-family: inet
>> user.smb: disable
>> features.read-only: off
>> features.worm: off
>> features.worm-file-level: on
>> features.retention-mode: enterprise
>> features.default-retention-period: 120
>> network.ping-timeout: 10
>> features.cache-invalidation: on
>> features.cache-invalidation-timeout: 600
>> performance.nl-cache: on
>> performance.nl-cache-timeout: 600
>> client.event-threads: 32
>> server.event-threads: 32
>> cluster.lookup-optimize: on
>> performance.stat-prefetch: on
>> performance.cache-invalidation: on
>> performance.md-cache-timeout: 600
>> performance.cache-samba-metadata: on
>> performance.cache-ima-xattrs: on
>> performance.io-thread-count: 64
>> cluster.use-compound-fops: on
>> performance.cache-size: 512MB
>> performance.cache-refresh-timeout: 10
>> performance.read-ahead: off
>> performance.write-behind-window-size: 4MB
>> performance.write-behind: on
>> storage.build-pgfid: on
>> features.ctime: on
>> cluster.quorum-type: fixed
>> cluster.quorum-count: 1
>> features.bitrot: on
>> features.scrub: Active
>> features.scrub-freq: daily
>>
>> For GlusterFS 5.5 its nearly the same except the fact that there were 2
>> options to enable ctime feature.
>>
>>
>>
>> Ctime stores additional metadata information as an extended attributes
>> which sometimes exceeds the default inode size. In such scenarios the
>> additional xattrs won't fit into the default size. This will result in
>> additional blocks to be used to store xattrs in the inide, which will
>> effect the latency. This is purely based on the i/o operations and the
>> total xattrs size stored in the inode.
>>
>> Is it possible for you to repeat the test by disabling ctime or
>> increasing the inode size to a higher value say 1024KB?
>>
> I will do so but for today I could not finish tests with ctime disabled
> (or higher inode value) because it takes a lot of time with v7.0 due to the
> low performance and I will perform it tomorrow. As soon as possible I give
> you the results.
> By the way: You really mean inode size on xfs layer 1024KB? Or do you mean
> 1024Bytes? We use per default 512Bytes, because this is the recommended
> size until now . But it seems to be that there is a need for a new
> recommendation when using ctime feature as a default. I can not image that
> this is the real cause for the low performance because in v5.5 we also use
> ctime feature with inode size 512Bytes.
>
> Regards
> David
>
>>
>> Our optimization for Samba looks like this (for every version):
>>
>> [global]
>> workgroup = SAMBA
>> netbios name = CLUSTER
>> kernel share modes = no
>> aio read size = 1
>> aio write size = 1
>> kernel oplocks = no
>> max open files = 100000
>> nt acl support = no
>> security = user
>> server min protocol = SMB2
>> store dos attributes = no
>> strict locking = no
>> full_audit:failure = pwrite_send pwrite_recv pwrite offload_write_send
>> offload_write_recv create_file open unlink connect disconnect rename chown
>> fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
>> full_audit:success = pwrite_send pwrite_recv pwrite offload_write_send
>> offload_write_recv create_file open unlink connect disconnect rename chown
>> fchown lchown chmod fchmod mkdir rmdir ntimes ftruncate fallocate
>> full_audit:facility = local5
>> durable handles = yes
>> posix locking = no
>> log level = 2
>> max log size = 100000
>> debug pid = yes
>>
>> What can be the cause for this rapid falling of the performance for small
>> files? Are some of our vol options not recommended anymore?
>> There were some patches concerning performance for small files in v6.0
>> und v7.0 :
>>
>> #1670031 <https://bugzilla.redhat.com/1670031>: performance regression
>> seen with smallfile workload tests
>>
>> #1659327 <https://bugzilla.redhat.com/1659327>: 43% regression in
>> small-file sequential read performance
>>
>> And one patch for the io-cache:
>>
>> #1659869 <https://bugzilla.redhat.com/1659869>: improvements to io-cache
>>
>> Regards
>>
>> David Spisla
>>
>>
>> ________
>>
>> Community Meeting Calendar:
>>
>> APAC Schedule -
>> Every 2nd and 4th Tuesday at 11:30 AM IST
>> Bridge: https://bluejeans.com/118564314
>>
>> NA/EMEA Schedule -
>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>> Bridge: https://bluejeans.com/118564314
>>
>> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191105/b9f66076/attachment.html>
-------------- next part --------------
Gluster5.5 (Ctime enabled)
==========================

fs-dl380-c1-n1:~ # gluster vo profile repo1 info
Brick: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                13722                125700                 15237 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3236                   746                    30 
 
   Block Size:               4096b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    2                  9051                  9050 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    7                   163                     0 
No. of Writes:                 3034                214018                    99 
 
   Block Size:             524288b+             1048576b+             2097152b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  431                  4898                  1732 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          19937      FORGET
      0.00       0.00 us       0.00 us       0.00 us          28745     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          27033  RELEASEDIR
      0.00     372.52 us     372.52 us     372.52 us              1     XATTROP
      0.00     155.97 us     148.40 us     163.74 us              3    SETXATTR
      0.00     355.85 us     319.36 us     392.34 us              3       MKDIR
      0.00      35.10 us      14.17 us      58.11 us             39         IPC
      0.00    2092.77 us     759.52 us    2785.95 us              3       RMDIR
      0.00      57.69 us      13.35 us     514.30 us            135     READDIR
      0.01     894.26 us      34.03 us   56491.66 us            182        READ
      0.01     374.15 us      22.21 us   19577.39 us            827      STATFS
      0.03      69.78 us      21.13 us     352.94 us           7778       FSTAT
      0.03     282.54 us      17.73 us   23688.04 us           2268   FSETXATTR
      0.04      60.61 us      14.01 us   10210.17 us          13348    FINODELK
      0.04     355.90 us      31.44 us   56158.91 us           2325        OPEN
      0.05     306.29 us     109.78 us     674.82 us           3665     SETATTR
      0.10     276.25 us      21.09 us 1058896.03 us           7498     INODELK
      0.16     507.07 us     181.63 us    5291.74 us           6607      CREATE
      0.18     297.21 us      90.56 us   11128.32 us          13316    FXATTROP
      0.24     773.47 us     179.92 us   27742.67 us           6606      UNLINK
      0.33     265.41 us      12.03 us   62361.64 us          26444     ENTRYLK
      0.88     234.65 us      18.24 us   15480.79 us          80510      LOOKUP
      0.93     750.11 us      12.26 us   14390.35 us          26600    GETXATTR
      1.04    1127.56 us      11.10 us  227020.48 us          19784       FLUSH
      1.09     881.86 us       1.22 us   33956.54 us          26502     OPENDIR
      4.70     595.48 us      68.68 us  230191.32 us         168825       WRITE
     90.14   33977.47 us      84.81 us  129099.85 us          56766    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        1958138      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1958010     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us           2261   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            128   CI_FORGET
 
    Duration: 6513 seconds
   Data Read: 21823488 bytes
Data Written: 38141710505 bytes
 
Interval 1 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                13722                125700                 15237 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3236                   746                    30 
 
   Block Size:               4096b+               65536b+              131072b+ 
 No. of Reads:                    0                     7                   163 
No. of Writes:                    2                  3034                     2 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   55                   431                  4898 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1732 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            126      FORGET
      0.00       0.00 us       0.00 us       0.00 us           8927     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26500  RELEASEDIR
      0.00     372.52 us     372.52 us     372.52 us              1     XATTROP
      0.00     155.97 us     148.40 us     163.74 us              3    SETXATTR
      0.00     355.85 us     319.36 us     392.34 us              3       MKDIR
      0.00      36.18 us      14.17 us      58.11 us             34         IPC
      0.00    2092.77 us     759.52 us    2785.95 us              3       RMDIR
      0.00      57.02 us      13.35 us     514.30 us            127     READDIR
      0.01     894.26 us      34.03 us   56491.66 us            182        READ
      0.01     380.67 us      22.21 us   19577.39 us            806      STATFS
      0.03      69.78 us      21.13 us     352.94 us           7778       FSTAT
      0.03     282.54 us      17.73 us   23688.04 us           2268   FSETXATTR
      0.04      60.61 us      14.01 us   10210.17 us          13348    FINODELK
      0.04     355.90 us      31.44 us   56158.91 us           2325        OPEN
      0.05     306.29 us     109.78 us     674.82 us           3665     SETATTR
      0.10     276.25 us      21.09 us 1058896.03 us           7498     INODELK
      0.16     507.07 us     181.63 us    5291.74 us           6607      CREATE
      0.18     297.21 us      90.56 us   11128.32 us          13316    FXATTROP
      0.24     773.47 us     179.92 us   27742.67 us           6606      UNLINK
      0.33     265.41 us      12.03 us   62361.64 us          26444     ENTRYLK
      0.88     234.65 us      18.24 us   15480.79 us          80496      LOOKUP
      0.93     750.43 us      12.26 us   14390.35 us          26587    GETXATTR
      1.04    1127.56 us      11.10 us  227020.48 us          19784       FLUSH
      1.09     882.00 us       1.22 us   33956.54 us          26498     OPENDIR
      4.70     595.47 us      68.68 us  230191.32 us         168825       WRITE
     90.14   33978.50 us      84.81 us  129099.85 us          56766    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        1958138      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1958010     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us           2261   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            128   CI_FORGET
 
    Duration: 789 seconds
   Data Read: 21823488 bytes
Data Written: 9485570217 bytes
 
Brick: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:              16384b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 8952                  8950                  2966 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:               217821                    88                   484 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4982                  1846 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          20194      FORGET
      0.00       0.00 us       0.00 us       0.00 us          26431     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26795  RELEASEDIR
      0.00     164.43 us     161.90 us     167.62 us              3    SETXATTR
      0.00      30.57 us      17.79 us      44.45 us             21         IPC
      0.00     390.62 us     383.31 us     397.93 us              2     XATTROP
      0.00      56.79 us      26.70 us     152.48 us             23   FSETXATTR
      0.00     447.69 us     400.71 us     510.30 us              3       MKDIR
      0.00     129.81 us      55.26 us     289.15 us             28        OPEN
      0.00      75.71 us      14.13 us     816.87 us             75     READDIR
      0.00    2188.47 us     796.44 us    2993.06 us              3       RMDIR
      0.01     160.12 us      21.94 us    6000.03 us            813      STATFS
      0.06      70.99 us      24.27 us   29788.04 us           7349     INODELK
      0.07      73.26 us      24.65 us     677.95 us           7784       FSTAT
      0.09      60.50 us      17.42 us     521.01 us          13194    FINODELK
      0.13     312.52 us     117.48 us     565.10 us           3652     SETATTR
      0.39     508.69 us     179.00 us    1052.82 us           6594      CREATE
      0.44     292.42 us      90.47 us    4546.87 us          13194    FXATTROP
      0.77     337.08 us      12.36 us   56472.78 us          19800       FLUSH
      0.81    1067.61 us     218.87 us   50116.71 us           6594      UNLINK
      1.08     356.53 us       1.54 us   22107.25 us          26472     OPENDIR
      1.12     368.18 us      13.26 us   68738.68 us          26406     ENTRYLK
      1.19    1003.64 us     166.98 us    6085.26 us          10330       WRITE
      2.29     753.23 us      12.07 us    5793.53 us          26507    GETXATTR
      2.44     253.59 us      16.41 us   13225.37 us          83900      LOOKUP
     89.11   12640.00 us      20.25 us  145809.04 us          61411    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            425      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             35     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             27   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            390   CI_FORGET
 
    Duration: 6513 seconds
   Data Read: 0 bytes
Data Written: 38958530560 bytes
 
Interval 1 Stats:
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 2966                     3                    49 
 
   Block Size:             524288b+             1048576b+             2097152b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  484                  4982                  1846 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            387      FORGET
      0.00       0.00 us       0.00 us       0.00 us           6622     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26472  RELEASEDIR
      0.00     164.43 us     161.90 us     167.62 us              3    SETXATTR
      0.00      30.40 us      17.79 us      44.45 us             19         IPC
      0.00     390.62 us     383.31 us     397.93 us              2     XATTROP
      0.00      56.79 us      26.70 us     152.48 us             23   FSETXATTR
      0.00     447.69 us     400.71 us     510.30 us              3       MKDIR
      0.00     129.81 us      55.26 us     289.15 us             28        OPEN
      0.00      77.61 us      14.13 us     816.87 us             71     READDIR
      0.00    2188.47 us     796.44 us    2993.06 us              3       RMDIR
      0.01     161.55 us      21.94 us    6000.03 us            792      STATFS
      0.06      70.99 us      24.27 us   29788.04 us           7349     INODELK
      0.07      73.26 us      24.65 us     677.95 us           7784       FSTAT
      0.09      60.50 us      17.42 us     521.01 us          13194    FINODELK
      0.13     312.52 us     117.48 us     565.10 us           3652     SETATTR
      0.39     508.69 us     179.00 us    1052.82 us           6594      CREATE
      0.44     292.42 us      90.47 us    4546.87 us          13194    FXATTROP
      0.77     337.08 us      12.36 us   56472.78 us          19800       FLUSH
      0.81    1067.61 us     218.87 us   50116.71 us           6594      UNLINK
      1.08     356.55 us       1.54 us   22107.25 us          26470     OPENDIR
      1.12     368.18 us      13.26 us   68738.68 us          26406     ENTRYLK
      1.19    1003.64 us     166.98 us    6085.26 us          10330       WRITE
      2.29     753.43 us      12.07 us    5793.53 us          26499    GETXATTR
      2.44     253.59 us      16.41 us   13225.37 us          83890      LOOKUP
     89.11   12640.71 us      20.25 us  145809.04 us          61411    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            425      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             35     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             27   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            390   CI_FORGET
 
    Duration: 789 seconds
   Data Read: 0 bytes
Data Written: 9811918848 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:              16384b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 8952                  8950                  2966 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                   24                     0                     0 
No. of Writes:               217821                    88                   484 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4982                  1846 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          26404      FORGET
      0.00       0.00 us       0.00 us       0.00 us          26426     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          27037  RELEASEDIR
      0.00     242.57 us     242.57 us     242.57 us              1    READDIRP
      0.00     270.43 us     224.72 us     316.13 us              2 REMOVEXATTR
      0.00     376.30 us     357.25 us     395.35 us              2     XATTROP
      0.00     411.18 us     363.84 us     462.52 us              3       MKDIR
      0.00      36.88 us      25.54 us      54.42 us             36         IPC
      0.00     176.85 us      32.37 us     492.36 us             15   FSETXATTR
      0.01     206.12 us     108.73 us     412.08 us             21        OPEN
      0.01     197.53 us      85.44 us     451.06 us             27        READ
      0.01    1995.17 us     760.64 us    2693.43 us              3       RMDIR
      0.01    1366.94 us     127.39 us    3611.57 us              5    SETXATTR
      0.02      88.63 us      25.71 us     863.15 us            135     READDIR
      0.13     116.18 us      34.05 us    2167.66 us            828      STATFS
      0.78      78.60 us      26.05 us   52960.99 us           7352     INODELK
      1.19      67.15 us      20.02 us    2608.91 us          13194    FINODELK
      1.58      59.32 us      18.09 us    2981.27 us          19800       FLUSH
      1.64     334.42 us     118.08 us     855.90 us           3654     SETATTR
      2.57      72.45 us      19.16 us     732.36 us          26406     ENTRYLK
      3.84     108.04 us       2.38 us     677.07 us          26502     OPENDIR
      4.56     515.76 us     191.84 us     883.80 us           6594      CREATE
      5.32     300.53 us      96.16 us    6531.99 us          13194    FXATTROP
      7.63     862.82 us     248.89 us    4093.19 us           6594      UNLINK
     14.59    1052.78 us     173.50 us    3653.35 us          10330       WRITE
     26.87     753.92 us      21.93 us    5703.54 us          26567    GETXATTR
     29.25     276.32 us      32.87 us    3104.49 us          78907      LOOKUP
      0.00       0.00 us       0.00 us       0.00 us           6646      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             30     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             17   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6616   CI_FORGET
 
    Duration: 6596 seconds
   Data Read: 3145728 bytes
Data Written: 38958530560 bytes
 
Interval 1 Stats:
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                    24                     0 
No. of Writes:                 2966                     3                    49 
 
   Block Size:             524288b+             1048576b+             2097152b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  484                  4982                  1846 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6597      FORGET
      0.00       0.00 us       0.00 us       0.00 us           6615     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26500  RELEASEDIR
      0.00     242.57 us     242.57 us     242.57 us              1    READDIRP
      0.00     270.43 us     224.72 us     316.13 us              2 REMOVEXATTR
      0.00     376.30 us     357.25 us     395.35 us              2     XATTROP
      0.00     411.18 us     363.84 us     462.52 us              3       MKDIR
      0.00      37.24 us      25.54 us      54.42 us             34         IPC
      0.00     176.85 us      32.37 us     492.36 us             15   FSETXATTR
      0.01     206.12 us     108.73 us     412.08 us             21        OPEN
      0.01     197.53 us      85.44 us     451.06 us             27        READ
      0.01    1995.17 us     760.64 us    2693.43 us              3       RMDIR
      0.01    1366.94 us     127.39 us    3611.57 us              5    SETXATTR
      0.02      88.84 us      25.71 us     863.15 us            127     READDIR
      0.13     116.49 us      34.05 us    2167.66 us            806      STATFS
      0.78      78.60 us      26.05 us   52960.99 us           7352     INODELK
      1.19      67.15 us      20.02 us    2608.91 us          13194    FINODELK
      1.58      59.32 us      18.09 us    2981.27 us          19800       FLUSH
      1.64     334.42 us     118.08 us     855.90 us           3654     SETATTR
      2.57      72.45 us      19.16 us     732.36 us          26406     ENTRYLK
      3.84     108.06 us       2.38 us     677.07 us          26498     OPENDIR
      4.56     515.76 us     191.84 us     883.80 us           6594      CREATE
      5.32     300.53 us      96.16 us    6531.99 us          13194    FXATTROP
      7.63     862.82 us     248.89 us    4093.19 us           6594      UNLINK
     14.59    1052.78 us     173.50 us    3653.35 us          10330       WRITE
     26.87     754.21 us      21.93 us    5703.54 us          26555    GETXATTR
     29.25     276.33 us      32.87 us    3104.49 us          78893      LOOKUP
      0.00       0.00 us       0.00 us       0.00 us           6646      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             30     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             17   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6616   CI_FORGET
 
    Duration: 789 seconds
   Data Read: 3145728 bytes
Data Written: 9811918848 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                13722                125700                 15237 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3236                   746                    30 
 
   Block Size:               4096b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    2                  9051                  9050 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    7                   139                     0 
No. of Writes:                 3034                214018                    99 
 
   Block Size:             524288b+             1048576b+             2097152b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  431                  4898                  1732 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          26420      FORGET
      0.00       0.00 us       0.00 us       0.00 us          28806     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          27035  RELEASEDIR
      0.00     373.31 us     373.31 us     373.31 us              1    READDIRP
      0.00     410.11 us     410.11 us     410.11 us              1     XATTROP
      0.00     172.89 us     140.25 us     229.22 us              3    SETXATTR
      0.00     348.41 us     319.63 us     404.97 us              3       MKDIR
      0.00      40.45 us      31.64 us      58.41 us             36         IPC
      0.00    2020.28 us     690.06 us    2754.73 us              3       RMDIR
      0.01      74.65 us      18.21 us     868.57 us            135     READDIR
      0.02     170.34 us      80.49 us     293.60 us            155        READ
      0.07     107.76 us      30.67 us    5006.30 us            827      STATFS
      0.14      77.87 us      27.17 us   17142.15 us           2329   FSETXATTR
      0.25     136.56 us      42.42 us    1019.27 us           2386        OPEN
      0.36      61.10 us      16.56 us    3013.27 us           7498     INODELK
      0.60      57.67 us      18.69 us     337.84 us          13355    FINODELK
      0.78      50.74 us      13.71 us     556.16 us          19784       FLUSH
      0.90     315.86 us     109.57 us    1634.45 us           3665     SETATTR
      1.26      61.32 us      17.83 us     759.14 us          26444     ENTRYLK
      2.03      98.28 us       2.27 us    4248.68 us          26502     OPENDIR
      2.58     500.80 us     174.65 us    1841.95 us           6607      CREATE
      3.16     304.42 us      93.33 us   46617.14 us          13316    FXATTROP
      4.43     859.31 us     203.35 us    3433.33 us           6606      UNLINK
     13.58     230.56 us      25.82 us    1441.09 us          75517      LOOKUP
     15.45     745.15 us      20.75 us    6030.94 us          26595    GETXATTR
     54.36     412.94 us      74.85 us    5222.73 us         168825       WRITE
      0.00       0.00 us       0.00 us       0.00 us        1966855      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1957919     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us           2325   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           8936   CI_FORGET
 
    Duration: 6596 seconds
   Data Read: 18677760 bytes
Data Written: 38141710505 bytes
 
Interval 1 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                13722                125700                 15237 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 3236                   746                    30 
 
   Block Size:               4096b+               65536b+              131072b+ 
 No. of Reads:                    0                     7                   139 
No. of Writes:                    2                  3034                     2 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   55                   431                  4898 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1732 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6609      FORGET
      0.00       0.00 us       0.00 us       0.00 us           8988     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          26500  RELEASEDIR
      0.00     373.31 us     373.31 us     373.31 us              1    READDIRP
      0.00     410.11 us     410.11 us     410.11 us              1     XATTROP
      0.00     172.89 us     140.25 us     229.22 us              3    SETXATTR
      0.00     348.41 us     319.63 us     404.97 us              3       MKDIR
      0.00      40.85 us      31.64 us      58.41 us             34         IPC
      0.00    2020.28 us     690.06 us    2754.73 us              3       RMDIR
      0.01      74.66 us      18.21 us     868.57 us            127     READDIR
      0.02     170.34 us      80.49 us     293.60 us            155        READ
      0.07     107.62 us      30.67 us    5006.30 us            806      STATFS
      0.14      77.87 us      27.17 us   17142.15 us           2329   FSETXATTR
      0.25     136.56 us      42.42 us    1019.27 us           2386        OPEN
      0.36      61.10 us      16.56 us    3013.27 us           7498     INODELK
      0.60      57.67 us      18.69 us     337.84 us          13355    FINODELK
      0.78      50.74 us      13.71 us     556.16 us          19784       FLUSH
      0.90     315.86 us     109.57 us    1634.45 us           3665     SETATTR
      1.26      61.32 us      17.83 us     759.14 us          26444     ENTRYLK
      2.03      98.29 us       2.27 us    4248.68 us          26498     OPENDIR
      2.58     500.80 us     174.65 us    1841.95 us           6607      CREATE
      3.16     304.42 us      93.33 us   46617.14 us          13316    FXATTROP
      4.43     859.31 us     203.35 us    3433.33 us           6606      UNLINK
     13.58     230.56 us      25.82 us    1441.09 us          75503      LOOKUP
     15.45     745.45 us      20.75 us    6030.94 us          26582    GETXATTR
     54.37     412.94 us      74.85 us    5222.73 us         168825       WRITE
      0.00       0.00 us       0.00 us       0.00 us        1966855      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        1957919     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us           2325   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           8936   CI_FORGET
 
    Duration: 789 seconds
   Data Read: 18677760 bytes
Data Written: 9485570217 bytes


Gluster7.0 (Ctime enabled)
==========================

fs-dl380-c1-n1:~ # gluster vo profile repo1 info
Brick: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                23977                143222                 10220 
 
   Block Size:                512b+                1024b+                8192b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2067                    51                     0 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     8                   171 
No. of Writes:                   26                  2954                     0 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   50                   480                  5002 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1841 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6590      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7432     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     129.79 us     114.92 us     140.65 us              3    SETXATTR
      0.00     386.13 us     331.50 us     440.75 us              2    READDIRP
      0.00     292.58 us     249.60 us     360.18 us              3     XATTROP
      0.00     576.10 us     446.41 us     775.50 us              3       MKDIR
      0.00    1857.67 us     820.07 us    2405.74 us              3       RMDIR
      0.00      33.78 us      14.96 us     120.77 us            216         IPC
      0.02      73.47 us      11.54 us     315.66 us            412   FSETXATTR
      0.02     169.11 us      43.46 us     405.47 us            188        READ
      0.04      69.85 us       9.31 us    2344.93 us            906     READDIR
      0.05      79.66 us       7.34 us     267.55 us           1151    GETXATTR
      0.06     123.14 us      28.68 us     517.29 us            846        OPEN
      0.32      85.70 us      25.91 us     399.66 us           6594   FGETXATTR
      0.48     107.57 us      25.52 us   14326.25 us           7828      STATFS
      0.49      43.59 us      11.85 us    2791.32 us          19801       FLUSH
      0.50      62.56 us      13.76 us    1445.20 us          14181    FINODELK
      0.67     177.44 us      56.71 us     687.67 us           6594       FSTAT
      0.92      61.48 us      11.24 us    2399.86 us          26408     ENTRYLK
      1.31     315.70 us      96.90 us     915.84 us           7305     SETATTR
      1.97     525.68 us     173.64 us    3103.01 us           6595      CREATE
      2.20     277.03 us      68.90 us   13473.77 us          13942    FXATTROP
      2.70     214.72 us       9.56 us    2757.47 us          22114      LOOKUP
      3.01     802.37 us     182.07 us    5259.34 us           6594      UNLINK
      3.08     101.50 us       0.65 us    4039.32 us          53271     OPENDIR
     36.60    4043.67 us      15.93 us  994013.08 us          15909     INODELK
     45.55     421.68 us      64.99 us    9380.82 us         189890       WRITE
      0.00       0.00 us       0.00 us       0.00 us        3677977      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3671345     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             36   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6632   CI_FORGET
 
    Duration: 5698 seconds
   Data Read: 22977840 bytes
Data Written: 9841829137 bytes
 
Interval 0 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                23977                143222                 10220 
 
   Block Size:                512b+                1024b+                8192b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2067                    51                     0 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     8                   171 
No. of Writes:                   26                  2954                     0 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   50                   480                  5002 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1841 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6590      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7432     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     129.79 us     114.92 us     140.65 us              3    SETXATTR
      0.00     386.13 us     331.50 us     440.75 us              2    READDIRP
      0.00     292.58 us     249.60 us     360.18 us              3     XATTROP
      0.00     576.10 us     446.41 us     775.50 us              3       MKDIR
      0.00    1857.67 us     820.07 us    2405.74 us              3       RMDIR
      0.00      33.78 us      14.96 us     120.77 us            216         IPC
      0.02      73.47 us      11.54 us     315.66 us            412   FSETXATTR
      0.02     169.11 us      43.46 us     405.47 us            188        READ
      0.04      69.85 us       9.31 us    2344.93 us            906     READDIR
      0.05      79.66 us       7.34 us     267.55 us           1151    GETXATTR
      0.06     123.14 us      28.68 us     517.29 us            846        OPEN
      0.32      85.70 us      25.91 us     399.66 us           6594   FGETXATTR
      0.48     107.70 us      25.52 us   14326.25 us           7828      STATFS
      0.49      43.59 us      11.85 us    2791.32 us          19801       FLUSH
      0.50      62.56 us      13.76 us    1445.20 us          14181    FINODELK
      0.67     177.44 us      56.71 us     687.67 us           6594       FSTAT
      0.92      61.48 us      11.24 us    2399.86 us          26408     ENTRYLK
      1.31     315.70 us      96.90 us     915.84 us           7305     SETATTR
      1.97     525.68 us     173.64 us    3103.01 us           6595      CREATE
      2.20     277.03 us      68.90 us   13473.77 us          13942    FXATTROP
      2.70     214.72 us       9.56 us    2757.47 us          22114      LOOKUP
      3.01     802.37 us     182.07 us    5259.34 us           6594      UNLINK
      3.08     101.50 us       0.65 us    4039.32 us          53271     OPENDIR
     36.60    4043.67 us      15.93 us  994013.08 us          15909     INODELK
     45.55     421.68 us      64.99 us    9380.82 us         189890       WRITE
      0.00       0.00 us       0.00 us       0.00 us        3677977      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3671345     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             36   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6632   CI_FORGET
 
    Duration: 5698 seconds
   Data Read: 22977840 bytes
Data Written: 9841829137 bytes
 
Brick: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    1                    24                  3023 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  320                     0                     0 
No. of Writes:                    0                    41                   436 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4856                  1768 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6605      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7075     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     138.84 us     112.92 us     160.96 us              3    SETXATTR
      0.00     399.64 us     344.42 us     454.87 us              2    READDIRP
      0.00     571.87 us     398.94 us     844.47 us              3       MKDIR
      0.01    2045.86 us     777.17 us    2709.66 us              3       RMDIR
      0.01      35.10 us      12.94 us      93.54 us            216         IPC
      0.06      72.26 us      12.14 us     334.52 us            432   FSETXATTR
      0.10     113.49 us      26.00 us     484.88 us            473        OPEN
      0.10     174.90 us      87.44 us     453.04 us            324        READ
      0.11      65.07 us      11.03 us    4413.29 us            907     READDIR
      0.11      67.32 us      13.89 us    1223.39 us            918    GETXATTR
      1.06      87.26 us      26.59 us     194.28 us           6606   FGETXATTR
      1.34      93.17 us      20.71 us    8258.97 us           7828      STATFS
      1.37      56.19 us      14.38 us     588.75 us          13240    FINODELK
      1.74      47.75 us       9.44 us    1288.06 us          19775       FLUSH
      2.27     186.16 us      58.73 us    1957.66 us           6606       FSTAT
      2.90      59.56 us      15.95 us    2550.84 us          26442     ENTRYLK
      4.20     310.72 us      92.26 us     920.97 us           7338     SETATTR
      6.31     517.93 us     194.94 us    3140.66 us           6606      CREATE
      6.84     279.99 us      71.20 us    2480.15 us          13240    FXATTROP
      8.44     214.97 us      15.99 us    3525.82 us          21284      LOOKUP
      9.83     807.18 us     187.73 us    3510.05 us           6606      UNLINK
      9.91     100.86 us       1.15 us    2942.26 us          53271     OPENDIR
     19.45    1039.19 us     126.76 us    5976.65 us          10149       WRITE
     23.82     873.49 us      13.82 us  996890.14 us          14792     INODELK
      0.00       0.00 us       0.00 us       0.00 us           6707      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             82     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             21   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6625   CI_FORGET
 
    Duration: 5698 seconds
   Data Read: 41943040 bytes
Data Written: 9455665152 bytes
 
Interval 0 Stats:
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    1                    24                  3023 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  320                     0                     0 
No. of Writes:                    0                    41                   436 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4856                  1768 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6605      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7075     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     138.84 us     112.92 us     160.96 us              3    SETXATTR
      0.00     399.64 us     344.42 us     454.87 us              2    READDIRP
      0.00     571.87 us     398.94 us     844.47 us              3       MKDIR
      0.01    2045.86 us     777.17 us    2709.66 us              3       RMDIR
      0.01      35.10 us      12.94 us      93.54 us            216         IPC
      0.06      72.26 us      12.14 us     334.52 us            432   FSETXATTR
      0.10     113.49 us      26.00 us     484.88 us            473        OPEN
      0.10     174.90 us      87.44 us     453.04 us            324        READ
      0.11      65.07 us      11.03 us    4413.29 us            907     READDIR
      0.11      67.32 us      13.89 us    1223.39 us            918    GETXATTR
      1.06      87.26 us      26.59 us     194.28 us           6606   FGETXATTR
      1.34      93.17 us      20.71 us    8258.97 us           7828      STATFS
      1.37      56.19 us      14.38 us     588.75 us          13240    FINODELK
      1.74      47.75 us       9.44 us    1288.06 us          19775       FLUSH
      2.27     186.16 us      58.73 us    1957.66 us           6606       FSTAT
      2.90      59.56 us      15.95 us    2550.84 us          26442     ENTRYLK
      4.20     310.72 us      92.26 us     920.97 us           7338     SETATTR
      6.31     517.93 us     194.94 us    3140.66 us           6606      CREATE
      6.84     279.99 us      71.20 us    2480.15 us          13240    FXATTROP
      8.44     214.97 us      15.99 us    3525.82 us          21284      LOOKUP
      9.83     807.18 us     187.73 us    3510.05 us           6606      UNLINK
      9.91     100.86 us       1.15 us    2942.26 us          53271     OPENDIR
     19.45    1039.20 us     126.76 us    5976.65 us          10149       WRITE
     23.82     873.49 us      13.82 us  996890.14 us          14792     INODELK
      0.00       0.00 us       0.00 us       0.00 us           6707      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             82     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             21   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6625   CI_FORGET
 
    Duration: 5698 seconds
   Data Read: 41943040 bytes
Data Written: 9455665152 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                23977                143222                 10220 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2067                    51                     0 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     1                   170 
No. of Writes:                   26                  2954                     0 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   50                   480                  5002 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1841 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            944      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7430     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     134.44 us      75.05 us     193.89 us              3    SETXATTR
      0.00     268.64 us     259.90 us     278.55 us              3     XATTROP
      0.00     439.49 us     203.80 us     682.65 us              3       MKDIR
      0.00    1853.60 us     823.20 us    2393.82 us              3       RMDIR
      0.00      34.05 us      13.77 us     101.66 us            215         IPC
      0.00      50.12 us      19.28 us     294.14 us            410   FSETXATTR
      0.00     176.78 us      47.52 us     268.32 us            173        READ
      0.00      67.16 us       9.59 us    4569.92 us            906     READDIR
      0.00     135.83 us      38.87 us   18143.17 us            837        OPEN
      0.02      76.76 us      23.00 us     192.51 us           6594   FGETXATTR
      0.04      65.77 us      10.26 us   19823.69 us          14227    FINODELK
      0.04      68.49 us      11.99 us   18024.04 us          15909     INODELK
      0.05     155.32 us      16.76 us   21625.26 us           7828      STATFS
      0.08     142.52 us      37.26 us    4159.63 us          14358       FSTAT
      0.08     296.31 us      85.27 us    2190.78 us           7305     SETATTR
      0.13     509.05 us     179.26 us    6280.47 us           6595      CREATE
      0.15     274.40 us      79.84 us   13303.23 us          13942    FXATTROP
      0.18     233.24 us       8.82 us   53288.69 us          19801       FLUSH
      0.19     738.29 us     191.82 us   26804.67 us           6594      UNLINK
      0.44     416.08 us      11.09 us   31352.67 us          27762      LOOKUP
      0.55     551.61 us      10.11 us   46117.06 us          26408     ENTRYLK
      0.74     367.23 us       0.64 us   34196.75 us          53271     OPENDIR
      2.73     372.74 us       8.60 us   61916.08 us         192531    GETXATTR
      4.06     560.96 us      62.01 us   62168.04 us         189890       WRITE
     90.53   13985.76 us      15.94 us   60188.98 us         170026    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        3672463      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3671509     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             37   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            954   CI_FORGET
 
    Duration: 5691 seconds
   Data Read: 22379229 bytes
Data Written: 9841829137 bytes
 
Interval 0 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                23977                143222                 10220 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2067                    51                     0 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                    0                     1                   170 
No. of Writes:                   26                  2954                     0 
 
   Block Size:             262144b+              524288b+             1048576b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   50                   480                  5002 
 
   Block Size:            2097152b+ 
 No. of Reads:                    0 
No. of Writes:                 1841 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            944      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7430     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     134.44 us      75.05 us     193.89 us              3    SETXATTR
      0.00     268.64 us     259.90 us     278.55 us              3     XATTROP
      0.00     439.49 us     203.80 us     682.65 us              3       MKDIR
      0.00    1853.60 us     823.20 us    2393.82 us              3       RMDIR
      0.00      34.05 us      13.77 us     101.66 us            215         IPC
      0.00      50.12 us      19.28 us     294.14 us            410   FSETXATTR
      0.00     176.78 us      47.52 us     268.32 us            173        READ
      0.00      67.16 us       9.59 us    4569.92 us            906     READDIR
      0.00     135.83 us      38.87 us   18143.17 us            837        OPEN
      0.02      76.76 us      23.00 us     192.51 us           6594   FGETXATTR
      0.04      65.77 us      10.26 us   19823.69 us          14227    FINODELK
      0.04      68.49 us      11.99 us   18024.04 us          15909     INODELK
      0.05     155.32 us      16.76 us   21625.26 us           7828      STATFS
      0.08     142.52 us      37.26 us    4159.63 us          14358       FSTAT
      0.08     296.31 us      85.27 us    2190.78 us           7305     SETATTR
      0.13     509.05 us     179.26 us    6280.47 us           6595      CREATE
      0.15     274.40 us      79.84 us   13303.23 us          13942    FXATTROP
      0.18     233.24 us       8.82 us   53288.69 us          19801       FLUSH
      0.19     738.29 us     191.82 us   26804.67 us           6594      UNLINK
      0.44     416.08 us      11.09 us   31352.67 us          27762      LOOKUP
      0.55     551.61 us      10.11 us   46117.06 us          26408     ENTRYLK
      0.74     367.23 us       0.64 us   34196.75 us          53271     OPENDIR
      2.73     372.74 us       8.60 us   61916.08 us         192531    GETXATTR
      4.06     560.96 us      62.01 us   62168.04 us         189890       WRITE
     90.53   13985.90 us      15.94 us   60188.98 us         170026    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        3672463      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3671509     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             37   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            954   CI_FORGET
 
    Duration: 5691 seconds
   Data Read: 22379229 bytes
Data Written: 9841829137 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     7 
No. of Writes:                    1                    24                  3023 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  320                     0                     0 
No. of Writes:                    0                    41                   436 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4856                  1768 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            249      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7069     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     163.77 us     143.80 us     187.63 us              3    SETXATTR
      0.00     539.28 us     199.35 us     909.49 us              3       MKDIR
      0.00    1861.11 us     734.86 us    2433.60 us              3       RMDIR
      0.00      34.80 us      14.08 us     120.65 us            215         IPC
      0.00      47.22 us      16.75 us     265.51 us            426   FSETXATTR
      0.00     105.54 us      34.69 us     438.87 us            474        OPEN
      0.00      58.10 us       9.99 us    2784.86 us            906     READDIR
      0.00     193.84 us      78.00 us    1070.84 us            338        READ
      0.01      78.11 us      27.12 us     267.42 us           6606   FGETXATTR
      0.02      62.29 us      15.49 us    2094.30 us          13240    FINODELK
      0.03      71.48 us      13.41 us    2627.65 us          14792     INODELK
      0.03     146.99 us      13.72 us   18406.23 us           7828      STATFS
      0.06     144.89 us      37.17 us    1728.42 us          14262       FSTAT
      0.06     294.14 us      95.08 us    1298.49 us           7338     SETATTR
      0.09     501.65 us     171.12 us    3673.36 us           6606      CREATE
      0.10     277.18 us      83.58 us    2255.78 us          13240    FXATTROP
      0.11     618.23 us     182.22 us   31669.04 us           6606      UNLINK
      0.19     247.20 us      12.45 us   56440.26 us          27624      LOOKUP
      0.19     269.37 us       9.01 us   29850.52 us          26442     ENTRYLK
      0.26     942.72 us     125.40 us   11143.54 us          10149       WRITE
      0.52     569.83 us       8.92 us    3526.39 us          33889    GETXATTR
      1.14    2122.15 us       8.75 us  170951.32 us          19775       FLUSH
      1.33     920.75 us       0.81 us   27584.76 us          53271     OPENDIR
     95.86   23866.45 us      76.91 us   66659.49 us         148246    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            341      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             91     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             30   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            250   CI_FORGET
 
    Duration: 5691 seconds
   Data Read: 42401792 bytes
Data Written: 9455665152 bytes
 
Interval 0 Stats:
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     7 
No. of Writes:                    1                    24                  3023 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  320                     0                     0 
No. of Writes:                    0                    41                   436 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4856                  1768 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            249      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7069     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53292  RELEASEDIR
      0.00     163.77 us     143.80 us     187.63 us              3    SETXATTR
      0.00     539.28 us     199.35 us     909.49 us              3       MKDIR
      0.00    1861.11 us     734.86 us    2433.60 us              3       RMDIR
      0.00      34.80 us      14.08 us     120.65 us            215         IPC
      0.00      47.22 us      16.75 us     265.51 us            426   FSETXATTR
      0.00     105.54 us      34.69 us     438.87 us            474        OPEN
      0.00      58.10 us       9.99 us    2784.86 us            906     READDIR
      0.00     193.84 us      78.00 us    1070.84 us            338        READ
      0.01      78.11 us      27.12 us     267.42 us           6606   FGETXATTR
      0.02      62.29 us      15.49 us    2094.30 us          13240    FINODELK
      0.03      71.48 us      13.41 us    2627.65 us          14792     INODELK
      0.03     146.99 us      13.72 us   18406.23 us           7828      STATFS
      0.06     144.89 us      37.17 us    1728.42 us          14262       FSTAT
      0.06     294.14 us      95.08 us    1298.49 us           7338     SETATTR
      0.09     501.65 us     171.12 us    3673.36 us           6606      CREATE
      0.10     277.18 us      83.58 us    2255.78 us          13240    FXATTROP
      0.11     618.23 us     182.22 us   31669.04 us           6606      UNLINK
      0.19     247.20 us      12.45 us   56440.26 us          27624      LOOKUP
      0.19     269.37 us       9.01 us   29850.52 us          26442     ENTRYLK
      0.26     942.72 us     125.40 us   11143.54 us          10149       WRITE
      0.52     569.83 us       8.92 us    3526.39 us          33889    GETXATTR
      1.14    2122.15 us       8.75 us  170951.32 us          19775       FLUSH
      1.33     920.75 us       0.81 us   27584.76 us          53271     OPENDIR
     95.86   23866.52 us      76.91 us   66659.49 us         148246    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            341      UPCALL
      0.00       0.00 us       0.00 us       0.00 us             91     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             30   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            250   CI_FORGET
 
    Duration: 5691 seconds
   Data Read: 42401792 bytes
Data Written: 9455665152 bytes

Gluster7.0 (Ctime disabled)
==========================

fs-dl380-c1-n1:~ # gluster vo profile repo1 info
Brick: fs-dl380-c1-n1:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:              16384b+               32768b+               65536b+ 
 No. of Reads:                    1                     0                     0 
No. of Writes:                    1                     4                  2964 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  228                     0                     0 
No. of Writes:                    4                    34                   567 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 5050                  1776 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            600      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7331     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53273  RELEASEDIR
      0.00     109.56 us     105.03 us     116.42 us              3    SETXATTR
      0.00    1659.72 us     821.79 us    2507.43 us              3       RMDIR
      0.00      34.48 us      12.81 us     139.35 us            212         IPC
      0.00      59.20 us      11.67 us     221.08 us            671   FSETXATTR
      0.00     218.16 us      78.14 us    1035.71 us            192        READ
      0.00      48.56 us       8.54 us    2820.26 us            884     READDIR
      0.00      84.66 us      25.29 us     287.73 us            736        OPEN
      0.00   22983.11 us     349.98 us   68209.96 us              3       MKDIR
      0.02      77.66 us      26.99 us     838.99 us           6594   FGETXATTR
      0.03      62.21 us      13.19 us     749.60 us          13234    FINODELK
      0.05     146.98 us      12.26 us   13924.50 us           7684      STATFS
      0.08     135.23 us      34.54 us     639.31 us          14446       FSTAT
      0.12     276.14 us      90.40 us    2438.14 us          10408     SETATTR
      0.13     238.80 us      69.29 us     936.48 us          13234    FXATTROP
      0.13     160.35 us       9.27 us   32737.36 us          19800       FLUSH
      0.19     673.60 us     164.31 us   20215.27 us           6594      UNLINK
      0.25     272.08 us       9.69 us   21821.00 us          21801      LOOKUP
      0.31    1104.92 us     183.89 us   91797.25 us           6594      CREATE
      0.31     278.52 us      10.21 us   28080.27 us          26406     ENTRYLK
      0.44     994.10 us     101.78 us    5068.01 us          10400       WRITE
      0.45     200.58 us       0.56 us   25870.75 us          53260     OPENDIR
      0.74     825.58 us      13.04 us 1000704.27 us          21033     INODELK
      2.23     273.73 us       7.12 us   45532.47 us         192388    GETXATTR
     94.49   13133.21 us      14.57 us   58588.95 us         169539    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            798      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            183     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             35   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            615   CI_FORGET
 
    Duration: 5511 seconds
   Data Read: 29910289 bytes
Data Written: 9811918848 bytes
 
Interval 0 Stats:
   Block Size:              16384b+               32768b+               65536b+ 
 No. of Reads:                    1                     0                     0 
No. of Writes:                    1                     4                  2964 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  228                     0                     0 
No. of Writes:                    4                    34                   567 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 5050                  1776 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            600      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7331     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53273  RELEASEDIR
      0.00     109.56 us     105.03 us     116.42 us              3    SETXATTR
      0.00    1659.72 us     821.79 us    2507.43 us              3       RMDIR
      0.00      34.48 us      12.81 us     139.35 us            212         IPC
      0.00      59.20 us      11.67 us     221.08 us            671   FSETXATTR
      0.00     218.16 us      78.14 us    1035.71 us            192        READ
      0.00      48.56 us       8.54 us    2820.26 us            884     READDIR
      0.00      84.66 us      25.29 us     287.73 us            736        OPEN
      0.00   22983.11 us     349.98 us   68209.96 us              3       MKDIR
      0.02      77.66 us      26.99 us     838.99 us           6594   FGETXATTR
      0.03      62.21 us      13.19 us     749.60 us          13234    FINODELK
      0.05     146.97 us      12.26 us   13924.50 us           7684      STATFS
      0.08     135.23 us      34.54 us     639.31 us          14446       FSTAT
      0.12     276.14 us      90.40 us    2438.14 us          10408     SETATTR
      0.13     238.80 us      69.29 us     936.48 us          13234    FXATTROP
      0.13     160.35 us       9.27 us   32737.36 us          19800       FLUSH
      0.19     673.60 us     164.31 us   20215.27 us           6594      UNLINK
      0.25     272.08 us       9.69 us   21821.00 us          21801      LOOKUP
      0.31    1104.92 us     183.89 us   91797.25 us           6594      CREATE
      0.31     278.52 us      10.21 us   28080.27 us          26406     ENTRYLK
      0.44     994.10 us     101.78 us    5068.01 us          10400       WRITE
      0.45     200.58 us       0.56 us   25870.75 us          53260     OPENDIR
      0.74     825.58 us      13.04 us 1000704.27 us          21033     INODELK
      2.23     273.73 us       7.12 us   45532.47 us         192388    GETXATTR
     94.49   13133.21 us      14.57 us   58588.95 us         169539    READDIRP
      0.00       0.00 us       0.00 us       0.00 us            798      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            183     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             35   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            615   CI_FORGET
 
    Duration: 5511 seconds
   Data Read: 29910289 bytes
Data Written: 9811918848 bytes
 
Brick: fs-dl380-c1-n1:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                20467                149369                  7938 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2007                   108                    20 
 
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    1                     0                     8 
No. of Writes:                    0                     4                  3033 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  406                     0                     0 
No. of Writes:                    3                    30                   535 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4913                  1693 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            252      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7739     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53273  RELEASEDIR
      0.00     133.82 us     112.34 us     171.91 us              3    SETXATTR
      0.00     471.81 us     281.35 us     584.13 us              3       MKDIR
      0.00    1854.68 us     711.96 us    2576.95 us              3       RMDIR
      0.00      36.88 us      14.03 us     127.56 us            212         IPC
      0.00      48.45 us      11.55 us    1037.28 us            687   FSETXATTR
      0.00      60.07 us       9.80 us    2853.63 us            884     READDIR
      0.00     168.55 us      34.75 us    1271.44 us            427        READ
      0.00      88.55 us      25.87 us    2524.01 us           1143        OPEN
      0.01      77.73 us      21.82 us     260.61 us           6606   FGETXATTR
      0.03      73.04 us      12.26 us   18818.68 us          14222    FINODELK
      0.03     159.41 us      16.51 us   12216.58 us           7684      STATFS
      0.06     138.52 us      35.06 us    2143.63 us          14399       FSTAT
      0.08     267.16 us      89.52 us    3110.29 us          10356     SETATTR
      0.10     248.62 us      75.81 us   14682.87 us          13976    FXATTROP
      0.10     562.03 us     147.59 us   16529.03 us           6606      UNLINK
      0.14     223.88 us      11.29 us   18317.24 us          22520      LOOKUP
      0.14     197.77 us      10.47 us   25278.58 us          26444     ENTRYLK
      0.15     843.89 us     183.22 us   64931.11 us           6607      CREATE
      0.55     581.94 us       7.99 us   14737.86 us          34131    GETXATTR
      0.56    1024.53 us       7.55 us  124416.11 us          19779       FLUSH
      0.86     580.75 us       0.64 us   26090.30 us          53260     OPENDIR
      1.81    2970.85 us      12.74 us  998791.97 us          22037     INODELK
      2.93     556.81 us      49.85 us  124973.59 us         190120       WRITE
     92.45   22765.88 us      85.84 us   71157.38 us         146904    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        3688995      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3688735     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             39   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            260   CI_FORGET
 
    Duration: 5511 seconds
   Data Read: 53769736 bytes
Data Written: 9485599852 bytes
 
Interval 0 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                20467                149369                  7938 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 2007                   108                    20 
 
   Block Size:               4096b+               32768b+               65536b+ 
 No. of Reads:                    1                     0                     8 
No. of Writes:                    0                     4                  3033 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  406                     0                     0 
No. of Writes:                    3                    30                   535 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4913                  1693 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            252      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7739     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53273  RELEASEDIR
      0.00     133.82 us     112.34 us     171.91 us              3    SETXATTR
      0.00     471.81 us     281.35 us     584.13 us              3       MKDIR
      0.00    1854.68 us     711.96 us    2576.95 us              3       RMDIR
      0.00      36.88 us      14.03 us     127.56 us            212         IPC
      0.00      48.45 us      11.55 us    1037.28 us            687   FSETXATTR
      0.00      60.07 us       9.80 us    2853.63 us            884     READDIR
      0.00     168.55 us      34.75 us    1271.44 us            427        READ
      0.00      88.55 us      25.87 us    2524.01 us           1143        OPEN
      0.01      77.73 us      21.82 us     260.61 us           6606   FGETXATTR
      0.03      73.04 us      12.26 us   18818.68 us          14222    FINODELK
      0.03     159.41 us      16.51 us   12216.58 us           7684      STATFS
      0.06     138.52 us      35.06 us    2143.63 us          14399       FSTAT
      0.08     267.16 us      89.52 us    3110.29 us          10356     SETATTR
      0.10     248.62 us      75.81 us   14682.87 us          13976    FXATTROP
      0.10     562.03 us     147.59 us   16529.03 us           6606      UNLINK
      0.14     223.88 us      11.29 us   18317.24 us          22520      LOOKUP
      0.14     197.77 us      10.47 us   25278.58 us          26444     ENTRYLK
      0.15     843.89 us     183.22 us   64931.11 us           6607      CREATE
      0.55     581.94 us       7.99 us   14737.86 us          34131    GETXATTR
      0.56    1024.53 us       7.55 us  124416.11 us          19779       FLUSH
      0.86     580.75 us       0.64 us   26090.30 us          53260     OPENDIR
      1.81    2970.85 us      12.74 us  998791.97 us          22037     INODELK
      2.93     556.81 us      49.85 us  124973.59 us         190120       WRITE
     92.45   22765.97 us      85.84 us   71157.38 us         146904    READDIRP
      0.00       0.00 us       0.00 us       0.00 us        3688995      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3688735     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             39   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us            260   CI_FORGET
 
    Duration: 5511 seconds
   Data Read: 53769736 bytes
Data Written: 9485599852 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick1/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:              16384b+               32768b+               65536b+ 
 No. of Reads:                    1                     0                     0 
No. of Writes:                    1                     4                  2964 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  228                     0                     0 
No. of Writes:                    4                    34                   567 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 5050                  1776 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6597      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7321     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53269  RELEASEDIR
      0.00     321.09 us     321.09 us     321.09 us              1    READDIRP
      0.00     148.18 us     123.64 us     181.27 us              3    SETXATTR
      0.01    1893.28 us     839.08 us    2464.71 us              3       RMDIR
      0.02      35.11 us      15.03 us      94.68 us            211         IPC
      0.11     225.85 us      90.38 us    1822.97 us            219        READ
      0.12      82.33 us      21.81 us     343.90 us            661   FSETXATTR
      0.13      65.70 us      12.22 us    2296.45 us            884     READDIR
      0.14      71.80 us      11.86 us     442.88 us            891    GETXATTR
      0.16   24100.47 us     399.39 us   71483.81 us              3       MKDIR
      0.21     130.93 us      39.97 us     544.82 us            726        OPEN
      1.25      86.56 us      28.62 us     191.50 us           6594   FGETXATTR
      1.68      58.02 us      14.39 us     627.77 us          13234    FINODELK
      1.93     114.37 us      28.65 us    4695.76 us           7684      STATFS
      2.22      51.18 us       9.72 us     625.99 us          19800       FLUSH
      2.48     171.78 us      46.71 us    1924.88 us           6594       FSTAT
      2.70      58.61 us      14.54 us     564.40 us          21033     INODELK
      3.62      62.50 us      14.57 us    1698.46 us          26406     ENTRYLK
      6.47     283.41 us      88.20 us    3009.70 us          10408     SETATTR
      7.17     247.13 us      58.80 us    2843.12 us          13234    FXATTROP
      9.85     208.74 us      13.05 us    2780.55 us          21520      LOOKUP
     10.95     756.62 us     190.35 us    3731.91 us           6594      UNLINK
     12.13     103.81 us       0.85 us    2546.53 us          53260     OPENDIR
     14.51    1003.21 us     195.60 us   68228.17 us           6594      CREATE
     22.12     969.65 us     106.46 us    5111.14 us          10400       WRITE
      0.00       0.00 us       0.00 us       0.00 us           6826      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            182     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             33   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6644   CI_FORGET
 
    Duration: 5415 seconds
   Data Read: 29910289 bytes
Data Written: 9811918848 bytes
 
Interval 0 Stats:
   Block Size:              16384b+               32768b+               65536b+ 
 No. of Reads:                    1                     0                     0 
No. of Writes:                    1                     4                  2964 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  228                     0                     0 
No. of Writes:                    4                    34                   567 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 5050                  1776 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6597      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7321     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53269  RELEASEDIR
      0.00     321.09 us     321.09 us     321.09 us              1    READDIRP
      0.00     148.18 us     123.64 us     181.27 us              3    SETXATTR
      0.01    1893.28 us     839.08 us    2464.71 us              3       RMDIR
      0.02      35.11 us      15.03 us      94.68 us            211         IPC
      0.11     225.85 us      90.38 us    1822.97 us            219        READ
      0.12      82.33 us      21.81 us     343.90 us            661   FSETXATTR
      0.13      65.70 us      12.22 us    2296.45 us            884     READDIR
      0.14      71.80 us      11.86 us     442.88 us            891    GETXATTR
      0.16   24100.47 us     399.39 us   71483.81 us              3       MKDIR
      0.21     130.93 us      39.97 us     544.82 us            726        OPEN
      1.25      86.56 us      28.62 us     191.50 us           6594   FGETXATTR
      1.68      58.02 us      14.39 us     627.77 us          13234    FINODELK
      1.93     114.42 us      28.65 us    4695.76 us           7684      STATFS
      2.22      51.18 us       9.72 us     625.99 us          19800       FLUSH
      2.48     171.78 us      46.71 us    1924.88 us           6594       FSTAT
      2.70      58.61 us      14.54 us     564.40 us          21033     INODELK
      3.62      62.50 us      14.57 us    1698.46 us          26406     ENTRYLK
      6.47     283.41 us      88.20 us    3009.70 us          10408     SETATTR
      7.17     247.13 us      58.80 us    2843.12 us          13234    FXATTROP
      9.85     208.74 us      13.05 us    2780.55 us          21520      LOOKUP
     10.94     756.62 us     190.35 us    3731.91 us           6594      UNLINK
     12.13     103.81 us       0.85 us    2546.53 us          53260     OPENDIR
     14.51    1003.21 us     195.60 us   68228.17 us           6594      CREATE
     22.12     969.65 us     106.46 us    5111.14 us          10400       WRITE
      0.00       0.00 us       0.00 us       0.00 us           6826      UPCALL
      0.00       0.00 us       0.00 us       0.00 us            182     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             33   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6644   CI_FORGET
 
    Duration: 5415 seconds
   Data Read: 29910289 bytes
Data Written: 9811918848 bytes
 
Brick: fs-dl380-c1-n2:/gluster/brick2/glusterbrick
--------------------------------------------------
Cumulative Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                20467                149369                  7938 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 2007                   108                    20 
 
   Block Size:               8192b+               32768b+               65536b+ 
 No. of Reads:                    1                     0                     7 
No. of Writes:                    0                     4                  3033 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  439                     0                     0 
No. of Writes:                    3                    30                   535 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4913                  1693 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6600      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7724     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53269  RELEASEDIR
      0.00     349.86 us     349.86 us     349.86 us              1    READDIRP
      0.00     116.74 us     101.44 us     126.75 us              3    SETXATTR
      0.00     558.42 us     316.86 us     742.85 us              3       MKDIR
      0.01    1865.44 us     662.50 us    2484.75 us              3       RMDIR
      0.01      37.88 us      17.37 us     159.84 us            211         IPC
      0.05      83.17 us      15.79 us     356.85 us            672   FSETXATTR
      0.06      77.13 us      13.87 us    2684.70 us            884     READDIR
      0.07     166.87 us      70.77 us     355.51 us            457        READ
      0.08      80.32 us      11.98 us    2209.62 us           1044    GETXATTR
      0.13     128.16 us      38.45 us     578.51 us           1128        OPEN
      0.51      83.63 us      25.96 us     271.98 us           6606   FGETXATTR
      0.72      54.96 us      13.52 us     346.33 us          14270    FINODELK
      0.86      47.02 us      10.73 us    2290.97 us          19779       FLUSH
      0.89     125.70 us      23.32 us   10003.42 us           7684      STATFS
      1.02     168.22 us      49.32 us     345.41 us           6606       FSTAT
      1.19      58.56 us      12.76 us    2668.50 us          22037     INODELK
      1.39      57.11 us      12.82 us     528.26 us          26444     ENTRYLK
      2.68     281.43 us      89.43 us    1381.86 us          10356     SETATTR
      3.17     246.41 us      64.92 us    1308.98 us          13976    FXATTROP
      4.26     208.86 us      16.05 us    2519.78 us          22152      LOOKUP
      4.41     725.07 us     157.72 us    3577.05 us           6606      UNLINK
      4.92     100.32 us       0.83 us    2926.52 us          53260     OPENDIR
      5.84     960.84 us     188.08 us   68080.79 us           6607      CREATE
     67.74     387.21 us      58.68 us    6087.15 us         190120       WRITE
      0.00       0.00 us       0.00 us       0.00 us        3695372      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3688727     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             38   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6645   CI_FORGET
 
    Duration: 5415 seconds
   Data Read: 58018517 bytes
Data Written: 9485599852 bytes
 
Interval 0 Stats:
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                20467                149369                  7938 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 2007                   108                    20 
 
   Block Size:               8192b+               32768b+               65536b+ 
 No. of Reads:                    1                     0                     7 
No. of Writes:                    0                     4                  3033 
 
   Block Size:             131072b+              262144b+              524288b+ 
 No. of Reads:                  439                     0                     0 
No. of Writes:                    3                    30                   535 
 
   Block Size:            1048576b+             2097152b+ 
 No. of Reads:                    0                     0 
No. of Writes:                 4913                  1693 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           6600      FORGET
      0.00       0.00 us       0.00 us       0.00 us           7724     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          53269  RELEASEDIR
      0.00     349.86 us     349.86 us     349.86 us              1    READDIRP
      0.00     116.74 us     101.44 us     126.75 us              3    SETXATTR
      0.00     558.42 us     316.86 us     742.85 us              3       MKDIR
      0.01    1865.44 us     662.50 us    2484.75 us              3       RMDIR
      0.01      37.88 us      17.37 us     159.84 us            211         IPC
      0.05      83.17 us      15.79 us     356.85 us            672   FSETXATTR
      0.06      77.13 us      13.87 us    2684.70 us            884     READDIR
      0.07     166.87 us      70.77 us     355.51 us            457        READ
      0.08      80.32 us      11.98 us    2209.62 us           1044    GETXATTR
      0.13     128.16 us      38.45 us     578.51 us           1128        OPEN
      0.51      83.63 us      25.96 us     271.98 us           6606   FGETXATTR
      0.72      54.96 us      13.52 us     346.33 us          14270    FINODELK
      0.86      47.02 us      10.73 us    2290.97 us          19779       FLUSH
      0.89     125.52 us      23.32 us   10003.42 us           7684      STATFS
      1.02     168.22 us      49.32 us     345.41 us           6606       FSTAT
      1.19      58.56 us      12.76 us    2668.50 us          22037     INODELK
      1.39      57.11 us      12.82 us     528.26 us          26444     ENTRYLK
      2.68     281.43 us      89.43 us    1381.86 us          10356     SETATTR
      3.17     246.41 us      64.92 us    1308.98 us          13976    FXATTROP
      4.26     208.86 us      16.05 us    2519.78 us          22152      LOOKUP
      4.41     725.07 us     157.72 us    3577.05 us           6606      UNLINK
      4.92     100.32 us       0.83 us    2926.52 us          53260     OPENDIR
      5.84     960.84 us     188.08 us   68080.79 us           6607      CREATE
     67.74     387.21 us      58.68 us    6087.15 us         190120       WRITE
      0.00       0.00 us       0.00 us       0.00 us        3695372      UPCALL
      0.00       0.00 us       0.00 us       0.00 us        3688727     CI_IATT
      0.00       0.00 us       0.00 us       0.00 us             38   CI_UNLINK
      0.00       0.00 us       0.00 us       0.00 us           6645   CI_FORGET
 
    Duration: 5415 seconds
   Data Read: 58018517 bytes
Data Written: 9485599852 bytes



More information about the Gluster-users mailing list