[Gluster-users] Performance drop when upgrading from 3.8 to 6.5

Riccardo Murri riccardo.murri at gmail.com
Wed Nov 6 08:04:49 UTC 2019


Dear Rafi, all,

please find attached two profile files; both are profiling the same command:
```
time rsync -a $SRC root at 172.23.187.207:/glusterfs
```

In both cases, the target is a Ubuntu 16.04 VM mounting a pure
distributed GlusterFS 7 filesystem on `/glusterfs`.  The GlusterFS 7
cluster is comprised of 3 identical Ubuntu 16.04 VMs, each with one
brick of 200GB.  I have turned ctime off, as suggested in a previous
email.

In the case of file `profile1.txt` (larger file), $SRC is a directory
tree containing ~94'500 files, collectively weighing ~376GB.  The
transfer takes between 250 and 300 minutes (I've made several attempts
now), for an average bandwidth of ~21MB/s.

In the case of `profile3.txt`, $SRC is a single tar file weighing
72GB.  It takes between 16 and 30 minutes to write it into the
GlusterFS 7 filesystem; average bandwidth is ~60MB/s.

To me, this seems to indicate that, while write performance on data is
good, metadata ops on GlusterFS 7 are rather slow, and much slower
than in the 3.x series.  Is there any other tweak that I may try to
apply?

Thanks,
Riccardo
-------------- next part --------------
Brick: server001:/srv/glusterfs
-------------------------------
Cumulative Stats:
   Block Size:                  2b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    4                     2                     9 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   32                    28                    37 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  108                   297                   575 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  984                  1994                  4334 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 8304                 16146                 33163 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                64431               4173687                   207 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         114999      FORGET
      0.00       0.00 us       0.00 us       0.00 us         130690     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1886  RELEASEDIR
      0.00     449.84 us     222.38 us    2924.41 us             31       MKNOD
      0.01     177.17 us     104.16 us     436.34 us            281        LINK
      0.01     128.41 us      44.77 us    2512.16 us            414    SETXATTR
      0.01      82.37 us      29.26 us    4792.82 us           1319       FSTAT
      0.02     414.20 us     181.30 us   10818.54 us            414       MKDIR
      0.02     741.77 us     103.38 us  158364.81 us            281      UNLINK
      0.29      77.67 us      17.57 us    4887.93 us          34118      STATFS
      0.37      50.29 us      10.83 us    7341.33 us          67134       FLUSH
      0.73      46.83 us       9.41 us    7990.78 us         140764     ENTRYLK
      0.85     219.73 us      75.32 us  125748.37 us          35106      RENAME
      1.21      51.71 us      10.32 us   10816.56 us         211798     INODELK
      1.33     377.95 us      38.65 us  150160.77 us          31778   FTRUNCATE
      1.64      97.82 us      23.85 us    6386.67 us         151947        STAT
      3.25     123.40 us      35.48 us   43775.28 us         238449     SETATTR
      4.31     581.91 us     123.91 us  261522.99 us          67134      CREATE
      7.30     137.63 us      23.45 us  153825.73 us         480472      LOOKUP
     78.65     321.99 us      32.95 us  169393.74 us        2211891       WRITE
 
    Duration: 156535 seconds
   Data Read: 0 bytes
Data Written: 555615327151 bytes
 
Interval 2 Stats:
   Block Size:                  2b+                   8b+                  16b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    3                     6                    24 
 
   Block Size:                 32b+                  64b+                 128b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   21                    27                    78 
 
   Block Size:                256b+                 512b+                1024b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  216                   420                   720 
 
   Block Size:               2048b+                4096b+                8192b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 1452                  3144                  6047 
 
   Block Size:              16384b+               32768b+               65536b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                11709                 24154                 47132 
 
   Block Size:             131072b+              262144b+ 
 No. of Reads:                    0                     0 
No. of Writes:              3043313                   203 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          95934      FORGET
      0.00       0.00 us       0.00 us       0.00 us          95334     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1886  RELEASEDIR
      0.00     115.60 us     115.60 us     115.60 us              1       FSTAT
      0.01     174.33 us      27.06 us    1320.67 us            160     ENTRYLK
      0.01     156.56 us      70.31 us     424.24 us            207    SETXATTR
      0.01     224.20 us      26.01 us    8305.15 us            320     INODELK
      0.02     494.80 us     272.11 us    3374.68 us            207       MKDIR
      0.31      98.46 us      22.81 us    4887.93 us          17615      STATFS
      0.36      62.66 us      14.29 us    7341.33 us          31778       FLUSH
      0.39     173.25 us      44.04 us    6386.67 us          12582        STAT
      2.17     377.95 us      38.65 us  150160.77 us          31778   FTRUNCATE
      2.37     136.70 us      36.35 us    8403.28 us          96163     SETATTR
      4.04     704.67 us     147.20 us  236913.74 us          31778      CREATE
      4.79     178.55 us      34.10 us  153825.73 us         148553      LOOKUP
     85.52     452.80 us      32.95 us  169393.74 us        1046218       WRITE
 
    Duration: 131024 seconds
   Data Read: 0 bytes
Data Written: 405160719969 bytes
 
Brick: server003:/srv/glusterfs
-------------------------------
Cumulative Stats:
   Block Size:                  2b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    4                    10                     4 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   18                    34                    48 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  120                   333                   635 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 1033                  1987                  4121 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 8155                 16236                 32801 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                63586               4141322                   316 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         113618      FORGET
      0.00       0.00 us       0.00 us       0.00 us         129309     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1886  RELEASEDIR
      0.00     495.23 us     288.06 us     804.27 us             41       MKNOD
      0.00     217.75 us      93.47 us     577.50 us            195        LINK
      0.00     225.43 us      73.01 us    2978.27 us            195      UNLINK
      0.01     150.04 us      44.00 us     871.24 us            414    SETXATTR
      0.02     517.76 us     180.55 us   22051.74 us            414       MKDIR
      0.25      93.72 us      25.26 us   43085.29 us          25159       FSTAT
      0.33      90.33 us      20.18 us   55535.26 us          34118      STATFS
      0.40      57.47 us      11.52 us   52544.91 us          66527       FLUSH
      0.92     248.34 us      80.04 us   90073.94 us          34982      RENAME
      1.06      71.62 us       9.24 us   11879.40 us         140152     ENTRYLK
      1.18     355.94 us      40.54 us  162449.67 us          31391   FTRUNCATE
      1.68      75.53 us      11.64 us   48217.75 us         210662     INODELK
      1.83     135.33 us      23.52 us   52342.27 us         127896        STAT
      3.48     139.16 us      31.07 us   54695.32 us         236418     SETATTR
      4.70     667.59 us     145.12 us  330303.47 us          66527      CREATE
      8.39     165.77 us      26.57 us   71983.81 us         478113      LOOKUP
     75.71     325.22 us      30.87 us  163014.93 us        2198443       WRITE
 
    Duration: 156534 seconds
   Data Read: 0 bytes
Data Written: 551285938615 bytes
 
Interval 2 Stats:
   Block Size:                  2b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    3                     9                     3 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   12                    24                    36 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   87                   243                   459 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  744                  1452                  3000 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 5928                 11841                 23831 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                46362               3014215                   214 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          94770      FORGET
      0.00       0.00 us       0.00 us       0.00 us          94173     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1886  RELEASEDIR
      0.00      87.19 us      66.28 us     132.42 us              4       FSTAT
      0.01     245.35 us      28.27 us    4816.51 us            118     ENTRYLK
      0.01     158.85 us      44.99 us     871.24 us            207    SETXATTR
      0.01     203.65 us      16.68 us    2946.41 us            236     INODELK
      0.02     504.48 us     180.55 us    1760.64 us            207       MKDIR
      0.37      99.75 us      20.18 us   55535.26 us          17615      STATFS
      0.38      56.72 us      12.16 us   52544.91 us          31391       FLUSH
      0.54     201.78 us      38.63 us   19489.60 us          12565        STAT
      2.37     355.94 us      40.54 us  162449.67 us          31391   FTRUNCATE
      2.67     132.20 us      31.07 us   54695.32 us          95002     SETATTR
      4.76     714.36 us     145.12 us  203552.82 us          31391      CREATE
      5.87     187.58 us      28.41 us   71983.81 us         147354      LOOKUP
     82.99     377.03 us      30.87 us  163014.93 us        1036143       WRITE
 
    Duration: 131024 seconds
   Data Read: 0 bytes
Data Written: 401247747261 bytes
 
Brick: server002:/srv/glusterfs
-------------------------------
Cumulative Stats:
   Block Size:                  1b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    4                     4                    19 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   26                    30                    67 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  116                   314                   593 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  991                  1735                  3881 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 7500                 14118                 30129 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                58629               4375220                   323 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         102675      FORGET
      0.00       0.00 us       0.00 us       0.00 us         118366     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1886  RELEASEDIR
      0.00     181.93 us     146.36 us     276.38 us             29        LINK
      0.00     187.28 us     123.07 us     330.46 us             32      UNLINK
      0.01     139.56 us      54.74 us     920.32 us            414    SETXATTR
      0.02     464.84 us     235.49 us    2327.06 us            436       MKNOD
      0.02     519.98 us     227.86 us   15214.02 us            414       MKDIR
      0.15      81.91 us      28.59 us   12976.99 us          17350       FSTAT
      0.31      84.29 us      19.20 us    6974.33 us          34118      STATFS
      0.34      57.06 us      11.32 us    9630.03 us          55522       FLUSH
      0.66     252.53 us      92.44 us  196497.81 us          24504      RENAME
      0.72      68.66 us      11.35 us   25278.45 us          98280     ENTRYLK
      1.11     331.48 us      33.51 us  140821.71 us          31422   FTRUNCATE
      1.15      73.45 us      10.57 us   18541.44 us         146748     INODELK
      1.49     111.72 us      25.23 us   22871.35 us         124788        STAT
      2.81     136.67 us      32.13 us   70865.47 us         192761     SETATTR
      3.50     591.07 us     154.01 us  190605.55 us          55522      CREATE
      6.91     164.63 us      26.50 us  160376.45 us         393766      LOOKUP
     80.81     313.61 us      36.26 us  190240.74 us        2416631       WRITE
 
    Duration: 156534 seconds
   Data Read: 0 bytes
Data Written: 581279350602 bytes
 
Interval 2 Stats:
   Block Size:                  1b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    3                     3                    15 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   21                    24                    51 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   93                   249                   474 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  792                  1383                  3108 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 5994                 11325                 24084 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                46490               3021260                   227 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          94484      FORGET
      0.00       0.00 us       0.00 us       0.00 us          94266     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           1886  RELEASEDIR
      0.00      67.79 us      67.67 us      67.92 us              2       FSTAT
      0.00     170.82 us      26.14 us    1003.43 us            136     ENTRYLK
      0.01     152.02 us      71.30 us     394.37 us            207    SETXATTR
      0.01     174.43 us      25.51 us    1770.47 us            272     INODELK
      0.02     505.75 us     256.81 us    5435.91 us            207       MKDIR
      0.35      96.17 us      20.31 us    6974.33 us          17615      STATFS
      0.39      59.63 us      13.88 us    7094.70 us          31422       FLUSH
      0.45     171.01 us      38.78 us    8237.50 us          12577        STAT
      2.17     331.48 us      33.51 us  140821.71 us          31422   FTRUNCATE
      2.71     136.71 us      32.13 us   68465.46 us          95095     SETATTR
      4.24     647.39 us     154.01 us  190605.55 us          31422      CREATE
      5.54     180.17 us      31.44 us   18103.81 us         147471      LOOKUP
     84.09     388.10 us      36.26 us  144188.99 us        1038528       WRITE
 
    Duration: 131024 seconds
   Data Read: 0 bytes
Data Written: 402204102246 bytes
 
-------------- next part --------------
Brick: server001:/srv/glusterfs
-------------------------------
Cumulative Stats:
   Block Size:                  2b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    4                     2                     9 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   32                    28                    37 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  108                   297                   575 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  984                  1994                  4334 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 8304                 16146                 33163 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                64431               4173687                   207 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         131518      FORGET
      0.00       0.00 us       0.00 us       0.00 us         130690     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           2585  RELEASEDIR
      0.00    1318.42 us    1318.42 us    1318.42 us              1     OPENDIR
      0.00     990.20 us     133.01 us    1847.39 us              2    READDIRP
      0.00     449.84 us     222.38 us    2924.41 us             31       MKNOD
      0.01     177.17 us     104.16 us     436.34 us            281        LINK
      0.01     128.41 us      44.77 us    2512.16 us            414    SETXATTR
      0.01      82.37 us      29.26 us    4792.82 us           1319       FSTAT
      0.02     414.20 us     181.30 us   10818.54 us            414       MKDIR
      0.02     741.77 us     103.38 us  158364.81 us            281      UNLINK
      0.29      77.67 us      17.57 us    4887.93 us          34120      STATFS
      0.37      50.29 us      10.83 us    7341.33 us          67134       FLUSH
      0.73      46.83 us       9.41 us    7990.78 us         140764     ENTRYLK
      0.85     219.73 us      75.32 us  125748.37 us          35106      RENAME
      1.21      51.71 us      10.32 us   10816.56 us         211798     INODELK
      1.33     377.95 us      38.65 us  150160.77 us          31778   FTRUNCATE
      1.64      97.82 us      23.85 us    6386.67 us         151947        STAT
      3.25     123.40 us      35.48 us   43775.28 us         238449     SETATTR
      4.31     581.91 us     123.91 us  261522.99 us          67134      CREATE
      7.30     137.67 us      23.45 us  153825.73 us         480502      LOOKUP
     78.64     321.99 us      32.95 us  169393.74 us        2211891       WRITE
 
    Duration: 167281 seconds
   Data Read: 0 bytes
Data Written: 555615327151 bytes
 
Interval 3 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          16519      FORGET
      0.00       0.00 us       0.00 us       0.00 us            699  RELEASEDIR
      1.12     127.24 us     115.79 us     138.68 us              2      STATFS
      5.81    1318.42 us    1318.42 us    1318.42 us              1     OPENDIR
      8.73     990.20 us     133.01 us    1847.39 us              2    READDIRP
     84.33     637.63 us      71.20 us    4184.80 us             30      LOOKUP
 
    Duration: 10746 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: server003:/srv/glusterfs
-------------------------------
Cumulative Stats:
   Block Size:                  2b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    4                    10                     4 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   18                    34                    48 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  120                   333                   635 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 1033                  1987                  4121 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 8155                 16236                 32801 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                63586               4141322                   316 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         130137      FORGET
      0.00       0.00 us       0.00 us       0.00 us         129309     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           2585  RELEASEDIR
      0.00    2914.70 us    2914.70 us    2914.70 us              1     OPENDIR
      0.00    3606.43 us    3606.43 us    3606.43 us              1    READDIRP
      0.00     495.23 us     288.06 us     804.27 us             41       MKNOD
      0.00     217.75 us      93.47 us     577.50 us            195        LINK
      0.00     225.43 us      73.01 us    2978.27 us            195      UNLINK
      0.01     150.04 us      44.00 us     871.24 us            414    SETXATTR
      0.02     517.76 us     180.55 us   22051.74 us            414       MKDIR
      0.25      93.72 us      25.26 us   43085.29 us          25159       FSTAT
      0.33      90.33 us      20.18 us   55535.26 us          34120      STATFS
      0.40      57.47 us      11.52 us   52544.91 us          66527       FLUSH
      0.92     248.34 us      80.04 us   90073.94 us          34982      RENAME
      1.06      71.62 us       9.24 us   11879.40 us         140152     ENTRYLK
      1.18     355.94 us      40.54 us  162449.67 us          31391   FTRUNCATE
      1.68      75.53 us      11.64 us   48217.75 us         210662     INODELK
      1.83     135.33 us      23.52 us   52342.27 us         127896        STAT
      3.48     139.16 us      31.07 us   54695.32 us         236418     SETATTR
      4.70     667.59 us     145.12 us  330303.47 us          66527      CREATE
      8.40     165.82 us      26.57 us   71983.81 us         478144      LOOKUP
     75.71     325.22 us      30.87 us  163014.93 us        2198443       WRITE
 
    Duration: 167280 seconds
   Data Read: 0 bytes
Data Written: 551285938615 bytes
 
Interval 3 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          16519      FORGET
      0.00       0.00 us       0.00 us       0.00 us            699  RELEASEDIR
      0.75     135.31 us     106.56 us     164.07 us              2      STATFS
      8.04    2914.70 us    2914.70 us    2914.70 us              1     OPENDIR
      9.94    3606.43 us    3606.43 us    3606.43 us              1    READDIRP
     81.27     950.80 us      70.24 us    8269.82 us             31      LOOKUP
 
    Duration: 10746 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: server002:/srv/glusterfs
-------------------------------
Cumulative Stats:
   Block Size:                  1b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    4                     4                    19 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                   26                    30                    67 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  116                   314                   593 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  991                  1735                  3881 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 7500                 14119                 30130 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                58629               5545227                   403 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         119195      FORGET
      0.00       0.00 us       0.00 us       0.00 us         118368     RELEASE
      0.00       0.00 us       0.00 us       0.00 us           2585  RELEASEDIR
      0.00    2477.61 us    2477.61 us    2477.61 us              1     OPENDIR
      0.00    1598.42 us      85.72 us    3111.12 us              2    READDIRP
      0.00     181.93 us     146.36 us     276.38 us             29        LINK
      0.00     187.28 us     123.07 us     330.46 us             32      UNLINK
      0.00     139.56 us      54.74 us     920.32 us            414    SETXATTR
      0.01     464.84 us     235.49 us    2327.06 us            436       MKNOD
      0.01     519.98 us     227.86 us   15214.02 us            414       MKDIR
      0.10      81.91 us      28.59 us   12976.99 us          17350       FSTAT
      0.20      84.34 us      19.20 us    6974.33 us          34120      STATFS
      0.22      57.06 us      11.32 us    9630.03 us          55523       FLUSH
      0.43     252.53 us      92.44 us  196497.81 us          24504      RENAME
      0.47      68.66 us      11.35 us   25278.45 us          98280     ENTRYLK
      0.72     331.48 us      33.51 us  140821.71 us          31423   FTRUNCATE
      0.75      73.45 us      10.57 us   18541.44 us         146748     INODELK
      0.96     111.72 us      25.23 us   22871.35 us         124788        STAT
      1.82     136.67 us      32.13 us   70865.47 us         192763     SETATTR
      2.27     591.24 us     154.01 us  190605.55 us          55524      CREATE
      4.49     164.69 us      26.50 us  160376.45 us         393812      LOOKUP
     87.55     352.87 us      36.26 us  448101.34 us        3586720       WRITE
 
    Duration: 167280 seconds
   Data Read: 0 bytes
Data Written: 734655823690 bytes
 
Interval 3 Stats:
   Block Size:              16384b+               32768b+              131072b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    1                     1               1170007 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   80 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          16520      FORGET
      0.00       0.00 us       0.00 us       0.00 us              2     RELEASE
      0.00       0.00 us       0.00 us       0.00 us            699  RELEASEDIR
      0.00      64.94 us      64.94 us      64.94 us              1       FLUSH
      0.00     349.55 us     349.55 us     349.55 us              1   FTRUNCATE
      0.00     436.78 us     333.18 us     540.39 us              2     SETATTR
      0.00     937.45 us     293.95 us    1580.94 us              2      STATFS
      0.00    2477.61 us    2477.61 us    2477.61 us              1     OPENDIR
      0.00    1598.42 us      85.72 us    3111.12 us              2    READDIRP
      0.00    5236.53 us    1352.56 us    9120.49 us              2      CREATE
      0.01     678.27 us      59.41 us    9202.17 us             46      LOOKUP
     99.99     433.94 us      86.61 us  448101.34 us        1170089       WRITE
 
    Duration: 10746 seconds
   Data Read: 0 bytes
Data Written: 153376473088 bytes
 


More information about the Gluster-users mailing list