[Gluster-users] glusterfsd process spinning

Franco Broi franco.broi at iongeo.com
Sun Jun 1 05:23:51 UTC 2014


The volume is almost completely idle now and the CPU for the brick
process has returned to normal. I've included the profile and I think it
shows the latency for the bad brick (data12) is unusually high, probably
indicating the filesystem is at fault after all??

On Sun, 2014-06-01 at 01:01 -0400, Pranith Kumar Karampuri wrote: 
> Franco,
>     Could you do the following to get more information:
> 
> "gluster volume profile <volname> start"
> 
> Wait for some time, this will start gathering what operations are coming to all the bricks"
> Now execute "gluster volume profile <volname> info" > /file/you/should/reply/to/this/mail/with
> 
> Then execute:
> gluster volume profile <volname> stop
> 
> Lets see if this throws any light on the problem at hand
> 
> Pranith
> ----- Original Message -----
> > From: "Franco Broi" <franco.broi at iongeo.com>
> > To: gluster-users at gluster.org
> > Sent: Sunday, June 1, 2014 9:02:48 AM
> > Subject: [Gluster-users] glusterfsd process spinning
> > 
> > Hi
> > 
> > I've been suffering from continual problems with my gluster filesystem
> > slowing down due to what I thought was congestion on a single brick
> > being caused by a problem with the underlying filesystem running slow
> > but I've just noticed that the glusterfsd process for that particular
> > brick is running at 100%+, even when the filesystem is almost idle.
> > 
> > I've done a couple of straces of the brick and another on the same
> > server, does the high number of futex errors give any clues as to what
> > might be wrong?
> > 
> > % time     seconds  usecs/call     calls    errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 45.58    0.027554           0    191665     20772 futex
> > 28.26    0.017084           0    137133           readv
> > 26.04    0.015743           0     66259           epoll_wait
> >   0.13    0.000077           3        23           writev
> >   0.00    0.000000           0         1           epoll_ctl
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00    0.060458                395081     20772 total
> > 
> > % time     seconds  usecs/call     calls    errors syscall
> > ------ ----------- ----------- --------- --------- ----------------
> > 99.25    0.334020         133      2516           epoll_wait
> >   0.40    0.001347           0      4090        26 futex
> >   0.35    0.001192           0      5064           readv
> >   0.00    0.000000           0        20           writev
> > ------ ----------- ----------- --------- --------- ----------------
> > 100.00    0.336559                 11690        26 total
> > 
> > 
> > 
> > Cheers,
> > 
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > 

-------------- next part --------------
Brick: nas3-10g:/data10/gvol
----------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  151                   240             313821945 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                  634                    19                    15 
No. of Writes:             11210623              45720500              24490029 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                   22                 17006                   950 
No. of Writes:            992498404            1024176715            1055850217 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                  954                  3561                  2564 
No. of Writes:            597572813             225090885             227894690 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              5826398               2161421               1101178 
No. of Writes:            398642340             174916651              47685643 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:              1749653              53735599            1098632674 
No. of Writes:             10906391              11251050              23780900 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   99 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         108512      FORGET
      0.00       0.00 us       0.00 us       0.00 us       26578056     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422494  RELEASEDIR
      0.00      51.00 us      51.00 us      51.00 us              1      ACCESS
      0.00      61.00 us      61.00 us      61.00 us              1        OPEN
      0.28      37.33 us      15.00 us     127.00 us            322        STAT
      0.47      40.70 us      13.00 us      94.00 us            492       FLUSH
      1.90   81542.00 us   81542.00 us   81542.00 us              1      CREATE
      2.18      46.26 us      18.00 us     119.00 us           2024      STATFS
      7.17      94.84 us      37.00 us     250.00 us           3251       WRITE
     16.39     329.89 us      38.00 us  131946.00 us           2136      LOOKUP
     71.62    1083.06 us      53.00 us  130116.00 us           2844        READ
 
    Duration: 1644260 seconds
   Data Read: 147664953639565 bytes
Data Written: 12749788231214 bytes
 
Interval 2 Stats:
   Block Size:             131072b+ 
 No. of Reads:                  490 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              8     RELEASE
      0.82      41.71 us      19.00 us      65.00 us             96       FLUSH
      1.27      33.68 us      15.00 us     127.00 us            184        STAT
      1.76      44.53 us      19.00 us     119.00 us            193      STATFS
      6.74     101.27 us      52.00 us     217.00 us            325      LOOKUP
     89.42     891.45 us      53.00 us  123191.00 us            490        READ
 
    Duration: 96 seconds
   Data Read: 64225280 bytes
Data Written: 0 bytes
 
Brick: nas3-10g:/data12/gvol
----------------------------
Cumulative Stats:
   Block Size:                  1b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    7             436253984               4926880 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    1                     0                     1 
No. of Writes:              1606681                  5947                  2506 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                  720                     6                    14 
No. of Writes:              7649781              30681943               4496119 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                   73                    52                  9825 
No. of Writes:              2777352               2097176              56527057 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:               484366                 11934                114306 
No. of Writes:             20436877               9192065               2441283 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:               186673             144740142                     0 
No. of Writes:              4296733               3896350                    16 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us            783      FORGET
      0.00       0.00 us       0.00 us       0.00 us         554946     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          46014  RELEASEDIR
      0.00      33.19 us      10.00 us      91.00 us             16      ACCESS
      0.02      46.22 us      15.00 us     120.00 us            322        STAT
      0.08      45.78 us      12.00 us     108.00 us           1102       FLUSH
      0.18      56.27 us      21.00 us     292.00 us           2024      STATFS
      1.29     354.23 us      43.00 us  113589.00 us           2294      LOOKUP
     14.40     100.54 us      33.00 us     576.00 us          89962       WRITE
     84.02    4839.60 us      48.00 us  705200.00 us          10905        READ
 
    Duration: 273953 seconds
   Data Read: 18992872093553 bytes
Data Written: 1969722263215 bytes
 
Interval 2 Stats:
   Block Size:             131072b+ 
 No. of Reads:                  714 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             18     RELEASE
      0.03      31.00 us      10.00 us      91.00 us             12      ACCESS
      0.55      39.12 us      15.00 us     113.00 us            184        STAT
      0.67      44.83 us      21.00 us     114.00 us            193      STATFS
      0.67      40.13 us      14.00 us      75.00 us            216       FLUSH
      2.66      96.76 us      46.00 us     266.00 us            357      LOOKUP
     95.43    1738.16 us      52.00 us  230583.00 us            714        READ
 
    Duration: 96 seconds
   Data Read: 93585408 bytes
Data Written: 0 bytes
 
Brick: nas3-10g:/data9/gvol
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     5                     0 
No. of Writes:                  226                    38                382035 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                  503                     1                    26 
No. of Writes:                 8121              44024302              29347327 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                   23                 14499                   696 
No. of Writes:           1097991782            1082664231             908683224 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                  562                  1538                  1359 
No. of Writes:            490505979             198697761             283101385 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              3475245                717394                516582 
No. of Writes:            434127250             182773820              47503801 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:              1285964              72630385            1093265270 
No. of Writes:             10739639              11453329              34706166 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                  103 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         100526      FORGET
      0.00       0.00 us       0.00 us       0.00 us       26961620     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422499  RELEASEDIR
      0.01      33.27 us      11.00 us      46.00 us             11      ACCESS
      0.41      50.28 us      15.00 us     109.00 us            322        STAT
      0.53      42.90 us      12.00 us      99.00 us            480       FLUSH
      2.73      52.71 us      20.00 us     134.00 us           2024      STATFS
     20.75     380.28 us      38.00 us  121585.00 us           2135      LOOKUP
     75.57    1068.90 us      50.00 us  243959.00 us           2766        READ
 
    Duration: 1644260 seconds
   Data Read: 148144416725315 bytes
Data Written: 14520329792942 bytes
 
Interval 2 Stats:
   Block Size:             131072b+ 
 No. of Reads:                  511 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              8     RELEASE
      0.07      29.88 us      11.00 us      44.00 us              8      ACCESS
      1.17      40.70 us      18.00 us      68.00 us             96       FLUSH
      2.54      46.32 us      15.00 us     109.00 us            184        STAT
      2.88      49.96 us      22.00 us     121.00 us            193      STATFS
     10.61     109.34 us      48.00 us     262.00 us            325      LOOKUP
     82.73     542.40 us      61.00 us   60856.00 us            511        READ
 
    Duration: 96 seconds
   Data Read: 66977792 bytes
Data Written: 0 bytes
 
Brick: nas3-10g:/data11/gvol
----------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  166                    18                432965 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                  415                    10                    22 
No. of Writes:                 8114              44848404              49704831 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                   46                493741                  1325 
No. of Writes:           1013779056             788776921             791567418 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                  756                  1828                  1810 
No. of Writes:            421897696             193233666             241937475 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              2232414               1165468                346570 
No. of Writes:            425717230             167837566              45478150 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:              1527982              37900085             925191366 
No. of Writes:             10730607               8915356              21685151 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                  110 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          82647      FORGET
      0.00       0.00 us       0.00 us       0.00 us       21556792     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422493  RELEASEDIR
      0.00      23.83 us      10.00 us      43.00 us             12      ACCESS
      0.17      39.05 us      14.00 us     102.00 us            322        STAT
      0.55      43.08 us      13.00 us     110.00 us            960       FLUSH
      1.21      44.83 us      18.00 us     126.00 us           2024      STATFS
      1.64      47.36 us      16.00 us     115.00 us           2603       FSTAT
      8.64     303.41 us      39.00 us   73264.00 us           2134      LOOKUP
      8.91      79.36 us      35.00 us     182.00 us           8414       WRITE
     78.88    2104.41 us      49.00 us  701544.00 us           2810        READ
 
    Duration: 1644260 seconds
   Data Read: 123844112929707 bytes
Data Written: 11977554070196 bytes
 
Interval 2 Stats:
   Block Size:                 16b+                 512b+                2048b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  590                    84                  1166 
 
   Block Size:               4096b+              131072b+ 
 No. of Reads:                    0                   446 
No. of Writes:                    6                     0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             16     RELEASE
      0.04      20.89 us      10.00 us      43.00 us              9      ACCESS
      1.27      33.66 us      14.00 us      91.00 us            184        STAT
      1.63      41.56 us      17.00 us      63.00 us            192       FLUSH
      1.64      41.70 us      20.00 us     110.00 us            193      STATFS
      5.51      46.10 us      18.00 us      74.00 us            585       FSTAT
      6.73     101.37 us      43.00 us     330.00 us            325      LOOKUP
     29.23      77.50 us      37.00 us     137.00 us           1846       WRITE
     53.96     592.26 us      49.00 us  114792.00 us            446        READ
 
    Duration: 96 seconds
   Data Read: 58458112 bytes
Data Written: 3675120 bytes
 
Brick: nas2-10g:/data6/gvol
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  174                    10                 70244 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                   50                     0                     5 
No. of Writes:                 5201               6014165               8506259 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    3                 11745                   651 
No. of Writes:           1434815362             702223997             503658696 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                  121                   476                   342 
No. of Writes:            147148463              83746530              79063284 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              4253441               1519343                562779 
No. of Writes:            189219348              68756300              19102239 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:               458648              17234860             659534440 
No. of Writes:              3670738               2242352               6425894 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   77 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          50581      FORGET
      0.00       0.00 us       0.00 us       0.00 us       14025563     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422483  RELEASEDIR
      0.01      46.20 us      36.00 us      75.00 us              5      ACCESS
      0.64      46.98 us      13.00 us     120.00 us            322        STAT
      0.88      43.21 us      11.00 us      99.00 us            480       FLUSH
      4.14      48.52 us      16.00 us     130.00 us           2024      STATFS
     44.41     495.23 us      32.00 us   61768.00 us           2126      LOOKUP
     49.92     529.94 us      49.00 us  146637.00 us           2233        READ
 
    Duration: 1642269 seconds
   Data Read: 87641300019913 bytes
Data Written: 4639013858093 bytes
 
Interval 2 Stats:
   Block Size:             131072b+ 
 No. of Reads:                  365 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              8     RELEASE
      0.09      50.33 us      36.00 us      75.00 us              3      ACCESS
      2.20      38.66 us      11.00 us      61.00 us             96       FLUSH
      4.81      44.11 us      13.00 us     102.00 us            184        STAT
      5.10      44.63 us      17.00 us      89.00 us            193      STATFS
     17.30      89.82 us      36.00 us     189.00 us            325      LOOKUP
     70.50     325.92 us      49.00 us   30989.00 us            365        READ
 
    Duration: 96 seconds
   Data Read: 47841280 bytes
Data Written: 0 bytes
 
Brick: nas2-10g:/data7/gvol
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  127                     7                 74738 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                   32                     0                     0 
No. of Writes:                 7917               6326619              22661985 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    2                 12193                   695 
No. of Writes:             21919447             121465603             507089230 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                  101                  1312                   491 
No. of Writes:             98551947              99287896              69556748 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:               364897               1287122                 30135 
No. of Writes:            162759130              42628532              11803784 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:               208091              15161747             695588501 
No. of Writes:              3882169               2502859               4392089 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   97 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          51107      FORGET
      0.00       0.00 us       0.00 us       0.00 us       11669143     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422488  RELEASEDIR
      0.00      65.00 us      65.00 us      65.00 us              1        OPEN
      0.01      49.00 us      14.00 us      91.00 us             16      ACCESS
      0.28      45.42 us      13.00 us     109.00 us            322        STAT
      0.44   23310.00 us   23310.00 us   23310.00 us              1      CREATE
      0.75      40.94 us      12.00 us     101.00 us            972       FLUSH
      1.93      50.64 us      16.00 us     153.00 us           2024      STATFS
     18.59     463.21 us      34.00 us  153463.00 us           2129      LOOKUP
     20.48      78.82 us      30.00 us     209.00 us          13782       WRITE
     57.52     935.01 us      44.00 us  107746.00 us           3263        READ
 
    Duration: 1642269 seconds
   Data Read: 92189487962936 bytes
Data Written: 3589594156624 bytes
 
Interval 2 Stats:
   Block Size:                256b+                 512b+                1024b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 6887                  4986                   140 
 
   Block Size:             131072b+ 
 No. of Reads:                  511 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             16     RELEASE
      0.03      55.75 us      14.00 us      91.00 us              8      ACCESS
      0.48      38.26 us      17.00 us      77.00 us            192       FLUSH
      0.52      43.19 us      13.00 us     109.00 us            184        STAT
      0.60      47.59 us      16.00 us     115.00 us            193      STATFS
      2.08      98.00 us      34.00 us     265.00 us            325      LOOKUP
     35.34    1059.94 us      44.00 us  107746.00 us            511        READ
     60.95      77.75 us      30.00 us     193.00 us          12013       WRITE
 
    Duration: 96 seconds
   Data Read: 66977792 bytes
Data Written: 6497280 bytes
 
Brick: nas2-10g:/data5/gvol
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  157                    63                 94918 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                   69                     1                     1 
No. of Writes:                 2032               6946372              15138774 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    4                 10350                   616 
No. of Writes:            673853459             415954645             462803157 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                   84                   436                   237 
No. of Writes:             91307202              75997986              43601841 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              1384551               1503179                315362 
No. of Writes:            152102275              42970573              11506162 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:               409211              11698640             656491132 
No. of Writes:              3414508               1959650               6498862 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                  129 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          48867      FORGET
      0.00       0.00 us       0.00 us       0.00 us       10202393     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422484  RELEASEDIR
      0.03      45.00 us      23.00 us      94.00 us             35      ACCESS
      0.25      46.58 us      12.00 us     154.00 us            322        STAT
      0.51      41.26 us      14.00 us     101.00 us            731       FLUSH
      1.69      49.61 us      16.00 us     125.00 us           2024      STATFS
     12.21     654.70 us      47.00 us  132377.00 us           1105        READ
     12.92     360.17 us      33.00 us   46417.00 us           2126      LOOKUP
     72.39      92.78 us      28.00 us     233.00 us          46230       WRITE
 
    Duration: 1642269 seconds
   Data Read: 86858503076859 bytes
Data Written: 3670162546832 bytes
 
Interval 2 Stats:
   Block Size:             131072b+ 
 No. of Reads:                   88 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             13     RELEASE
      1.06      40.75 us      23.00 us      71.00 us             16      ACCESS
     10.05      39.83 us      14.00 us      89.00 us            155       FLUSH
     12.16      40.58 us      12.00 us      87.00 us            184        STAT
     12.66      88.34 us      54.00 us     240.00 us             88        READ
     14.45      45.96 us      16.00 us      79.00 us            193      STATFS
     49.61      93.72 us      35.00 us     165.00 us            325      LOOKUP
 
    Duration: 96 seconds
   Data Read: 11534336 bytes
Data Written: 0 bytes
 
Brick: nas2-10g:/data8/gvol
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     2 
No. of Writes:                  141                     8                 65118 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                   48                     0                     4 
No. of Writes:                 4802               6760117              28814469 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    7                 10507                   625 
No. of Writes:             23235333             124052110             477018875 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                   72                   454                   269 
No. of Writes:            100740601              77797389              63701386 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              4799964               2133846                448385 
No. of Writes:            155757464              45770857              10418699 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:              1093962              26191376             656875259 
No. of Writes:              2756571               1784738               5124758 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   88 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          49393      FORGET
      0.00       0.00 us       0.00 us       0.00 us       12888692     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422479  RELEASEDIR
      0.00      62.00 us      62.00 us      62.00 us              1        OPEN
      0.02      47.64 us      11.00 us      93.00 us             11      ACCESS
      0.45   13366.00 us   13366.00 us   13366.00 us              1      CREATE
      0.50      45.46 us      12.00 us     112.00 us            322        STAT
      0.75      50.07 us      23.00 us     112.00 us            440    READLINK
      0.87      42.01 us      13.00 us      98.00 us            612       FLUSH
      3.38      49.33 us      17.00 us     127.00 us           2024      STATFS
      5.53      90.01 us      29.00 us     177.00 us           1815       WRITE
     31.95     418.34 us      33.00 us  101223.00 us           2258      LOOKUP
     56.57     678.00 us      49.00 us  311465.00 us           2467        READ
 
    Duration: 1642269 seconds
   Data Read: 87911734109028 bytes
Data Written: 3450048479478 bytes
 
Interval 2 Stats:
   Block Size:                256b+              131072b+ 
 No. of Reads:                    0                   369 
No. of Writes:                   14                     0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             11     RELEASE
      0.09      42.88 us      11.00 us      74.00 us              8      ACCESS
      0.31      83.50 us      68.00 us     128.00 us             14       WRITE
      1.15      54.65 us      23.00 us     112.00 us             80    READLINK
      1.72      49.68 us      14.00 us      98.00 us            131       FLUSH
      2.04      41.97 us      12.00 us     107.00 us            184        STAT
      2.31      45.49 us      19.00 us     111.00 us            193      STATFS
      8.36      92.14 us      35.00 us     383.00 us            344      LOOKUP
     84.03     863.67 us      50.00 us  142787.00 us            369        READ
 
    Duration: 96 seconds
   Data Read: 48365568 bytes
Data Written: 5264 bytes
 
Brick: nas4-10g:/data13/gvol
----------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  124                    11                 60682 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                    0                     0                     4 
No. of Writes:                  793              10025717              10374704 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    2                  1553                    20 
No. of Writes:              1884073               8729975              90222445 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                   68                   183                    96 
No. of Writes:             14709941              11216938               8118854 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                16557                 13443                 14207 
No. of Writes:             95258335              28252430               8459854 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:                52993                471640             204527123 
No. of Writes:              2759266               1514093               7579155 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   77 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           4661      FORGET
      0.00       0.00 us       0.00 us       0.00 us        1439364     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          99546  RELEASEDIR
      0.00      33.80 us      13.00 us      45.00 us              5      ACCESS
      0.01      38.71 us      11.00 us     106.00 us            322        STAT
      0.03      38.96 us       6.00 us     118.00 us            960       FLUSH
      0.08      51.00 us      17.00 us     124.00 us           2024      STATFS
      0.49     516.72 us      46.00 us  150624.00 us           1182        READ
      0.64     378.49 us      35.00 us   78068.00 us           2126      LOOKUP
     15.63      58.73 us       9.00 us    1207.00 us         333723       FSTAT
     83.12     102.33 us      29.00 us    1739.00 us        1018542       WRITE
 
    Duration: 424409 seconds
   Data Read: 26841972052360 bytes
Data Written: 2563585335420 bytes
 
Interval 2 Stats:
   Block Size:                 16b+                  32b+                4096b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                76341                150580                  8664 
 
   Block Size:               8192b+               16384b+              131072b+ 
 No. of Reads:                    0                     0                   112 
No. of Writes:                 4284                    16                     0 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             16     RELEASE
      0.00      28.67 us      13.00 us      41.00 us              3      ACCESS
      0.02      32.59 us      11.00 us     106.00 us            184        STAT
      0.03      37.77 us       6.00 us     103.00 us            192       FLUSH
      0.03      48.23 us      19.00 us     121.00 us            193      STATFS
      0.12     106.78 us      41.00 us     245.00 us            325      LOOKUP
      0.19     471.36 us      49.00 us   20113.00 us            112        READ
     14.49      54.34 us       9.00 us    1207.00 us          75290       FSTAT
     85.12     100.23 us      29.00 us    1739.00 us         239885       WRITE
 
    Duration: 96 seconds
   Data Read: 14680064 bytes
Data Written: 148440000 bytes
 
Brick: nas4-10g:/data15/gvol
----------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     1                     0 
No. of Writes:                  107                     0                 68152 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                 3884               8300782               8516312 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    2                232782                     7 
No. of Writes:              3461081              15911651              91527571 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                   60                   338                    74 
No. of Writes:             16154130              12732671               5311766 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:                33878                247197                 20326 
No. of Writes:             95504061              29337720               8187030 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:               177564                619264             247831629 
No. of Writes:              2473961               1419888               2575526 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   68 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           5376      FORGET
      0.00       0.00 us       0.00 us       0.00 us        1266475     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         109404  RELEASEDIR
      0.00      93.00 us      93.00 us      93.00 us              1      ACCESS
      0.20      34.01 us      13.00 us     102.00 us            322        STAT
      0.58      38.66 us      12.00 us     114.00 us            840       FLUSH
      1.59      43.99 us      20.00 us     151.00 us           2024      STATFS
     16.50     433.87 us      43.00 us   77186.00 us           2127      LOOKUP
     81.13    1187.53 us      53.00 us  355507.00 us           3822        READ
 
    Duration: 431368 seconds
   Data Read: 32534712233462 bytes
Data Written: 1913767903640 bytes
 
Interval 2 Stats:
   Block Size:             131072b+ 
 No. of Reads:                  631 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             14     RELEASE
      0.66      29.80 us      13.00 us      85.00 us            184        STAT
      0.82      40.67 us      13.00 us      73.00 us            168       FLUSH
      0.98      42.52 us      20.00 us     113.00 us            193      STATFS
      3.98     102.29 us      46.00 us     270.00 us            326      LOOKUP
     93.56    1241.13 us      60.00 us  156647.00 us            631        READ
 
    Duration: 96 seconds
   Data Read: 82706432 bytes
Data Written: 0 bytes
 
Brick: nas4-10g:/data14/gvol
----------------------------
Cumulative Stats:
   Block Size:                  1b+                   4b+                   8b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  150                 56945                  3144 
 
   Block Size:                 16b+                  32b+                  64b+ 
 No. of Reads:                    0                     5                     1 
No. of Writes:              6945350              13373779               6572652 
 
   Block Size:                128b+                 256b+                 512b+ 
 No. of Reads:                 2600                     7                    84 
No. of Writes:             15657875              96604095              27470300 
 
   Block Size:               1024b+                2048b+                4096b+ 
 No. of Reads:                  280                    93                 39962 
No. of Writes:             13158968              18008018             104566143 
 
   Block Size:               8192b+               16384b+               32768b+ 
 No. of Reads:               247581                 61525                424768 
No. of Writes:             32013956               8099478               2463916 
 
   Block Size:              65536b+              131072b+              262144b+ 
 No. of Reads:              1762271             275293932                     0 
No. of Writes:              2375462              14621188                   104 
 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           9289      FORGET
      0.00       0.00 us       0.00 us       0.00 us        1567656     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         137113  RELEASEDIR
      0.00      32.83 us      17.00 us      44.00 us              6      ACCESS
      0.23      36.76 us      14.00 us     118.00 us            322        STAT
      0.63      38.06 us      13.00 us      97.00 us            840       FLUSH
      1.88      46.82 us      19.00 us     179.00 us           2024      STATFS
     13.09     101.45 us      64.00 us     252.00 us           6510       WRITE
     15.69     372.21 us      34.00 us   69938.00 us           2126      LOOKUP
     68.47    1150.92 us      54.00 us  340203.00 us           3001        READ
 
    Duration: 516081 seconds
   Data Read: 36220486986088 bytes
Data Written: 3706236391485 bytes
 
Interval 2 Stats:
   Block Size:             131072b+ 
 No. of Reads:                  482 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             14     RELEASE
      0.01      41.00 us      38.00 us      44.00 us              3      ACCESS
      0.41      33.18 us      14.00 us     118.00 us            184        STAT
      0.44      38.89 us      14.00 us      90.00 us            168       FLUSH
      0.61      46.86 us      22.00 us     112.00 us            193      STATFS
      2.33     106.87 us      49.00 us     297.00 us            325      LOOKUP
     96.20    2972.94 us      62.00 us  340203.00 us            482        READ
 
    Duration: 96 seconds
   Data Read: 63176704 bytes
Data Written: 0 bytes
 
Brick: nas4-10g:/data16/gvol
----------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  257                    42                343818 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                  461                     0                    10 
No. of Writes:                39602              41604782              45688731 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                   24                 20447                   744 
No. of Writes:           1562272399            2661869208            1282744508 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                  831                  2591                  1943 
No. of Writes:            582413527             197225820             316982056 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              4723753               3196854                673805 
No. of Writes:            612708974             257698531              64463657 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:              1375642              46137369            1214376777 
No. of Writes:             13992804              14030587              31263025 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                  202 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us         104780      FORGET
      0.00       0.00 us       0.00 us       0.00 us       23962953     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         545444  RELEASEDIR
      0.09      40.62 us      13.00 us      87.00 us            131       FLUSH
      0.19      35.21 us      13.00 us     108.00 us            322        STAT
      1.50      43.27 us      21.00 us     126.00 us           2024      STATFS
      7.61     906.02 us      51.00 us   85240.00 us            491        READ
      9.41     258.78 us      43.00 us   28299.00 us           2126      LOOKUP
     81.20      85.58 us      33.00 us     207.00 us          55476       WRITE
 
    Duration: 2083859 seconds
   Data Read: 162319927356315 bytes
Data Written: 17503378837925 bytes
 
Interval 2 Stats:
   Block Size:             131072b+ 
 No. of Reads:                   37 
No. of Writes:                    0 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              2     RELEASE
      1.84      38.12 us      13.00 us      59.00 us             24       FLUSH
      7.84     105.49 us      75.00 us     137.00 us             37        READ
     11.19      30.28 us      13.00 us      78.00 us            184        STAT
     15.74      40.61 us      21.00 us     102.00 us            193      STATFS
     63.38      97.08 us      49.00 us     278.00 us            325      LOOKUP
 
    Duration: 96 seconds
   Data Read: 4849664 bytes
Data Written: 0 bytes
 
Brick: nas1-10g:/data3/gvol
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                12866                 24859                323807 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                  412                    11                    41 
No. of Writes:                10645              41105912              25134078 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                   48                 27542                   975 
No. of Writes:           1292875339             843578811             744024598 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                 1037                  2743                  2651 
No. of Writes:            514219067             157387945             221165334 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              7095518               1659221                682413 
No. of Writes:            377770795             149646392              26457687 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:               756642              21515090             893474661 
No. of Writes:              6883983               7293778              23116967 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   73 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          78022      FORGET
      0.00       0.00 us       0.00 us       0.00 us       21609000     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422486  RELEASEDIR
      0.05      36.47 us      14.00 us      96.00 us             17      ACCESS
      0.98      38.82 us      14.00 us     100.00 us            322        STAT
      7.31      46.00 us      20.00 us     174.00 us           2024      STATFS
     91.66     546.81 us      32.00 us   53829.00 us           2134      LOOKUP
 
    Duration: 1643022 seconds
   Data Read: 118613759650761 bytes
Data Written: 10929293256154 bytes
 
Interval 2 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.49      28.00 us      14.00 us      65.00 us              8      ACCESS
     13.26      32.66 us      14.00 us      60.00 us            184        STAT
     17.70      41.56 us      21.00 us     111.00 us            193      STATFS
     68.55      95.59 us      44.00 us     285.00 us            325      LOOKUP
 
    Duration: 95 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: nas1-10g:/data4/gvol
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                 1316                  1165                102009 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                   86                     0                     2 
No. of Writes:                 8839               9266893              15627653 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    4                  1896                   522 
No. of Writes:           1398213448             920823375             693751586 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                  135                   797                  1065 
No. of Writes:            250611823              77787294             131519640 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              3491363               1095468               1207276 
No. of Writes:            263505479              90307664              20141698 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:              1399405               8171475             306788386 
No. of Writes:              5206176               3607247              12105216 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                  196 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          51459      FORGET
      0.00       0.00 us       0.00 us       0.00 us       20885594     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422485  RELEASEDIR
      0.00      63.00 us      63.00 us      63.00 us              1        OPEN
      0.00      40.38 us       9.00 us      88.00 us              8      ACCESS
      0.02      51.71 us      16.00 us     108.00 us             34       FLUSH
      0.12   10179.00 us   10179.00 us   10179.00 us              1      CREATE
      0.17      43.90 us      14.00 us     116.00 us            322        STAT
      1.30      53.90 us      19.00 us     128.00 us           2024      STATFS
     13.36     522.98 us      39.00 us   89328.00 us           2136      LOOKUP
     85.02     108.00 us      31.00 us    1321.00 us          65823       WRITE
 
    Duration: 1643022 seconds
   Data Read: 40856739676704 bytes
Data Written: 6647121567894 bytes
 
Interval 2 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.42      32.00 us       9.00 us      62.00 us              6      ACCESS
     15.15      37.99 us      14.00 us      97.00 us            184        STAT
     18.03      43.12 us      22.00 us     105.00 us            193      STATFS
     66.41      94.30 us      42.00 us     437.00 us            325      LOOKUP
 
    Duration: 95 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: nas1-10g:/data2/gvol
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     3                     0 
No. of Writes:                  152                    86                228980 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                  305                     1                     0 
No. of Writes:                 8038              40193578              18301406 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    4                  8174                   755 
No. of Writes:           1179643287             773912641             723775462 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                  530                   825                  1563 
No. of Writes:            345033982              98562726             189979361 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              4004519               1662938                580793 
No. of Writes:            350504951             149569626              37434932 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:               968566              22065542             628847152 
No. of Writes:              7361059               5337716              29462826 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   95 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          61432      FORGET
      0.00       0.00 us       0.00 us       0.00 us       17730243     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422491  RELEASEDIR
      0.00      65.00 us      65.00 us      65.00 us              1        OPEN
      0.02      44.74 us      16.00 us      76.00 us             23       FLUSH
      0.03      43.65 us      11.00 us      91.00 us             52      ACCESS
      0.21      44.87 us      13.00 us     105.00 us            322        STAT
      0.39   27014.00 us   27014.00 us   27014.00 us              1      CREATE
      1.49      50.56 us      18.00 us     151.00 us           2024      STATFS
     17.10     548.98 us      40.00 us   62271.00 us           2136      LOOKUP
     80.76      94.12 us      32.00 us     225.00 us          58844       WRITE
 
    Duration: 1643022 seconds
   Data Read: 83957540265576 bytes
Data Written: 11073257699744 bytes
 
Interval 2 Stats:
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1     RELEASE
      0.65      44.82 us      16.00 us      66.00 us             11       FLUSH
      1.33      37.33 us      11.00 us      90.00 us             27      ACCESS
     10.24      42.33 us      13.00 us      98.00 us            184        STAT
     11.90      46.90 us      18.00 us     108.00 us            193      STATFS
     75.88     177.52 us      42.00 us   26117.00 us            325      LOOKUP
 
    Duration: 95 seconds
   Data Read: 0 bytes
Data Written: 0 bytes
 
Brick: nas1-10g:/data1/gvol
---------------------------
Cumulative Stats:
   Block Size:                  1b+                   2b+                   4b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                  151                    13                 81196 
 
   Block Size:                  8b+                  16b+                  32b+ 
 No. of Reads:                   14                     2                     0 
No. of Writes:                 1603               7608086               7955758 
 
   Block Size:                 64b+                 128b+                 256b+ 
 No. of Reads:                    3                  4314                   557 
No. of Writes:            738224969             430264864             261225993 
 
   Block Size:                512b+                1024b+                2048b+ 
 No. of Reads:                   65                    75                   186 
No. of Writes:             98925790              37066531              45695936 
 
   Block Size:               4096b+                8192b+               16384b+ 
 No. of Reads:              4902061               1406500                639922 
No. of Writes:            143325821              43522270               8861235 
 
   Block Size:              32768b+               65536b+              131072b+ 
 No. of Reads:               778372               6662218             311945542 
No. of Writes:              2914316               1746266               6558560 
 
   Block Size:             262144b+ 
 No. of Reads:                    0 
No. of Writes:                   76 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us          41056      FORGET
      0.00       0.00 us       0.00 us       0.00 us        4546126     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         422484  RELEASEDIR
      0.00      69.00 us      69.00 us      69.00 us              1        OPEN
      0.01      43.44 us      36.00 us      67.00 us             16      ACCESS
      0.02      38.74 us      12.00 us     101.00 us             23       FLUSH
      0.29      51.97 us      15.00 us     161.00 us            322        STAT
      0.56   32456.00 us   32456.00 us   32456.00 us              1      CREATE
      2.24      63.82 us      19.00 us     140.00 us           2024      STATFS
     18.89     509.61 us      42.00 us  102367.00 us           2139      LOOKUP
     77.99     105.72 us      32.00 us    1151.00 us          42561       WRITE
 
    Duration: 1643022 seconds
   Data Read: 41404034760227 bytes
Data Written: 3321036917134 bytes
 
Interval 2 Stats:
   Block Size:               4096b+                8192b+ 
 No. of Reads:                    0                     0 
No. of Writes:                    8                     3 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1     RELEASE
      0.49      39.43 us      36.00 us      42.00 us              7      ACCESS
      0.51      26.09 us      12.00 us     101.00 us             11       FLUSH
      1.54      79.36 us      38.00 us     163.00 us             11       WRITE
     15.33      47.07 us      15.00 us     111.00 us            184        STAT
     18.71      54.77 us      21.00 us     140.00 us            193      STATFS
     63.43     110.27 us      42.00 us     270.00 us            325      LOOKUP
 
    Duration: 95 seconds
   Data Read: 0 bytes
Data Written: 114408 bytes
 


More information about the Gluster-users mailing list