[Gluster-users] Slow read performance
Thomas Wakefield
twake at cola.iges.org
Mon Mar 11 16:02:24 UTC 2013
Is there a way to make a ramdisk support extended attributes?
These are my current sysctl settings (and I have tried many different options):
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.ipv4.tcp_syncookies = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 68719476736
kernel.shmall = 4294967296
kernel.panic = 5
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 87380 67108864
net.ipv4.tcp_wmem = 4096 65536 67108864
net.core.netdev_max_backlog = 250000
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_mtu_probing = 1
Here is the output from a dd write and dd read.
[root at cpu_crew1 ~]# dd if=/dev/zero of=/shared/working/benchmark/test.cpucrew1 bs=512k count=10000 ; dd if=/shared/working/benchmark/test.cpucrew1 of=/dev/null bs=512k
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB) copied, 7.21958 seconds, 726 MB/s
10000+0 records in
10000+0 records out
5242880000 bytes (5.2 GB) copied, 86.4165 seconds, 60.7 MB/s
[root at disk2 ~]# gluster volume profile shared info
Brick: fs-disk2:/storage/disk2g
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 2
No. of Writes: 20 35 6331
Block Size: 8b+ 16b+ 32b+
No. of Reads: 12 1 6
No. of Writes: 31681 1224295 161
Block Size: 64b+ 128b+ 256b+
No. of Reads: 34 35 314
No. of Writes: 163 234 539
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 165 616 1286
No. of Writes: 904 2081 30248
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 66206 1228556 92209
No. of Writes: 6973257 42743609 5711989
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 65742 75482 29711040
No. of Writes: 3521861 1512528 13722755
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 45836 FORGET
0.00 0.00 us 0.00 us 0.00 us 239256 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.42 281.00 us 281.00 us 281.00 us 1 STAT
2.64 145.75 us 30.00 us 572.00 us 12 OPENDIR
15.16 772.92 us 277.00 us 1186.00 us 13 STATFS
81.78 2356.26 us 64.00 us 39949.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3915245128069 bytes
Data Written: 2653770269732 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 2
No. of Writes: 20 35 6331
Block Size: 8b+ 16b+ 32b+
No. of Reads: 12 1 6
No. of Writes: 31681 1224295 161
Block Size: 64b+ 128b+ 256b+
No. of Reads: 34 35 314
No. of Writes: 163 234 539
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 165 616 1286
No. of Writes: 904 2081 30248
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 66206 1228556 92209
No. of Writes: 6973257 42743609 5711989
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 65742 75482 29711040
No. of Writes: 3521861 1512528 13722755
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 45836 FORGET
0.00 0.00 us 0.00 us 0.00 us 239256 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.42 281.00 us 281.00 us 281.00 us 1 STAT
2.64 145.75 us 30.00 us 572.00 us 12 OPENDIR
15.16 772.92 us 277.00 us 1186.00 us 13 STATFS
81.78 2356.26 us 64.00 us 39949.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3915245128069 bytes
Data Written: 2653770269732 bytes
Brick: fs-disk2:/storage/disk2h
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 10 32 2246
Block Size: 8b+ 16b+ 32b+
No. of Reads: 2 1 7
No. of Writes: 14532 1324118 138
Block Size: 64b+ 128b+ 256b+
No. of Reads: 27 30 320
No. of Writes: 139 238 512
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 285 1353 1582
No. of Writes: 744 3796 40897
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 36625 665635 56092
No. of Writes: 8987796 50778993 6891891
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 46596 62818 30538792
No. of Writes: 3564231 1690585 15524821
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 44650 FORGET
0.00 0.00 us 0.00 us 0.00 us 224762 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.07 70.00 us 70.00 us 70.00 us 1 STAT
1.08 93.08 us 38.00 us 194.00 us 12 OPENDIR
8.59 680.85 us 277.00 us 1245.00 us 13 STATFS
90.26 4042.70 us 64.00 us 90228.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4016932217957 bytes
Data Written: 3028611159470 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 10 32 2246
Block Size: 8b+ 16b+ 32b+
No. of Reads: 2 1 7
No. of Writes: 14532 1324118 138
Block Size: 64b+ 128b+ 256b+
No. of Reads: 27 30 320
No. of Writes: 139 238 512
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 285 1353 1582
No. of Writes: 744 3796 40897
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 36625 665635 56092
No. of Writes: 8987796 50778993 6891891
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 46596 62818 30538792
No. of Writes: 3564231 1690585 15524821
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 44650 FORGET
0.00 0.00 us 0.00 us 0.00 us 224762 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.07 70.00 us 70.00 us 70.00 us 1 STAT
1.08 93.08 us 38.00 us 194.00 us 12 OPENDIR
8.59 680.85 us 277.00 us 1245.00 us 13 STATFS
90.26 4042.70 us 64.00 us 90228.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4016932217957 bytes
Data Written: 3028611159470 bytes
Brick: fs-disk2:/storage/disk2q
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 8
No. of Writes: 17 17 66
Block Size: 16b+ 32b+ 64b+
No. of Reads: 6 8 25
No. of Writes: 1374671 186 194
Block Size: 128b+ 256b+ 512b+
No. of Reads: 72 442 116
No. of Writes: 2462 600 930
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 575 1042 65613
No. of Writes: 6338 9129 10297439
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 1231874 91724 62974
No. of Writes: 45897801 7596377 3383890
Block Size: 65536b+ 131072b+
No. of Reads: 68721 31115020
No. of Writes: 1628168 14997024
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 32635 FORGET
0.00 0.00 us 0.00 us 0.00 us 220295 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.00 19.00 us 19.00 us 19.00 us 1 FLUSH
0.00 25.00 us 25.00 us 25.00 us 1 STAT
0.00 31.00 us 31.00 us 31.00 us 1 FSTAT
0.01 83.17 us 30.00 us 193.00 us 12 OPENDIR
0.07 708.46 us 311.00 us 1598.00 us 13 STATFS
0.59 3486.70 us 44.00 us 77044.00 us 23 LOOKUP
99.33 2393.23 us 16.00 us 111841.00 us 5604 READ
Duration: 1986036 seconds
Data Read: 4098599948827 bytes
Data Written: 2934296167395 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 8
No. of Writes: 17 17 66
Block Size: 16b+ 32b+ 64b+
No. of Reads: 6 8 25
No. of Writes: 1374671 186 194
Block Size: 128b+ 256b+ 512b+
No. of Reads: 72 442 116
No. of Writes: 2462 600 930
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 575 1042 65613
No. of Writes: 6338 9129 10297439
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 1231874 91724 62974
No. of Writes: 45897801 7596377 3383890
Block Size: 65536b+ 131072b+
No. of Reads: 68721 31115020
No. of Writes: 1628168 14997024
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 32635 FORGET
0.00 0.00 us 0.00 us 0.00 us 220295 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.00 19.00 us 19.00 us 19.00 us 1 FLUSH
0.00 25.00 us 25.00 us 25.00 us 1 STAT
0.00 31.00 us 31.00 us 31.00 us 1 FSTAT
0.01 83.17 us 30.00 us 193.00 us 12 OPENDIR
0.07 708.46 us 311.00 us 1598.00 us 13 STATFS
0.59 3486.70 us 44.00 us 77044.00 us 23 LOOKUP
99.33 2393.23 us 16.00 us 111841.00 us 5604 READ
Duration: 1986036 seconds
Data Read: 4098599948827 bytes
Data Written: 2934296167395 bytes
Brick: fs-disk2:/storage/disk2v
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 4
No. of Writes: 76 82 2263
Block Size: 8b+ 16b+ 32b+
No. of Reads: 8 7 21
No. of Writes: 776 1276660 472
Block Size: 64b+ 128b+ 256b+
No. of Reads: 33 50 278
No. of Writes: 412 1268 793
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 109 309 804
No. of Writes: 1128 3946 9187
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 31227 526511 49389
No. of Writes: 8508834 40471184 7221907
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 39915 52502 31497648
No. of Writes: 3628832 2605591 16727412
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 69158 FORGET
0.00 0.00 us 0.00 us 0.00 us 758156 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335452 RELEASEDIR
0.07 84.00 us 84.00 us 84.00 us 1 STAT
0.68 64.83 us 30.00 us 192.00 us 12 OPENDIR
9.64 850.00 us 356.00 us 1794.00 us 13 STATFS
89.61 4465.26 us 58.00 us 100071.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4140191390006 bytes
Data Written: 3197280002013 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 4
No. of Writes: 76 82 2263
Block Size: 8b+ 16b+ 32b+
No. of Reads: 8 7 21
No. of Writes: 776 1276660 472
Block Size: 64b+ 128b+ 256b+
No. of Reads: 33 50 278
No. of Writes: 412 1268 793
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 109 309 804
No. of Writes: 1128 3946 9187
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 31227 526511 49389
No. of Writes: 8508834 40471184 7221907
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 39915 52502 31497648
No. of Writes: 3628832 2605591 16727412
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 69158 FORGET
0.00 0.00 us 0.00 us 0.00 us 758156 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335452 RELEASEDIR
0.07 84.00 us 84.00 us 84.00 us 1 STAT
0.68 64.83 us 30.00 us 192.00 us 12 OPENDIR
9.64 850.00 us 356.00 us 1794.00 us 13 STATFS
89.61 4465.26 us 58.00 us 100071.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4140191390006 bytes
Data Written: 3197280002013 bytes
Brick: fs-disk2:/storage/disk2t
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 6
No. of Writes: 14 1689 54
Block Size: 16b+ 32b+ 64b+
No. of Reads: 2 15 33
No. of Writes: 647495 101 155
Block Size: 128b+ 256b+ 512b+
No. of Reads: 33 327 139
No. of Writes: 1531 668 1939
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 742 1025 45656
No. of Writes: 1334 9233 11018359
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 827151 63253 47699
No. of Writes: 43612138 7285204 3582392
Block Size: 65536b+ 131072b+
No. of Reads: 55855 30729346
No. of Writes: 1106904 14289709
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 48562 FORGET
0.00 0.00 us 0.00 us 0.00 us 296118 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335467 RELEASEDIR
0.18 90.00 us 90.00 us 90.00 us 1 STAT
1.68 70.58 us 33.00 us 120.00 us 12 OPENDIR
18.04 699.46 us 356.00 us 1689.00 us 13 STATFS
80.10 1755.22 us 48.00 us 30037.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4042566814850 bytes
Data Written: 2740415347513 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 6
No. of Writes: 14 1689 54
Block Size: 16b+ 32b+ 64b+
No. of Reads: 2 15 33
No. of Writes: 647495 101 155
Block Size: 128b+ 256b+ 512b+
No. of Reads: 33 327 139
No. of Writes: 1531 668 1939
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 742 1025 45656
No. of Writes: 1334 9233 11018359
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 827151 63253 47699
No. of Writes: 43612138 7285204 3582392
Block Size: 65536b+ 131072b+
No. of Reads: 55855 30729346
No. of Writes: 1106904 14289709
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 48562 FORGET
0.00 0.00 us 0.00 us 0.00 us 296118 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335467 RELEASEDIR
0.18 90.00 us 90.00 us 90.00 us 1 STAT
1.68 70.58 us 33.00 us 120.00 us 12 OPENDIR
18.04 699.46 us 356.00 us 1689.00 us 13 STATFS
80.10 1755.22 us 48.00 us 30037.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4042566814850 bytes
Data Written: 2740415347513 bytes
Brick: fs-disk2:/storage/disk2w
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 1 0 0
No. of Writes: 1 25 4611
Block Size: 8b+ 16b+ 32b+
No. of Reads: 8 0 5
No. of Writes: 14778 1007095 214
Block Size: 64b+ 128b+ 256b+
No. of Reads: 17 25 215
No. of Writes: 1579 902 590
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 88 268 839
No. of Writes: 627 3397 30790
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 29913 497020 44552
No. of Writes: 7681162 40100903 5305584
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 38631 46410 29855714
No. of Writes: 3106123 1432531 23341972
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 59919 FORGET
0.00 0.00 us 0.00 us 0.00 us 289226 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335467 RELEASEDIR
0.12 112.00 us 112.00 us 112.00 us 1 STAT
0.68 53.58 us 29.00 us 109.00 us 12 OPENDIR
10.83 790.92 us 304.00 us 2093.00 us 13 STATFS
88.38 3648.91 us 49.00 us 81427.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3924075139505 bytes
Data Written: 3854579862798 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 1 0 0
No. of Writes: 1 25 4611
Block Size: 8b+ 16b+ 32b+
No. of Reads: 8 0 5
No. of Writes: 14778 1007095 214
Block Size: 64b+ 128b+ 256b+
No. of Reads: 17 25 215
No. of Writes: 1579 902 590
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 88 268 839
No. of Writes: 627 3397 30790
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 29913 497020 44552
No. of Writes: 7681162 40100903 5305584
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 38631 46410 29855714
No. of Writes: 3106123 1432531 23341972
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 59919 FORGET
0.00 0.00 us 0.00 us 0.00 us 289225 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335467 RELEASEDIR
0.12 112.00 us 112.00 us 112.00 us 1 STAT
0.68 53.58 us 29.00 us 109.00 us 12 OPENDIR
10.83 790.92 us 304.00 us 2093.00 us 13 STATFS
88.38 3648.91 us 49.00 us 81427.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3924075139505 bytes
Data Written: 3854579862798 bytes
Brick: fs-disk2:/storage/disk2o
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 1
No. of Writes: 32 3309 21858
Block Size: 16b+ 32b+ 64b+
No. of Reads: 4 1 17
No. of Writes: 1158026 239 203
Block Size: 128b+ 256b+ 512b+
No. of Reads: 26 353 57
No. of Writes: 1637 492 601
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 263 806 51857
No. of Writes: 3491 13944 6771528
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 966761 74162 56167
No. of Writes: 46972247 5711679 3569591
Block Size: 65536b+ 131072b+
No. of Reads: 63886 28210167
No. of Writes: 1507101 11692443
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 63359 FORGET
0.00 0.00 us 0.00 us 0.00 us 286551 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335473 RELEASEDIR
0.15 145.00 us 145.00 us 145.00 us 1 STAT
0.78 64.25 us 26.00 us 120.00 us 12 OPENDIR
8.99 684.54 us 275.00 us 1563.00 us 13 STATFS
90.08 3876.26 us 67.00 us 70115.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3714770261739 bytes
Data Written: 2433168135741 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 1
No. of Writes: 32 3309 21858
Block Size: 16b+ 32b+ 64b+
No. of Reads: 4 1 17
No. of Writes: 1158026 239 203
Block Size: 128b+ 256b+ 512b+
No. of Reads: 26 353 57
No. of Writes: 1637 492 601
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 263 806 51857
No. of Writes: 3491 13944 6771528
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 966761 74162 56167
No. of Writes: 46972247 5711679 3569591
Block Size: 65536b+ 131072b+
No. of Reads: 63886 28210167
No. of Writes: 1507101 11692443
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 63359 FORGET
0.00 0.00 us 0.00 us 0.00 us 286551 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335473 RELEASEDIR
0.15 145.00 us 145.00 us 145.00 us 1 STAT
0.78 64.25 us 26.00 us 120.00 us 12 OPENDIR
8.99 684.54 us 275.00 us 1563.00 us 13 STATFS
90.08 3876.26 us 67.00 us 70115.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3714770261739 bytes
Data Written: 2433168135741 bytes
Brick: fs-disk2:/storage/disk2u
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 7
No. of Writes: 25 24297 61424
Block Size: 16b+ 32b+ 64b+
No. of Reads: 2 6 14
No. of Writes: 980690 242 271
Block Size: 128b+ 256b+ 512b+
No. of Reads: 22 470 122
No. of Writes: 918 498 702
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 692 1233 20426
No. of Writes: 4092 17595 8297742
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 306285 33094 36653
No. of Writes: 40734552 5320245 3384446
Block Size: 65536b+ 131072b+
No. of Reads: 47884 29929679
No. of Writes: 1261085 23614808
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 36356 FORGET
0.00 0.00 us 0.00 us 0.00 us 235964 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335452 RELEASEDIR
0.00 35.00 us 35.00 us 35.00 us 1 STAT
0.00 19.00 us 15.00 us 23.00 us 2 FLUSH
0.00 42.00 us 42.00 us 42.00 us 1 SETATTR
0.00 83.50 us 44.00 us 123.00 us 2 OPEN
0.00 70.92 us 32.00 us 116.00 us 12 OPENDIR
0.05 701.00 us 296.00 us 1545.00 us 13 STATFS
0.96 7538.16 us 52.00 us 128086.00 us 25 LOOKUP
22.21 4371915.00 us 4371915.00 us 4371915.00 us 1 TRUNCATE
23.06 113.42 us 15.00 us 3034.00 us 40030 READ
53.73 264.45 us 97.00 us 105894.00 us 40000 WRITE
Duration: 1986036 seconds
Data Read: 3932182878499 bytes
Data Written: 3897490217239 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 7
No. of Writes: 25 24297 61424
Block Size: 16b+ 32b+ 64b+
No. of Reads: 2 6 14
No. of Writes: 980690 242 271
Block Size: 128b+ 256b+ 512b+
No. of Reads: 22 470 122
No. of Writes: 918 498 702
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 692 1233 20426
No. of Writes: 4092 17595 8297742
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 306285 33094 36653
No. of Writes: 40734552 5320245 3384446
Block Size: 65536b+ 131072b+
No. of Reads: 47884 29929679
No. of Writes: 1261085 23614808
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 36356 FORGET
0.00 0.00 us 0.00 us 0.00 us 235964 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335452 RELEASEDIR
0.00 35.00 us 35.00 us 35.00 us 1 STAT
0.00 19.00 us 15.00 us 23.00 us 2 FLUSH
0.00 42.00 us 42.00 us 42.00 us 1 SETATTR
0.00 83.50 us 44.00 us 123.00 us 2 OPEN
0.00 70.92 us 32.00 us 116.00 us 12 OPENDIR
0.05 701.00 us 296.00 us 1545.00 us 13 STATFS
0.96 7538.16 us 52.00 us 128086.00 us 25 LOOKUP
22.21 4371915.00 us 4371915.00 us 4371915.00 us 1 TRUNCATE
23.06 113.42 us 15.00 us 3034.00 us 40030 READ
53.73 264.45 us 97.00 us 105894.00 us 40000 WRITE
Duration: 1986036 seconds
Data Read: 3932182878499 bytes
Data Written: 3897490217239 bytes
Brick: fs-disk2:/storage/disk2p
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 5
No. of Writes: 2 31 6494
Block Size: 8b+ 16b+ 32b+
No. of Reads: 5 1 7
No. of Writes: 31270 1410945 215
Block Size: 64b+ 128b+ 256b+
No. of Reads: 18 82 305
No. of Writes: 275 894 598
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 93 342 1011
No. of Writes: 642 6170 45584
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 38805 689118 55529
No. of Writes: 6451465 43176016 8441769
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 46299 53069 31432059
No. of Writes: 3454401 1692618 17475572
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 60570 FORGET
0.00 0.00 us 0.00 us 0.00 us 241838 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.04 56.00 us 56.00 us 56.00 us 1 STAT
1.06 133.83 us 24.00 us 661.00 us 12 OPENDIR
6.68 781.31 us 299.00 us 2241.00 us 13 STATFS
92.22 6094.22 us 54.00 us 123993.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4133248604192 bytes
Data Written: 3235536161348 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 5
No. of Writes: 2 31 6494
Block Size: 8b+ 16b+ 32b+
No. of Reads: 5 1 7
No. of Writes: 31270 1410945 215
Block Size: 64b+ 128b+ 256b+
No. of Reads: 18 82 305
No. of Writes: 275 894 598
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 93 342 1011
No. of Writes: 642 6170 45584
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 38805 689118 55529
No. of Writes: 6451465 43176016 8441769
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 46299 53069 31432059
No. of Writes: 3454401 1692618 17475572
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 60570 FORGET
0.00 0.00 us 0.00 us 0.00 us 241838 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.04 56.00 us 56.00 us 56.00 us 1 STAT
1.06 133.83 us 24.00 us 661.00 us 12 OPENDIR
6.68 781.31 us 299.00 us 2241.00 us 13 STATFS
92.22 6094.22 us 54.00 us 123993.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4133248604192 bytes
Data Written: 3235536161348 bytes
Brick: fs-disk2:/storage/disk2l
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 1 8 9009
Block Size: 8b+ 16b+ 32b+
No. of Reads: 2 1 4
No. of Writes: 44135 1490959 37
Block Size: 64b+ 128b+ 256b+
No. of Reads: 22 31 544
No. of Writes: 82 942 424
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 174 579 896
No. of Writes: 449 878 18783
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 36095 667458 60986
No. of Writes: 7779558 48360336 6948692
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 55504 74463 31421258
No. of Writes: 3488854 1733753 12896511
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 52642 FORGET
0.00 0.00 us 0.00 us 0.00 us 233089 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.04 33.00 us 33.00 us 33.00 us 1 STAT
0.09 83.00 us 83.00 us 83.00 us 1 FLUSH
0.14 123.00 us 123.00 us 123.00 us 1 UNLINK
1.27 95.50 us 32.00 us 220.00 us 12 OPENDIR
12.48 864.92 us 319.00 us 1918.00 us 13 STATFS
85.98 3227.00 us 60.00 us 73688.00 us 24 LOOKUP
Duration: 1986036 seconds
Data Read: 4134086460445 bytes
Data Written: 2658148435749 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 1 8 9009
Block Size: 8b+ 16b+ 32b+
No. of Reads: 2 1 4
No. of Writes: 44135 1490959 37
Block Size: 64b+ 128b+ 256b+
No. of Reads: 22 31 544
No. of Writes: 82 942 424
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 174 579 896
No. of Writes: 449 878 18783
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 36095 667458 60986
No. of Writes: 7779558 48360336 6948692
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 55504 74463 31421258
No. of Writes: 3488854 1733753 12896511
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 52642 FORGET
0.00 0.00 us 0.00 us 0.00 us 233089 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.04 33.00 us 33.00 us 33.00 us 1 STAT
0.09 83.00 us 83.00 us 83.00 us 1 FLUSH
0.14 123.00 us 123.00 us 123.00 us 1 UNLINK
1.27 95.50 us 32.00 us 220.00 us 12 OPENDIR
12.48 864.92 us 319.00 us 1918.00 us 13 STATFS
85.98 3227.00 us 60.00 us 73688.00 us 24 LOOKUP
Duration: 1986036 seconds
Data Read: 4134086460445 bytes
Data Written: 2658148435749 bytes
Brick: fs-disk2:/storage/disk2k
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 2
No. of Writes: 2 32 15
Block Size: 8b+ 16b+ 32b+
No. of Reads: 8 0 20
No. of Writes: 126 1354976 230
Block Size: 64b+ 128b+ 256b+
No. of Reads: 23 43 436
No. of Writes: 181 162 454
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 285 1141 1380
No. of Writes: 630 1389 40332
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 19746 310845 43561
No. of Writes: 9013735 43371892 7025089
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 53805 84882 37057928
No. of Writes: 3406899 1794373 18383965
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 60415 FORGET
0.00 0.00 us 0.00 us 0.00 us 278544 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.13 97.00 us 97.00 us 97.00 us 1 STAT
1.18 75.33 us 36.00 us 170.00 us 12 OPENDIR
13.28 780.00 us 288.00 us 1955.00 us 13 STATFS
85.41 2836.39 us 65.00 us 49275.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4870792010134 bytes
Data Written: 3334665501795 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 2
No. of Writes: 2 32 15
Block Size: 8b+ 16b+ 32b+
No. of Reads: 8 0 20
No. of Writes: 126 1354976 230
Block Size: 64b+ 128b+ 256b+
No. of Reads: 23 43 436
No. of Writes: 181 162 454
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 285 1141 1380
No. of Writes: 630 1389 40332
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 19746 310845 43561
No. of Writes: 9013735 43371892 7025089
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 53805 84882 37057928
No. of Writes: 3406899 1794373 18383965
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 60415 FORGET
0.00 0.00 us 0.00 us 0.00 us 278544 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.13 97.00 us 97.00 us 97.00 us 1 STAT
1.18 75.33 us 36.00 us 170.00 us 12 OPENDIR
13.28 780.00 us 288.00 us 1955.00 us 13 STATFS
85.41 2836.39 us 65.00 us 49275.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4870792010134 bytes
Data Written: 3334665501795 bytes
Brick: fs-disk2:/storage/disk2r
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 9
No. of Writes: 43 3068 26544
Block Size: 16b+ 32b+ 64b+
No. of Reads: 3 12 25
No. of Writes: 1161244 278 444
Block Size: 128b+ 256b+ 512b+
No. of Reads: 30 334 174
No. of Writes: 1288 590 664
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 712 1333 43449
No. of Writes: 3655 13879 5701186
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 748206 66005 55221
No. of Writes: 47541074 9200916 3530676
Block Size: 65536b+ 131072b+
No. of Reads: 70004 32903535
No. of Writes: 1478448 14907554
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 43493 FORGET
0.00 0.00 us 0.00 us 0.00 us 278416 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335473 RELEASEDIR
0.04 61.00 us 61.00 us 61.00 us 1 STAT
1.07 123.08 us 35.00 us 666.00 us 12 OPENDIR
7.04 744.23 us 277.00 us 1366.00 us 13 STATFS
91.84 5490.39 us 53.00 us 122589.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4328727368494 bytes
Data Written: 2928729198120 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 9
No. of Writes: 43 3068 26544
Block Size: 16b+ 32b+ 64b+
No. of Reads: 3 12 25
No. of Writes: 1161244 278 444
Block Size: 128b+ 256b+ 512b+
No. of Reads: 30 334 174
No. of Writes: 1288 590 664
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 712 1333 43449
No. of Writes: 3655 13879 5701186
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 748206 66005 55221
No. of Writes: 47541074 9200916 3530676
Block Size: 65536b+ 131072b+
No. of Reads: 70004 32903535
No. of Writes: 1478448 14907554
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 43493 FORGET
0.00 0.00 us 0.00 us 0.00 us 278416 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335473 RELEASEDIR
0.04 61.00 us 61.00 us 61.00 us 1 STAT
1.07 123.08 us 35.00 us 666.00 us 12 OPENDIR
7.04 744.23 us 277.00 us 1366.00 us 13 STATFS
91.84 5490.39 us 53.00 us 122589.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4328727368494 bytes
Data Written: 2928729198120 bytes
Brick: fs-disk2:/storage/disk2f
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 126 195 315
Block Size: 8b+ 16b+ 32b+
No. of Reads: 4 0 2
No. of Writes: 613 1113470 807
Block Size: 64b+ 128b+ 256b+
No. of Reads: 25 54 126
No. of Writes: 871 1288 1754
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 105 181 1083
No. of Writes: 2521 4672 32257
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 64111 1188567 91102
No. of Writes: 5779122 49772428 8907380
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 67199 79279 29491849
No. of Writes: 3545695 1334590 13562444
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 55620 FORGET
0.00 0.00 us 0.00 us 0.00 us 231692 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.04 69.00 us 69.00 us 69.00 us 1 STAT
0.72 100.75 us 31.00 us 505.00 us 12 OPENDIR
5.65 731.54 us 296.00 us 2123.00 us 13 STATFS
93.59 6848.30 us 54.00 us 152037.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3886563048636 bytes
Data Written: 2744594192453 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 126 195 315
Block Size: 8b+ 16b+ 32b+
No. of Reads: 4 0 2
No. of Writes: 613 1113470 807
Block Size: 64b+ 128b+ 256b+
No. of Reads: 25 54 126
No. of Writes: 871 1288 1754
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 105 181 1083
No. of Writes: 2521 4672 32257
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 64111 1188567 91102
No. of Writes: 5779122 49772428 8907380
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 67199 79279 29491849
No. of Writes: 3545695 1334590 13562444
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 55620 FORGET
0.00 0.00 us 0.00 us 0.00 us 231688 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.04 69.00 us 69.00 us 69.00 us 1 STAT
0.72 100.75 us 31.00 us 505.00 us 12 OPENDIR
5.65 731.54 us 296.00 us 2123.00 us 13 STATFS
93.59 6848.30 us 54.00 us 152037.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3886563048636 bytes
Data Written: 2744594192453 bytes
Brick: fs-disk2:/storage/disk2m
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 62 81 3778
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 2 10
No. of Writes: 28538 1010483 478
Block Size: 64b+ 128b+ 256b+
No. of Reads: 23 37 386
No. of Writes: 356 584 826
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 156 590 978
No. of Writes: 1261 1992 19168
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 22980 387448 36474
No. of Writes: 7417500 47836776 8493369
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 35463 47791 29619466
No. of Writes: 3750483 1492030 13143833
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 56305 FORGET
0.00 0.00 us 0.00 us 0.00 us 330243 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.03 38.00 us 38.00 us 38.00 us 1 STAT
0.66 82.58 us 32.00 us 177.00 us 12 OPENDIR
5.67 659.69 us 306.00 us 1660.00 us 13 STATFS
93.64 6153.57 us 51.00 us 126411.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3892131654878 bytes
Data Written: 2689492141208 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 62 81 3778
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 2 10
No. of Writes: 28538 1010483 478
Block Size: 64b+ 128b+ 256b+
No. of Reads: 23 37 386
No. of Writes: 356 584 826
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 156 590 978
No. of Writes: 1261 1992 19168
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 22980 387448 36474
No. of Writes: 7417500 47836776 8493369
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 35463 47791 29619466
No. of Writes: 3750483 1492030 13143833
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 56305 FORGET
0.00 0.00 us 0.00 us 0.00 us 330243 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.03 38.00 us 38.00 us 38.00 us 1 STAT
0.66 82.58 us 32.00 us 177.00 us 12 OPENDIR
5.67 659.69 us 306.00 us 1660.00 us 13 STATFS
93.64 6153.57 us 51.00 us 126411.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3892131654878 bytes
Data Written: 2689492141208 bytes
Brick: fs-disk2:/storage/disk2a
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 1 19 6028
Block Size: 8b+ 16b+ 32b+
No. of Reads: 9 2 5
No. of Writes: 30940 1237328 244
Block Size: 64b+ 128b+ 256b+
No. of Reads: 21 26 289
No. of Writes: 106 174 540
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 124 545 1075
No. of Writes: 837 3921 26110
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 49323 908252 66678
No. of Writes: 6410563 45570376 6177128
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 47174 52841 26726899
No. of Writes: 3820825 1574618 14071645
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 40611 FORGET
0.00 0.00 us 0.00 us 0.00 us 239813 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.00 25.00 us 25.00 us 25.00 us 1 STAT
0.00 60.00 us 60.00 us 60.00 us 1 FLUSH
0.01 104.67 us 37.00 us 284.00 us 12 OPENDIR
0.13 1754.00 us 338.00 us 15066.00 us 13 STATFS
0.53 3808.24 us 54.00 us 92532.00 us 25 LOOKUP
0.61 13582.12 us 76.00 us 100164.00 us 8 READDIRP
17.75 57626.15 us 1811.00 us 507843.00 us 55 FSYNC
29.76 115.64 us 19.00 us 33986.00 us 45960 WRITE
51.22 90.07 us 19.00 us 7422.00 us 101563 READ
Duration: 1986036 seconds
Data Read: 3518405301400 bytes
Data Written: 2768283947733 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 1 19 6028
Block Size: 8b+ 16b+ 32b+
No. of Reads: 9 2 5
No. of Writes: 30940 1237328 244
Block Size: 64b+ 128b+ 256b+
No. of Reads: 21 26 289
No. of Writes: 106 174 540
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 124 545 1075
No. of Writes: 837 3921 26110
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 49323 908252 66678
No. of Writes: 6410563 45570376 6177128
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 47174 52841 26726899
No. of Writes: 3820825 1574618 14071645
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 40611 FORGET
0.00 0.00 us 0.00 us 0.00 us 239812 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.00 25.00 us 25.00 us 25.00 us 1 STAT
0.00 60.00 us 60.00 us 60.00 us 1 FLUSH
0.01 104.67 us 37.00 us 284.00 us 12 OPENDIR
0.13 1754.00 us 338.00 us 15066.00 us 13 STATFS
0.53 3808.24 us 54.00 us 92532.00 us 25 LOOKUP
0.61 13582.12 us 76.00 us 100164.00 us 8 READDIRP
17.75 57626.15 us 1811.00 us 507843.00 us 55 FSYNC
29.76 115.64 us 19.00 us 33986.00 us 45960 WRITE
51.22 90.07 us 19.00 us 7422.00 us 101563 READ
Duration: 1986036 seconds
Data Read: 3518405301400 bytes
Data Written: 2768283947733 bytes
Brick: fs-disk2:/storage/disk2j
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 2 5
No. of Writes: 24 837 13216
Block Size: 16b+ 32b+ 64b+
No. of Reads: 1 5 18
No. of Writes: 1020349 165 77
Block Size: 128b+ 256b+ 512b+
No. of Reads: 35 259 143
No. of Writes: 123 464 677
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 568 1235 38393
No. of Writes: 1685 28804 6671957
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 672430 58238 51661
No. of Writes: 40563190 7676616 3592549
Block Size: 65536b+ 131072b+
No. of Reads: 65729 29401751
No. of Writes: 1416702 12381773
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 47936 FORGET
0.00 0.00 us 0.00 us 0.00 us 306250 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.09 156.00 us 156.00 us 156.00 us 1 STAT
0.81 119.67 us 34.00 us 364.00 us 12 OPENDIR
6.55 892.77 us 280.00 us 1649.00 us 13 STATFS
33.85 112.50 us 53.00 us 9059.00 us 533 WRITE
58.70 4521.65 us 53.00 us 100893.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3868440883905 bytes
Data Written: 2492643392426 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 2 5
No. of Writes: 24 837 13216
Block Size: 16b+ 32b+ 64b+
No. of Reads: 1 5 18
No. of Writes: 1020349 165 77
Block Size: 128b+ 256b+ 512b+
No. of Reads: 35 259 143
No. of Writes: 123 464 677
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 568 1235 38393
No. of Writes: 1685 28804 6671957
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 672430 58238 51661
No. of Writes: 40563190 7676616 3592549
Block Size: 65536b+ 131072b+
No. of Reads: 65729 29401751
No. of Writes: 1416702 12381773
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 47936 FORGET
0.00 0.00 us 0.00 us 0.00 us 306249 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.09 156.00 us 156.00 us 156.00 us 1 STAT
0.81 119.67 us 34.00 us 364.00 us 12 OPENDIR
6.55 892.77 us 280.00 us 1649.00 us 13 STATFS
33.85 112.50 us 53.00 us 9059.00 us 533 WRITE
58.70 4521.65 us 53.00 us 100893.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3868440883905 bytes
Data Written: 2492643392426 bytes
Brick: fs-disk2:/storage/disk2d
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 2 1 9
No. of Writes: 23 61 7028
Block Size: 16b+ 32b+ 64b+
No. of Reads: 4 9 18
No. of Writes: 753276 104 317
Block Size: 128b+ 256b+ 512b+
No. of Reads: 23 326 165
No. of Writes: 934 325 540
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 791 1187 19864
No. of Writes: 3304 35873 5996879
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 306443 33611 34280
No. of Writes: 42368594 8960333 4239658
Block Size: 65536b+ 131072b+
No. of Reads: 49603 28859798
No. of Writes: 1087801 12883793
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 76783 FORGET
0.00 0.00 us 0.00 us 0.00 us 303450 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.05 68.00 us 68.00 us 68.00 us 1 STAT
0.73 86.25 us 32.00 us 241.00 us 12 OPENDIR
5.91 646.23 us 272.00 us 1301.00 us 13 STATFS
93.32 5770.74 us 78.00 us 116761.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3792036640532 bytes
Data Written: 2573766816955 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 2 1 9
No. of Writes: 23 61 7028
Block Size: 16b+ 32b+ 64b+
No. of Reads: 4 9 18
No. of Writes: 753276 104 317
Block Size: 128b+ 256b+ 512b+
No. of Reads: 23 326 165
No. of Writes: 934 325 540
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 791 1187 19864
No. of Writes: 3304 35873 5996879
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 306443 33611 34280
No. of Writes: 42368594 8960333 4239658
Block Size: 65536b+ 131072b+
No. of Reads: 49603 28859798
No. of Writes: 1087801 12883793
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 76783 FORGET
0.00 0.00 us 0.00 us 0.00 us 303450 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.05 68.00 us 68.00 us 68.00 us 1 STAT
0.73 86.25 us 32.00 us 241.00 us 12 OPENDIR
5.91 646.23 us 272.00 us 1301.00 us 13 STATFS
93.32 5770.74 us 78.00 us 116761.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3792036640532 bytes
Data Written: 2573766816955 bytes
Brick: fs-disk2:/storage/disk2i
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 20 3578 27548
Block Size: 16b+ 32b+ 64b+
No. of Reads: 3 6 22
No. of Writes: 724964 48 45
Block Size: 128b+ 256b+ 512b+
No. of Reads: 23 337 92
No. of Writes: 76 329 444
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 704 992 33244
No. of Writes: 872 20763 5178705
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 586213 48447 36707
No. of Writes: 44974188 5415301 3731632
Block Size: 65536b+ 131072b+
No. of Reads: 46456 27953620
No. of Writes: 1022266 15271937
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 52503 FORGET
0.00 0.00 us 0.00 us 0.00 us 255832 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.13 105.00 us 105.00 us 105.00 us 1 STAT
1.30 84.92 us 33.00 us 213.00 us 12 OPENDIR
13.80 831.38 us 280.00 us 1633.00 us 13 STATFS
84.76 2886.22 us 47.00 us 63229.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3675283066972 bytes
Data Written: 2810419451093 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 20 3578 27548
Block Size: 16b+ 32b+ 64b+
No. of Reads: 3 6 22
No. of Writes: 724964 48 45
Block Size: 128b+ 256b+ 512b+
No. of Reads: 23 337 92
No. of Writes: 76 329 444
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 704 992 33244
No. of Writes: 872 20763 5178705
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 586213 48447 36707
No. of Writes: 44974188 5415301 3731632
Block Size: 65536b+ 131072b+
No. of Reads: 46456 27953620
No. of Writes: 1022266 15271937
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 52503 FORGET
0.00 0.00 us 0.00 us 0.00 us 255832 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.13 105.00 us 105.00 us 105.00 us 1 STAT
1.30 84.92 us 33.00 us 213.00 us 12 OPENDIR
13.80 831.38 us 280.00 us 1633.00 us 13 STATFS
84.76 2886.22 us 47.00 us 63229.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3675283066972 bytes
Data Written: 2810419451093 bytes
Brick: fs-disk2:/storage/disk2e
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 1 6
No. of Writes: 20 16239 63143
Block Size: 16b+ 32b+ 64b+
No. of Reads: 3 11 18
No. of Writes: 1263764 175 84
Block Size: 128b+ 256b+ 512b+
No. of Reads: 17 533 159
No. of Writes: 132 372 670
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 763 1393 37962
No. of Writes: 6789 49897 6336454
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 666476 56464 50579
No. of Writes: 36558716 7338359 3327733
Block Size: 65536b+ 131072b+
No. of Reads: 61462 32430220
No. of Writes: 1613059 17873798
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 52379 FORGET
0.00 0.00 us 0.00 us 0.00 us 230163 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.14 169.00 us 169.00 us 169.00 us 1 STAT
0.76 77.00 us 34.00 us 173.00 us 12 OPENDIR
9.09 845.92 us 263.00 us 2666.00 us 13 STATFS
90.00 4731.83 us 56.00 us 104925.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4264934347534 bytes
Data Written: 3183057746349 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 1 6
No. of Writes: 20 16239 63143
Block Size: 16b+ 32b+ 64b+
No. of Reads: 3 11 18
No. of Writes: 1263764 175 84
Block Size: 128b+ 256b+ 512b+
No. of Reads: 17 533 159
No. of Writes: 132 372 670
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 763 1393 37962
No. of Writes: 6789 49897 6336454
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 666476 56464 50579
No. of Writes: 36558716 7338359 3327733
Block Size: 65536b+ 131072b+
No. of Reads: 61462 32430220
No. of Writes: 1613059 17873798
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 52379 FORGET
0.00 0.00 us 0.00 us 0.00 us 230163 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.14 169.00 us 169.00 us 169.00 us 1 STAT
0.76 77.00 us 34.00 us 173.00 us 12 OPENDIR
9.09 845.92 us 263.00 us 2666.00 us 13 STATFS
90.00 4731.83 us 56.00 us 104925.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4264934347534 bytes
Data Written: 3183057746349 bytes
Brick: fs-disk2:/storage/disk2b
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 1 2
No. of Writes: 30 84 7511
Block Size: 16b+ 32b+ 64b+
No. of Reads: 2 5 29
No. of Writes: 1655560 185 422
Block Size: 128b+ 256b+ 512b+
No. of Reads: 22 259 112
No. of Writes: 1646 701 450
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 633 1091 27587
No. of Writes: 950 11269 6848760
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 469437 42466 40711
No. of Writes: 40172731 7534709 3517428
Block Size: 65536b+ 131072b+
No. of Reads: 52771 32605919
No. of Writes: 1884565 12076893
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 51766 FORGET
0.00 0.00 us 0.00 us 0.00 us 238087 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.05 127.00 us 127.00 us 127.00 us 1 STAT
0.48 103.33 us 36.00 us 241.00 us 12 OPENDIR
3.19 632.00 us 262.00 us 1235.00 us 13 STATFS
45.09 5055.96 us 64.00 us 112717.00 us 23 LOOKUP
51.20 16505.50 us 69.00 us 123439.00 us 8 READDIRP
Duration: 1986036 seconds
Data Read: 4284923708152 bytes
Data Written: 2510476723393 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 1 2
No. of Writes: 30 84 7511
Block Size: 16b+ 32b+ 64b+
No. of Reads: 2 5 29
No. of Writes: 1655560 185 422
Block Size: 128b+ 256b+ 512b+
No. of Reads: 22 259 112
No. of Writes: 1646 701 450
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 633 1091 27587
No. of Writes: 950 11269 6848760
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 469437 42466 40711
No. of Writes: 40172731 7534709 3517428
Block Size: 65536b+ 131072b+
No. of Reads: 52771 32605919
No. of Writes: 1884565 12076893
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 51766 FORGET
0.00 0.00 us 0.00 us 0.00 us 238087 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335488 RELEASEDIR
0.05 127.00 us 127.00 us 127.00 us 1 STAT
0.48 103.33 us 36.00 us 241.00 us 12 OPENDIR
3.19 632.00 us 262.00 us 1235.00 us 13 STATFS
45.09 5055.96 us 64.00 us 112717.00 us 23 LOOKUP
51.20 16505.50 us 69.00 us 123439.00 us 8 READDIRP
Duration: 1986036 seconds
Data Read: 4284923708152 bytes
Data Written: 2510476723393 bytes
Brick: fs-disk2:/storage/disk2x
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 1 0
No. of Writes: 3 38 13386
Block Size: 8b+ 16b+ 32b+
No. of Reads: 10 2 21
No. of Writes: 61870 1251517 340
Block Size: 64b+ 128b+ 256b+
No. of Reads: 11 26 366
No. of Writes: 105 235 402
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 45 299 726
No. of Writes: 557 1385 9453
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 62983 1185549 86014
No. of Writes: 5857418 44739090 7120219
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 57842 58475 30049117
No. of Writes: 3448151 1472086 13736011
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 46569 FORGET
0.00 0.00 us 0.00 us 0.00 us 221882 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335452 RELEASEDIR
0.03 49.00 us 49.00 us 49.00 us 1 STAT
0.48 70.08 us 24.00 us 155.00 us 12 OPENDIR
6.51 872.38 us 318.00 us 1644.00 us 13 STATFS
17.03 91.55 us 49.00 us 753.00 us 324 WRITE
75.94 5750.13 us 43.00 us 119003.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3957221521620 bytes
Data Written: 2691859417510 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 1 0
No. of Writes: 3 38 13386
Block Size: 8b+ 16b+ 32b+
No. of Reads: 10 2 21
No. of Writes: 61870 1251517 340
Block Size: 64b+ 128b+ 256b+
No. of Reads: 11 26 366
No. of Writes: 105 235 402
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 45 299 726
No. of Writes: 557 1385 9453
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 62983 1185549 86014
No. of Writes: 5857418 44739090 7120219
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 57842 58475 30049117
No. of Writes: 3448151 1472086 13736011
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 46569 FORGET
0.00 0.00 us 0.00 us 0.00 us 221882 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335452 RELEASEDIR
0.03 49.00 us 49.00 us 49.00 us 1 STAT
0.48 70.08 us 24.00 us 155.00 us 12 OPENDIR
6.51 872.38 us 318.00 us 1644.00 us 13 STATFS
17.03 91.55 us 49.00 us 753.00 us 324 WRITE
75.94 5750.13 us 43.00 us 119003.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 3957221521620 bytes
Data Written: 2691859417510 bytes
Brick: fs-disk2:/storage/disk2n
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 1
No. of Writes: 1 13 4444
Block Size: 8b+ 16b+ 32b+
No. of Reads: 7 2 11
No. of Writes: 28553 1067713 57
Block Size: 64b+ 128b+ 256b+
No. of Reads: 15 77 290
No. of Writes: 267 1711 362
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 163 767 1370
No. of Writes: 398 3222 46824
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 45635 831262 71660
No. of Writes: 10618719 46250340 7444079
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 62360 79696 30939710
No. of Writes: 3474292 1425708 14514345
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57638 FORGET
0.00 0.00 us 0.00 us 0.00 us 273547 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335473 RELEASEDIR
0.11 88.00 us 88.00 us 88.00 us 1 STAT
2.08 145.08 us 32.00 us 867.00 us 12 OPENDIR
10.58 679.46 us 287.00 us 1777.00 us 13 STATFS
87.23 3167.17 us 48.00 us 70013.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4073195963222 bytes
Data Written: 2843693209398 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 1
No. of Writes: 1 13 4444
Block Size: 8b+ 16b+ 32b+
No. of Reads: 7 2 11
No. of Writes: 28553 1067713 57
Block Size: 64b+ 128b+ 256b+
No. of Reads: 15 77 290
No. of Writes: 267 1711 362
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 163 767 1370
No. of Writes: 398 3222 46824
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 45635 831262 71660
No. of Writes: 10618719 46250340 7444079
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 62360 79696 30939710
No. of Writes: 3474292 1425708 14514345
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 57638 FORGET
0.00 0.00 us 0.00 us 0.00 us 273547 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335473 RELEASEDIR
0.11 88.00 us 88.00 us 88.00 us 1 STAT
2.08 145.08 us 32.00 us 867.00 us 12 OPENDIR
10.58 679.46 us 287.00 us 1777.00 us 13 STATFS
87.23 3167.17 us 48.00 us 70013.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4073195963222 bytes
Data Written: 2843693209398 bytes
Brick: fs-disk2:/storage/disk2s
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 8
No. of Writes: 32 10435 55686
Block Size: 16b+ 32b+ 64b+
No. of Reads: 2 3 25
No. of Writes: 1128338 253 98
Block Size: 128b+ 256b+ 512b+
No. of Reads: 40 626 584
No. of Writes: 826 432 569
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 3722 3204 38092
No. of Writes: 3533 13438 14705697
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 665490 59640 47624
No. of Writes: 47468109 5763490 3321768
Block Size: 65536b+ 131072b+
No. of Reads: 60196 31026864
No. of Writes: 1373769 19646558
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 37669 FORGET
0.00 0.00 us 0.00 us 0.00 us 236246 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335467 RELEASEDIR
0.02 36.00 us 36.00 us 36.00 us 1 STAT
0.65 80.25 us 35.00 us 123.00 us 12 OPENDIR
6.03 683.85 us 301.00 us 1207.00 us 13 STATFS
28.14 120.64 us 58.00 us 724.00 us 344 WRITE
65.16 4178.48 us 53.00 us 93204.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4080778290391 bytes
Data Written: 3488412950060 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 8
No. of Writes: 32 10435 55686
Block Size: 16b+ 32b+ 64b+
No. of Reads: 2 3 25
No. of Writes: 1128338 253 98
Block Size: 128b+ 256b+ 512b+
No. of Reads: 40 626 584
No. of Writes: 826 432 569
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 3722 3204 38092
No. of Writes: 3533 13438 14705697
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 665490 59640 47624
No. of Writes: 47468109 5763490 3321768
Block Size: 65536b+ 131072b+
No. of Reads: 60196 31026864
No. of Writes: 1373769 19646558
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 37669 FORGET
0.00 0.00 us 0.00 us 0.00 us 236243 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335467 RELEASEDIR
0.02 36.00 us 36.00 us 36.00 us 1 STAT
0.65 80.25 us 35.00 us 123.00 us 12 OPENDIR
6.03 683.85 us 301.00 us 1207.00 us 13 STATFS
28.14 120.64 us 58.00 us 724.00 us 344 WRITE
65.16 4178.48 us 53.00 us 93204.00 us 23 LOOKUP
Duration: 1986036 seconds
Data Read: 4080778290391 bytes
Data Written: 3488412950060 bytes
Brick: fs-disk3:/storage/disk3a
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 29 9207 14136
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 1506450 195 128
Block Size: 128b+ 256b+ 512b+
No. of Reads: 2 3 2
No. of Writes: 147 861 669
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 17 790 55454
No. of Writes: 6671 4656 14525176
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 1104707 77003 49913
No. of Writes: 48697616 3345008 3395449
Block Size: 65536b+ 131072b+
No. of Reads: 56765 14814839
No. of Writes: 1759924 40537209
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 180781 FORGET
0.00 0.00 us 0.00 us 0.00 us 139184 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335451 RELEASEDIR
0.33 136.00 us 136.00 us 136.00 us 1 STAT
1.34 45.92 us 31.00 us 75.00 us 12 OPENDIR
7.29 231.31 us 183.00 us 285.00 us 13 STATFS
91.05 1632.91 us 51.00 us 35331.00 us 23 LOOKUP
Duration: 1986170 seconds
Data Read: 1958970926561 bytes
Data Written: 6224168303087 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 29 9207 14136
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 1506450 195 128
Block Size: 128b+ 256b+ 512b+
No. of Reads: 2 3 2
No. of Writes: 147 861 669
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 17 790 55454
No. of Writes: 6671 4656 14525176
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 1104707 77003 49913
No. of Writes: 48697616 3345008 3395449
Block Size: 65536b+ 131072b+
No. of Reads: 56765 14814839
No. of Writes: 1759924 40537209
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 180781 FORGET
0.00 0.00 us 0.00 us 0.00 us 139184 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335451 RELEASEDIR
0.33 136.00 us 136.00 us 136.00 us 1 STAT
1.34 45.92 us 31.00 us 75.00 us 12 OPENDIR
7.29 231.31 us 183.00 us 285.00 us 13 STATFS
91.05 1632.91 us 51.00 us 35331.00 us 23 LOOKUP
Duration: 1986170 seconds
Data Read: 1958970926561 bytes
Data Written: 6224168303087 bytes
Brick: fs-disk3:/storage/disk3b
-------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 2 30 17
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 1 0
No. of Writes: 107 1260245 169
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 2 1
No. of Writes: 458 1781 942
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 4 9 465
No. of Writes: 722 4519 22658
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 11517 222524 19810
No. of Writes: 13504637 54863025 3252134
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 17651 24629 10317751
No. of Writes: 3412859 1576536 39766617
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 181199 FORGET
0.00 0.00 us 0.00 us 0.00 us 164707 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335451 RELEASEDIR
0.11 41.00 us 41.00 us 41.00 us 1 STAT
1.71 52.42 us 31.00 us 108.00 us 12 OPENDIR
8.70 246.38 us 189.00 us 400.00 us 13 STATFS
89.48 1431.78 us 55.00 us 30842.00 us 23 LOOKUP
Duration: 1986170 seconds
Data Read: 1357524224644 bytes
Data Written: 6138513371375 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 0
No. of Writes: 2 30 17
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 1 0
No. of Writes: 107 1260245 169
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 2 1
No. of Writes: 458 1781 942
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 4 9 465
No. of Writes: 722 4519 22658
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 11517 222524 19810
No. of Writes: 13504637 54863025 3252134
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 17651 24629 10317751
No. of Writes: 3412859 1576536 39766617
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 181199 FORGET
0.00 0.00 us 0.00 us 0.00 us 164707 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335451 RELEASEDIR
0.11 41.00 us 41.00 us 41.00 us 1 STAT
1.71 52.42 us 31.00 us 108.00 us 12 OPENDIR
8.70 246.38 us 189.00 us 400.00 us 13 STATFS
89.48 1431.78 us 55.00 us 30842.00 us 23 LOOKUP
Duration: 1986170 seconds
Data Read: 1357524224644 bytes
Data Written: 6138513371375 bytes
Brick: fs-disk3:/storage/disk3c
-------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 21 26595 55485
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 1516716 176 108
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 3 2
No. of Writes: 102 626 633
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 11 465 41485
No. of Writes: 2042 9289 16365920
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 830172 64623 52430
No. of Writes: 52379250 2117265 3642628
Block Size: 65536b+ 131072b+
No. of Reads: 69128 12576665
No. of Writes: 1884882 40963295
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 181378 FORGET
0.00 0.00 us 0.00 us 0.00 us 169199 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335451 RELEASEDIR
0.10 46.00 us 46.00 us 46.00 us 1 STAT
1.21 44.92 us 29.00 us 90.00 us 12 OPENDIR
7.47 256.69 us 175.00 us 463.00 us 13 STATFS
91.23 1773.00 us 53.00 us 38905.00 us 23 LOOKUP
Duration: 1986170 seconds
Data Read: 1664583476377 bytes
Data Written: 6303233665142 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 21 26595 55485
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 1516716 176 108
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 3 2
No. of Writes: 102 626 633
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 11 465 41485
No. of Writes: 2042 9289 16365920
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 830172 64623 52430
No. of Writes: 52379250 2117265 3642628
Block Size: 65536b+ 131072b+
No. of Reads: 69128 12576665
No. of Writes: 1884882 40963295
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 181378 FORGET
0.00 0.00 us 0.00 us 0.00 us 169199 RELEASE
0.00 0.00 us 0.00 us 0.00 us 15335451 RELEASEDIR
0.10 46.00 us 46.00 us 46.00 us 1 STAT
1.21 44.92 us 29.00 us 90.00 us 12 OPENDIR
7.47 256.69 us 175.00 us 463.00 us 13 STATFS
91.23 1773.00 us 53.00 us 38905.00 us 23 LOOKUP
Duration: 1986170 seconds
Data Read: 1664583476377 bytes
Data Written: 6303233665142 bytes
[root at disk2 ~]# gluster volume profile shared stop
On Mar 9, 2013, at 12:01 PM, Anand Avati <anand.avati at gmail.com> wrote:
> Can you run some read experiments using a ramfs backed brick? Also, what are your jumbo frames setting? Can you try tweaking that? Is there anything suspicious in 'top' in either the client or the server? It would also be very useful if you turned on profiling and give us the output.
>
> server# gluster volume profile <name> on
>
> client# dd ... # test 1
>
> server# gluster volume profile <name> info
>
> client# dd .... # test 2
>
> server# gluster volume profile <name> info
>
> Thanks!
> Avati
>
>
> On Fri, Mar 8, 2013 at 6:40 AM, Thomas Wakefield <twake at cola.iges.org> wrote:
> I am still confused how ext4 is suddenly slow to read when it's behind Gluster, but plenty fast stand alone reading?
>
> And it writes really fast from both the server and client.
>
>
>
> On Mar 8, 2013, at 4:07 AM, "Jon Tegner" <tegner at renget.se> wrote:
>
>> We had issues with ext4 about a bit less than a year ago, at that time I upgraded the servers to CentOS-6.2. But that gave us large problems (more than slow reads). Since I didn't want to reformat the disks at that time (and switch to XFS) I went back to CentOS-5.5 (which we had used before). On some link (think it was https://bugzilla.redhat.com/show_bug.cgi?id=713546 but can't seem to reach that now) it was stated that the ext4-issue was present even on later versions of CentOS-5 (I _think_ 5.8 was affected).
>>
>> Are there hope that the "ext4-issue" will be solved in later kernels/versions of gluster? If not, it seems one is eventually forced to switch to XFS.
>>
>> Regards,
>>
>> /jon
>>
>> On Mar 8, 2013 03:27 "Thomas Wakefield" <twake at iges.org> wrote:
>>
>>>
>>> inode size is 256.
>>>
>>>
>>>
>>>
>>> Pretty stuck with these settings and ext4. I missed the memo that
>>> Gluster started to prefer xfs, back in the 2.x days xfs was not the
>>> preferred filesystem. At this point it's a 340TB filesystem with 160TB
>>> used. I just added more space, and was doing some followup testing and
>>> wasn't impressed with the results. But I am sure I was happier before
>>> with the performance.
>>>
>>>
>>>
>>>
>>> Still running CentOS 5.8
>>>
>>>
>>>
>>>
>>> Anything else I could look at?
>>>
>>>
>>>
>>>
>>> Thanks, Tom
>>>
>>>
>>>
>>>
>>>
>>> On Mar 7, 2013, at 5:04 PM, Bryan Whitehead <<driver at megahappy.net>>
>>> wrote:
>>>
>>>
>>> > I'm sure you know, but xfs is the recommended filesystem for
>>> > glusterfs. Ext4 has a number of issues. (Particularly on
>>> > CentOS/Redhat6).
>>> >
>>> >
>>> >
>>> > The default inode size for ext4 (and xfs) is small for the number of
>>> > extended attributes glusterfs uses. This causes a minor hit in
>>> > performance on xfs if theextended attributes grow more than 265 (xfs
>>> > default size). In xfs, this is fixed by setting the size of an inode
>>> > to 512. How big the impact is on ext4 is something I don't know
>>> > offhand. But looking at a couple of boxes I have it looks like some
>>> > ext4 filesystems have 128 inode size and some have 256 inode size
>>> > (both of which are too small for glusterfs). The performance hit is
>>> > everytimeextended attributes need to be read several inodes need to be
>>> > seeked and found.
>>> >
>>> >
>>> >
>>> > run "dumpe2fs -h <blockdevice> | grep size" on your ext4 mountpoints.
>>> >
>>> >
>>> >
>>> >
>>> > If it is not too much of a bother - I'd try xfs as your filesystem for
>>> > the bricks
>>> >
>>> >
>>> >
>>> >
>>> > mkfs.xfs -i size=512 <blockdevice>
>>> >
>>> >
>>> >
>>> >
>>> > Please see this for more detailed info:
>>> >
>>> > <https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Storage/2.0/ht
>>> > ml-single/Administration_Guide/index.html#chap-User_Guide-Setting_Volu
>>> > mes>
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > On Thu, Mar 7, 2013 at 12:08 PM, Thomas Wakefield
>>> > <<twake at cola.iges.org>> wrote:
>>> >
>>> > > Everything is built as ext4, no options other than
>>> > > lazy_itable_init=1 when I built the filesystems.
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > Server mount example:
>>> > >
>>> > > LABEL=disk2a /storage/disk2a ext4defaults 0 0
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > Client mount:
>>> > >
>>> > > fs-disk2:/shared /shared glusterfs defaults 0 0
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > Remember, the slow reads are only from gluster clients, the disks
>>> > > are really fast when I am local on the server testing the disks.
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > -Tom
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Mar 7, 2013, at 1:09 PM, Bryan Whitehead <<driver at megahappy.net>>
>>> > > wrote:
>>> > >
>>> > >
>>> > > > Was just thinking, what is your mount options for your bricks
>>> > > > (using inode64?)? Also, you are using xfs... right?
>>> > > >
>>> > > >
>>> > > >
>>> > > > When you created the filesystems did you allocate more inode
>>> > > > space? -i size=512 ?
>>> > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > On Thu, Mar 7, 2013 at 5:49 AM, Thomas Wakefield
>>> > > > <<twake at cola.iges.org>> wrote:
>>> > > >
>>> > > > > Still looking for help.
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > >
>>> > > > > On Mar 4, 2013, at 7:43 AM, Thomas Wakefield <<twake at iges.org>>
>>> > > > > wrote:
>>> > > > >
>>> > > > >
>>> > > > > > Also, I tested an NFS mount over the same 10GB link, and was
>>> > > > > > able to pull almost 200MB/s. But Gluster is still much slower.
>>> > > > > > Also I tested running it for a longer test, 105GB of data, and
>>> > > > > > still showed that writing is MUCH faster. Which makes no sense
>>> > > > > > when the disks can read 2x as fast as they can write.
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > Any other thoughts?
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > [root at cpu_crew1 ~]# dd if=/dev/zero
>>> > > > > > of=/shared/working/benchmark/test.cpucrew1 bs=512k
>>> > > > > > count=200000 ; dd if=/shared/working/benchmark/test.cpucrew1
>>> > > > > > of=/dev/null bs=512k
>>> > > > > >
>>> > > > > > 200000+0 records in
>>> > > > > >
>>> > > > > > 200000+0 records out
>>> > > > > >
>>> > > > > > 104857600000 bytes (105 GB) copied, 159.135 seconds, 659 MB/s
>>> > > > > >
>>> > > > > > 200000+0 records in
>>> > > > > >
>>> > > > > > 200000+0 records out
>>> > > > > >
>>> > > > > > 104857600000 bytes (105 GB) copied, 1916.87 seconds, 54.7 MB/s
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > >
>>> > > > > > On Mar 1, 2013, at 9:58 AM, Thomas Wakefield
>>> > > > > > <<twake at iges.org>> wrote:
>>> > > > > >
>>> > > > > >
>>> > > > > > > The max setting for performance.read-ahead-page-count is 16,
>>> > > > > > > which I did just try. No significant change.
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > Any other setting options?
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Feb 28, 2013, at 10:18 PM, Anand Avati
>>> > > > > > > <<anand.avati at gmail.com>> wrote:
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > > Can you try "gluster volume set <volname>
>>> > > > > > > > performance.read-ahead-page-count 64" or some value higher
>>> > > > > > > > or lower?
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > Avati
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Thu, Feb 28, 2013 at 7:15 PM, Thomas Wakefield
>>> > > > > > > > <<twake at iges.org>> wrote:
>>> > > > > > > >
>>> > > > > > > > > Good point, forgot to set a blcoksize, here are the
>>> > > > > > > > > redone dd tests:
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > [root at cpu_crew1 ~]# dd
>>> > > > > > > > > if=/shared/working/benchmark/test.cpucrew1 of=/dev/null
>>> > > > > > > > > bs=128k
>>> > > > > > > > >
>>> > > > > > > > > 40000+0 records in
>>> > > > > > > > >
>>> > > > > > > > > 40000+0 records out
>>> > > > > > > > >
>>> > > > > > > > > 5242880000 bytes (5.2 GB) copied, 65.4928 seconds, 80.1
>>> > > > > > > > > MB/s
>>> > > > > > > > >
>>> > > > > > > > > [root at cpu_crew1 ~]# dd
>>> > > > > > > > > if=/shared/working/benchmark/test.cpucrew1 of=/dev/null
>>> > > > > > > > > bs=1M
>>> > > > > > > > >
>>> > > > > > > > > 5000+0 records in
>>> > > > > > > > >
>>> > > > > > > > > 5000+0 records out
>>> > > > > > > > >
>>> > > > > > > > > 5242880000 bytes (5.2 GB) copied, 49.0907 seconds, 107
>>> > > > > > > > > MB/s
>>> > > > > > > > >
>>> > > > > > > > > [root at cpu_crew1 ~]# dd
>>> > > > > > > > > if=/shared/working/benchmark/test.cpucrew1 of=/dev/null
>>> > > > > > > > > bs=4M
>>> > > > > > > > >
>>> > > > > > > > > 1250+0 records in
>>> > > > > > > > >
>>> > > > > > > > > 1250+0 records out
>>> > > > > > > > >
>>> > > > > > > > > 5242880000 bytes (5.2 GB) copied, 44.5724 seconds, 118
>>> > > > > > > > > MB/s
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > Still not impressive.
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > -Tom
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > On Feb 28, 2013, at 8:42 PM, Jeff Anderson-Lee
>>> > > > > > > > > <<jonah at eecs.berkeley.edu>> wrote:
>>> > > > > > > > >
>>> > > > > > > > >
>>> > > > > > > > > > Thomas,
>>> > > > > > > > > >
>>> > > > > > > > > > You have not specified a block size, so you are doing
>>> > > > > > > > > > a huge number of small(ish) reads with associated
>>> > > > > > > > > > round trips. What happens with dd bs=128k ..?
>>> > > > > > > > > >
>>> > > > > > > > > > Jeff Anderson-Lee
>>> > > > > > > > > >
>>> > > > > > > > > > On 2/28/2013 5:30 PM, Thomas Wakefield wrote:
>>> > > > > > > > > >
>>> > > > > > > > > >
>>> > > > > > > > > > > Did a fresh dd test just to confirm, same results:
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > > [root at cpu_crew1 benchmark]# dd if=/dev/zero
>>> > > > > > > > > > > of=/shared/working/benchmark/test.cpucrew1 bs=512k
>>> > > > > > > > > > > count=10000
>>> > > > > > > > > > >
>>> > > > > > > > > > > 10000+0 records in
>>> > > > > > > > > > >
>>> > > > > > > > > > > 10000+0 records out
>>> > > > > > > > > > >
>>> > > > > > > > > > > 5242880000 bytes (5.2 GB) copied, 7.43695 seconds,
>>> > > > > > > > > > > 705 MB/s
>>> > > > > > > > > > >
>>> > > > > > > > > > > [root at cpu_crew1 benchmark]# dd
>>> > > > > > > > > > > if=/shared/working/benchmark/test.cpucrew1
>>> > > > > > > > > > > of=/dev/null
>>> > > > > > > > > > >
>>> > > > > > > > > > > 552126+0 records in
>>> > > > > > > > > > >
>>> > > > > > > > > > > 552125+0 records out
>>> > > > > > > > > > >
>>> > > > > > > > > > > 282688000 bytes (283 MB) copied, 37.8514 seconds,
>>> > > > > > > > > > > 7.5 MB/s
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > > On Feb 28, 2013, at 8:14 PM, Bryan Whitehead
>>> > > > > > > > > > > <<driver at megahappy.net>> wrote:
>>> > > > > > > > > > >
>>> > > > > > > > > > >
>>> > > > > > > > > > > > How are you doing the reading? Is this still an
>>> > > > > > > > > > > > iozone benchmark?
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > if you simply dd if=/glustermount/bigfile
>>> > > > > > > > > > > > of=/dev/null, is the speed better?
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > On Thu, Feb 28, 2013 at 5:05 PM, Thomas Wakefield
>>> > > > > > > > > > > > <<twake at iges.org>> wrote:
>>> > > > > > > > > > > >
>>> > > > > > > > > > > > > I get great speed locally, it's only when I add
>>> > > > > > > > > > > > > gluster in that it slows down. I get 2GB/s
>>> > > > > > > > > > > > > locally to the exact same brick. It's gluster
>>> > > > > > > > > > > > > that is having the read issue (80MB/s). But
>>> > > > > > > > > > > > > Gluster can write just fine, 800MB/s.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > The blockdev idea is a good one, and I have
>>> > > > > > > > > > > > > already done it. Thanks though.
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > -Tom
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > On Feb 28, 2013, at 7:53 PM, Ling Ho
>>> > > > > > > > > > > > > <<ling at slac.stanford.edu>> wrote:
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > >
>>> > > > > > > > > > > > > > Tom,
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > What type of disks do you have? If they are
>>> > > > > > > > > > > > > > raid 5 or 6, have you try setting the
>>> > > > > > > > > > > > > > read-ahead size to 8192 or 16384 (blockdev
>>> > > > > > > > > > > > > > --setra 8192 /dev/<sd?> ?
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > ...
>>> > > > > > > > > > > > > > ling
>>> > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > On 02/28/2013 04:23 PM, Thomas Wakefield
>>> > > > > > > > > > > > > > wrote:
>>> > > > > > > > > > > > > > > Did anyone else have any ideas on
>>> > > > > > > > > > > > > > > performance tuning for reads?
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > On Feb 27, 2013, at 9:29 PM, Thomas
>>> > > > > > > > > > > > > > > Wakefield <<twake at iges.org>> wrote:
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > Bryan-
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > Yes I can write at 700-800MBytes/sec, but
>>> > > > > > > > > > > > > > > > i can only read at 70-80 MBytes/sec. I
>>> > > > > > > > > > > > > > > > would be very happy if I could get it to
>>> > > > > > > > > > > > > > > > read at the same speed it can write at.
>>> > > > > > > > > > > > > > > > And the 70-80 is sequential, not random
>>> > > > > > > > > > > > > > > > for reads, same exact test commands on the
>>> > > > > > > > > > > > > > > > disk server are in the 2+GB/s range, so I
>>> > > > > > > > > > > > > > > > know the disk server can do it.
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > -Tom
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > On Feb 27, 2013, at 7:41 PM, Bryan
>>> > > > > > > > > > > > > > > > Whitehead <<driver at megahappy.net>> wrote:
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > Are your figures 700-800MByte/sec?
>>> > > > > > > > > > > > > > > > > Because that is probably as fast as your
>>> > > > > > > > > > > > > > > > > 10G nic cards are able to do. You can
>>> > > > > > > > > > > > > > > > > test that by trying to push a large
>>> > > > > > > > > > > > > > > > > amount of data over nc or ftp.
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > Might want to try Infiniband. 40G cards
>>> > > > > > > > > > > > > > > > > are pretty routine.
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > On Wed, Feb 27, 2013 at 3:45 PM, Thomas
>>> > > > > > > > > > > > > > > > > Wakefield <<twake at iges.org>> wrote:
>>> > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > I also get the same performance
>>> > > > > > > > > > > > > > > > > > running iozone for large file
>>> > > > > > > > > > > > > > > > > > sizes, iozone -u 1 -r 512k -s 2G -I
>>> > > > > > > > > > > > > > > > > > -F.
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > Large file IO is what I need the
>>> > > > > > > > > > > > > > > > > > system to do. I am just shocked at the
>>> > > > > > > > > > > > > > > > > > huge difference between local IO and
>>> > > > > > > > > > > > > > > > > > gluster client IO. I know there should
>>> > > > > > > > > > > > > > > > > > be some difference, but 10x is
>>> > > > > > > > > > > > > > > > > > unacceptable.
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > -Tom
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > On Feb 27, 2013, at 5:31 PM, Bryan
>>> > > > > > > > > > > > > > > > > > Whitehead <<driver at megahappy.net>>
>>> > > > > > > > > > > > > > > > > > wrote:
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > Every time you open/close a file or
>>> > > > > > > > > > > > > > > > > > > a directory you will have to wait
>>> > > > > > > > > > > > > > > > > > > for locks which take time. This is
>>> > > > > > > > > > > > > > > > > > > totally expected.
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > Why don't you share what you want to
>>> > > > > > > > > > > > > > > > > > > do? iozone benchmarks look like crap
>>> > > > > > > > > > > > > > > > > > > but serving qcow2 files to qemu
>>> > > > > > > > > > > > > > > > > > > works fantastic for me. What are you
>>> > > > > > > > > > > > > > > > > > > doing? Make a benchmark that does
>>> > > > > > > > > > > > > > > > > > > that. If you are going to have many
>>> > > > > > > > > > > > > > > > > > > files with a wide variety of sizes
>>> > > > > > > > > > > > > > > > > > > glusterfs/fuse might not be what you
>>> > > > > > > > > > > > > > > > > > > are looking for.
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > On Wed, Feb 27, 2013 at 12:56 PM,
>>> > > > > > > > > > > > > > > > > > > Thomas Wakefield
>>> > > > > > > > > > > > > > > > > > > <<twake at cola.iges.org>> wrote:
>>> > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > I have tested everything, small
>>> > > > > > > > > > > > > > > > > > > > and large files. I have used file
>>> > > > > > > > > > > > > > > > > > > > sizes ranging from 128k up to
>>> > > > > > > > > > > > > > > > > > > > multiple GB files. All the reads
>>> > > > > > > > > > > > > > > > > > > > are bad.
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > Here is a fairly exhaustive iozone
>>> > > > > > > > > > > > > > > > > > > > auto test:
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > random random bkwd record stride
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > KB reclen write rewrite read
>>> > > > > > > > > > > > > > > > > > > > reread read write read rewrite
>>> > > > > > > > > > > > > > > > > > > > read fwrite frewrite fread freread
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 64 4 40222 63492 26868 30060 1620
>>> > > > > > > > > > > > > > > > > > > > 71037 1572 70570 31294 77096 72475
>>> > > > > > > > > > > > > > > > > > > > 14736 13928
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 64 8 99207 116366 13591 13513 3214
>>> > > > > > > > > > > > > > > > > > > > 97690 3155 109978
>>> > > > > > > > > > > > > > > > > > > > <tel:3155%20%C2%A0%20109978> 28920
>>> > > > > > > > > > > > > > > > > > > > 152018 158480 18936 17625
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 64 16 230257 253766 25156 28713
>>> > > > > > > > > > > > > > > > > > > > 10867 223732 8873 244297 54796
>>> > > > > > > > > > > > > > > > > > > > 303383 312204 15062 13545
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 64 32 255943 234481 5735102
>>> > > > > > > > > > > > > > > > > > > > 7100397 11897 318502 13681 347801
>>> > > > > > > > > > > > > > > > > > > > 24214 695778 528618 25838 28094
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 64 64 214096 681644 6421025
>>> > > > > > > > > > > > > > > > > > > > 7100397 27453 292156 28117 621657
>>> > > > > > > > > > > > > > > > > > > > 27338 376062 512471 28569 32534
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 128 4 74329 75468 26428
>>> > > > > > > > > > > > > > > > > > > > <tel:75468%20%C2%A0%20%C2%A026428>
>>> > > > > > > > > > > > > > > > > > > > 41089 1131 72857 1118 66976 1597
>>> > > > > > > > > > > > > > > > > > > > 73778 78343 13351 13026
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 128 8 100862 135170 24966
>>> > > > > > > > > > > > > > > > > > > > <tel:135170%20%C2%A0%20%C2%A024966
>>> > > > > > > > > > > > > > > > > > > > > 16734 2617 118966 2560 120406
>>> > > > > > > > > > > > > > > > > > > > > 39156 125121 146613 16177 16180
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 128 16 115114 253983 28212 17854
>>> > > > > > > > > > > > > > > > > > > > 5307 246180 5431 229843 47335
>>> > > > > > > > > > > > > > > > > > > > 255920 271173 27256 24445
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 128 32 256042 391360 39848 64258
>>> > > > > > > > > > > > > > > > > > > > 11329 290230 9905 429563 38176
>>> > > > > > > > > > > > > > > > > > > > 490380 463696 20917 19219
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 128 64 248573 592699 4557257
>>> > > > > > > > > > > > > > > > > > > > 6812590 19583 452366 29263 603357
>>> > > > > > > > > > > > > > > > > > > > 42967 814915 692017 76327 37604
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 128 128 921183 526444 5603747
>>> > > > > > > > > > > > > > > > > > > > 5379161 45614 390222 65441 826202
>>> > > > > > > > > > > > > > > > > > > > 41384 662962 1040839 78526 39023
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 256 4 76212 77337 40295
>>> > > > > > > > > > > > > > > > > > > > <tel:77337%20%C2%A0%20%C2%A040295>
>>> > > > > > > > > > > > > > > > > > > > 32125 1289 71866 1261 64645 1436
>>> > > > > > > > > > > > > > > > > > > > 57309 53048 23073 29550
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 256 8 126922 141976 26237
>>> > > > > > > > > > > > > > > > > > > > <tel:141976%20%C2%A0%20%C2%A026237
>>> > > > > > > > > > > > > > > > > > > > > 25130 2566 128058 2565 138981
>>> > > > > > > > > > > > > > > > > > > > > <tel:2565%20%C2%A0%20138981>
>>> > > > > > > > > > > > > > > > > > > > > 2985 125060 133603 22840 24955
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 256 16 242883 263636 41850 24371
>>> > > > > > > > > > > > > > > > > > > > 4902 250009 5290 248792 89353
>>> > > > > > > > > > > > > > > > > > > > 243821 247303 26965 26199
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 256 32 409074 439732 40101 39335
>>> > > > > > > > > > > > > > > > > > > > 11953 436870 11209 430218 83743
>>> > > > > > > > > > > > > > > > > > > > 409542 479390 30821 27750
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 256 64 259935 571502 64840 71847
>>> > > > > > > > > > > > > > > > > > > > 22537 <tel:71847%20%C2%A0%2022537>
>>> > > > > > > > > > > > > > > > > > > > 617161 23383 392047 91852 672010
>>> > > > > > > > > > > > > > > > > > > > 802614 41673 53111
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 256 128 847597 812329 185517 83198
>>> > > > > > > > > > > > > > > > > > > > 49383 <tel:83198%20%C2%A0%2049383>
>>> > > > > > > > > > > > > > > > > > > > 708831 44668 794889 74267 1180188
>>> > > > > > > > > > > > > > > > > > > > 1662639 54303 41018
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 256 256 481324 709299 5217259
>>> > > > > > > > > > > > > > > > > > > > 5320671 44668 719277 40954 808050
>>> > > > > > > > > > > > > > > > > > > > 41302 790209 771473 62224 35754
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 512 4 77667 75226 35102 29696 1337
>>> > > > > > > > > > > > > > > > > > > > 66262 1451 67680 1413 69265 69142
>>> > > > > > > > > > > > > > > > > > > > 42084 27897
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 512 8 134311 144341 30144
>>> > > > > > > > > > > > > > > > > > > > <tel:144341%20%C2%A0%20%C2%A030144
>>> > > > > > > > > > > > > > > > > > > > > 24646 2102 134143 2209 134699
>>> > > > > > > > > > > > > > > > > > > > > 2296 108110 128616 25104 29123
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 512 16 200085 248787 30235 25697
>>> > > > > > > > > > > > > > > > > > > > 4196 247240 4179 256116
>>> > > > > > > > > > > > > > > > > > > > <tel:4179%20%C2%A0%20256116> 4768
>>> > > > > > > > > > > > > > > > > > > > 250003 226436 32351 28455
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 512 32 330341 439805 26440 39284
>>> > > > > > > > > > > > > > > > > > > > 8744 457611 8006 424168
>>> > > > > > > > > > > > > > > > > > > > <tel:8006%20%C2%A0%20424168>
>>> > > > > > > > > > > > > > > > > > > > 125953 425935 448813 27660 26951
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 512 64 483906 733729 48747 41121
>>> > > > > > > > > > > > > > > > > > > > 16032 555938 17424 587256 187343
>>> > > > > > > > > > > > > > > > > > > > 366977 735740 41700 41548
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 512 128 836636 907717 69359 94921
>>> > > > > > > > > > > > > > > > > > > > 42443 <tel:94921%20%C2%A0%2042443>
>>> > > > > > > > > > > > > > > > > > > > 761031 36828 964378 123165 651383
>>> > > > > > > > > > > > > > > > > > > > 695697 58368 44459
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 512 256 520879 860437 145534
>>> > > > > > > > > > > > > > > > > > > > 135523 40267 847532 31585 663252
>>> > > > > > > > > > > > > > > > > > > > 69696 1270846 1492545 48822 48092
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 512 512 782951 973118 3099691
>>> > > > > > > > > > > > > > > > > > > > 2942541 42328 871966 46218 911184
>>> > > > > > > > > > > > > > > > > > > > 49791 953248 1036527 52723 48347
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 1024 4 76218 69362 36431 28711
>>> > > > > > > > > > > > > > > > > > > > 1137 66171 1174 68938 1125 70566
>>> > > > > > > > > > > > > > > > > > > > 70845 34942 28914
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 1024 8 126045 140524 37836
>>> > > > > > > > > > > > > > > > > > > > <tel:140524%20%C2%A0%20%C2%A037836
>>> > > > > > > > > > > > > > > > > > > > > 15664 2698 126000 2557 125566
>>> > > > > > > > > > > > > > > > > > > > > 2567 110858 127255 26764 27945
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 1024 16 243398 261429 40238 23263
>>> > > > > > > > > > > > > > > > > > > > 3987 246400 3882 260746 4093
>>> > > > > > > > > > > > > > > > > > > > 236652 236874 31429 25076
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 1024 32 383109 422076 41731 41605
>>> > > > > > > > > > > > > > > > > > > > 8277 473441 7775 415261 8588
>>> > > > > > > > > > > > > > > > > > > > 394765 407306 40089 28537
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 1024 64 590145 619156 39623 53267
>>> > > > > > > > > > > > > > > > > > > > 15051 722717 14624 753000 257294
>>> > > > > > > > > > > > > > > > > > > > 597784 620946 38619 44073
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 1024 128 1077836 1124099 56192
>>> > > > > > > > > > > > > > > > > > > > <tel:1124099%20%C2%A0%20%C2%A05619
>>> > > > > > > > > > > > > > > > > > > > 2> 64916 36851 1102176 37198
>>> > > > > > > > > > > > > > > > > > > > 1082454 281548 829175 792604 47975
>>> > > > > > > > > > > > > > > > > > > > 51913
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 1024 256 941918 1074331 72783
>>> > > > > > > > > > > > > > > > > > > > 81450 26778
>>> > > > > > > > > > > > > > > > > > > > <tel:81450%20%C2%A0%2026778>
>>> > > > > > > > > > > > > > > > > > > > 1099636 32395 1060013 183218
>>> > > > > > > > > > > > > > > > > > > > 1024121 995171 44371 45448
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 1024 512 697483 1130312 100324
>>> > > > > > > > > > > > > > > > > > > > 114682 48215 1041758 41480 1058967
>>> > > > > > > > > > > > > > > > > > > > 90156 994020 1563622 56328 46370
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 1024 1024 931702 1087111 4609294
>>> > > > > > > > > > > > > > > > > > > > 4199201 44191 949834 45594 970656
>>> > > > > > > > > > > > > > > > > > > > 56674 933525 1075676 44876 46115
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 4 71438 67066 58319 38913
>>> > > > > > > > > > > > > > > > > > > > 1147 44147 1043 42916 967 66416
>>> > > > > > > > > > > > > > > > > > > > 67205 45953 96750
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 8 141926 134567 61101 55445
>>> > > > > > > > > > > > > > > > > > > > 2596 77528 2564 80402 4258 124211
>>> > > > > > > > > > > > > > > > > > > > 120747 53888 100337
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 16 254344 255585 71550 74500
>>> > > > > > > > > > > > > > > > > > > > 5410 139365 5201 141484 5171
>>> > > > > > > > > > > > > > > > > > > > 205521 213113 67048 57304
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 32 397833 411261 56676 80027
>>> > > > > > > > > > > > > > > > > > > > 10440 <tel:80027%20%C2%A0%2010440>
>>> > > > > > > > > > > > > > > > > > > > 260034 10126 230238 10814 391665
>>> > > > > > > > > > > > > > > > > > > > 383379 79333 60877
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 64 595167 687205 64262 87327
>>> > > > > > > > > > > > > > > > > > > > 20772 <tel:87327%20%C2%A0%2020772>
>>> > > > > > > > > > > > > > > > > > > > 456430 19960 477064 23190 540220
>>> > > > > > > > > > > > > > > > > > > > 563096 86812 92565
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 128 833585 933403 121926
>>> > > > > > > > > > > > > > > > > > > > 118621 37700 690020 37575 733254
>>> > > > > > > > > > > > > > > > > > > > 567449 712337 734006 92011 104934
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 256 799003 949499 143688
>>> > > > > > > > > > > > > > > > > > > > 125659 40871
>>> > > > > > > > > > > > > > > > > > > > <tel:125659%20%C2%A0%2040871>
>>> > > > > > > > > > > > > > > > > > > > 892757 37977 880494 458281 836263
>>> > > > > > > > > > > > > > > > > > > > 901375 131332 110237
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 512 979936 1040724 120896
>>> > > > > > > > > > > > > > > > > > > > 138013 54381 859783 48721 780491
>>> > > > > > > > > > > > > > > > > > > > 279203 1068824 1087085 97886 98078
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 1024 901754 987938 53352
>>> > > > > > > > > > > > > > > > > > > > 53043 72727
>>> > > > > > > > > > > > > > > > > > > > <tel:53043%20%C2%A0%2072727>
>>> > > > > > > > > > > > > > > > > > > > 1054522 68269 992275 181253
>>> > > > > > > > > > > > > > > > > > > > 1309480 1524983 121600 95585
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 2048 2048 831890 1021540 4257067
>>> > > > > > > > > > > > > > > > > > > > 3302797 75672 984203 80181 826209
>>> > > > > > > > > > > > > > > > > > > > 94278 966920 1027159 111832 105921
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 4 66195 67316 62171 74785
>>> > > > > > > > > > > > > > > > > > > > 1328 28963 1329 26397 1223 71470
>>> > > > > > > > > > > > > > > > > > > > 69317 55903 84915
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 8 122221 120057 90537 60958
>>> > > > > > > > > > > > > > > > > > > > 2598 47312 2468 59783 2640 128674
>>> > > > > > > > > > > > > > > > > > > > 127872 41285 40422
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 16 238321 239251 29336 32121
>>> > > > > > > > > > > > > > > > > > > > 4153 89262 3986 96930 4608 229970
>>> > > > > > > > > > > > > > > > > > > > 237108 55039 56983
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 32 417110 421356 30974 50000
>>> > > > > > > > > > > > > > > > > > > > 8382 156676 7886 153841 7900
>>> > > > > > > > > > > > > > > > > > > > 359585 367288 26611 25952
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 64 648008 668066 32193 29389
>>> > > > > > > > > > > > > > > > > > > > 14830 273265 14822 282211 19653
>>> > > > > > > > > > > > > > > > > > > > 581898 620798 51281 50218
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 128 779422 848564 55594 60253
>>> > > > > > > > > > > > > > > > > > > > 37108 <tel:60253%20%C2%A0%2037108>
>>> > > > > > > > > > > > > > > > > > > > 451296 35908 491361 37567 738163
>>> > > > > > > > > > > > > > > > > > > > 728059 67681 66440
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 256 865623 886986 71368 63947
>>> > > > > > > > > > > > > > > > > > > > 44255 <tel:63947%20%C2%A0%2044255>
>>> > > > > > > > > > > > > > > > > > > > 645961 42689 719491 736707 819696
>>> > > > > > > > > > > > > > > > > > > > 837641 57059 60347
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 512 852099 889650 68870 73891
>>> > > > > > > > > > > > > > > > > > > > 31185 845224 30259 830153 392334
>>> > > > > > > > > > > > > > > > > > > > 910442 961983 60083 55558
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 1024 710357 867810 29377
>>> > > > > > > > > > > > > > > > > > > > 29522 49954 846640 43665 926298
>>> > > > > > > > > > > > > > > > > > > > 213677 986226 1115445 55130 59205
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 2048 826479 908420 43191
>>> > > > > > > > > > > > > > > > > > > > 42075 59684 904022 58601 855664
>>> > > > > > > > > > > > > > > > > > > > 115105 1418322 1524415 60548 66066
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 4096 4096 793351 855111 3232454
>>> > > > > > > > > > > > > > > > > > > > 3673419 66018 861413 48833 847852
>>> > > > > > > > > > > > > > > > > > > > 45914 852268 842075 42980 48374
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 4 67340 69421 42198 31740 994
>>> > > > > > > > > > > > > > > > > > > > 23251 1166 16813 837 73827 73126
>>> > > > > > > > > > > > > > > > > > > > 25169 29610
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 8 137150 125622 29131 36439
>>> > > > > > > > > > > > > > > > > > > > 2051 44342 1988 48930 2315 134183
>>> > > > > > > > > > > > > > > > > > > > 135367 31080 33573
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 16 237366 220826 24810 26584
>>> > > > > > > > > > > > > > > > > > > > 3576 88004 3769 78717 4289 233751
>>> > > > > > > > > > > > > > > > > > > > 235355 23302 28742
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 32 457447 454404 31594 27750
>>> > > > > > > > > > > > > > > > > > > > 8141 142022 7846 143984 9322
>>> > > > > > > > > > > > > > > > > > > > 353147 396188 34203 33265
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 64 670645 655259 28630 23255
>>> > > > > > > > > > > > > > > > > > > > 16669 237476 16965 244968 15607
>>> > > > > > > > > > > > > > > > > > > > 590365 575320 49998 43305
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 128 658676 760982 44197 47802
>>> > > > > > > > > > > > > > > > > > > > 28693 379523 26614 378328 27184
>>> > > > > > > > > > > > > > > > > > > > 720997 702038 51707 49733
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 256 643370 698683 56233 63165
>>> > > > > > > > > > > > > > > > > > > > 28846 <tel:63165%20%C2%A0%2028846>
>>> > > > > > > > > > > > > > > > > > > > 543952 27745 576739 44014 701007
>>> > > > > > > > > > > > > > > > > > > > 725534 59611 58985
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 512 696884 776793 67258 52705
>>> > > > > > > > > > > > > > > > > > > > 18711 698854 21004 694124 621695
>>> > > > > > > > > > > > > > > > > > > > 784812 773331 43101 47659
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 1024 729664 810451 15470
>>> > > > > > > > > > > > > > > > > > > > 15875 31318 801490 38123 812944
>>> > > > > > > > > > > > > > > > > > > > 301222 804323 832765 54308 53376
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 2048 749217 68757 21914 22667
>>> > > > > > > > > > > > > > > > > > > > 48971 783309 48132 782738 172848
>>> > > > > > > > > > > > > > > > > > > > 907408 929324 51156 50565
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 4096 707677 763960 32063
>>> > > > > > > > > > > > > > > > > > > > 31928 47809
>>> > > > > > > > > > > > > > > > > > > > <tel:31928%20%C2%A0%2047809>
>>> > > > > > > > > > > > > > > > > > > > 751692 49560 786339 93445 1046761
>>> > > > > > > > > > > > > > > > > > > > 1297876 48037 51680
>>> > > > > > > > > > > > > > > > > > > > <tel:48037%20%C2%A0%20%C2%A051680>
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 8192 8192 623817 746288 2815955
>>> > > > > > > > > > > > > > > > > > > > 3137358 48722 741633 35428 753787
>>> > > > > > > > > > > > > > > > > > > > 49626 803683 823800 48977 52895
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 4 72372 73651 34471 30788
>>> > > > > > > > > > > > > > > > > > > > 960 23610 903 22316 891 71445
>>> > > > > > > > > > > > > > > > > > > > 71138 56451 55129
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 8 137920 141704 50830 33857
>>> > > > > > > > > > > > > > > > > > > > 1935 41934 2275 35588 3608 130757
>>> > > > > > > > > > > > > > > > > > > > 137801 51621 48525
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 16 245369 242460 41808 29770
>>> > > > > > > > > > > > > > > > > > > > 3605 75682 4355 75315 4767 241100
>>> > > > > > > > > > > > > > > > > > > > 239693 53263 30785
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 32 448877 433956 31846 35010
>>> > > > > > > > > > > > > > > > > > > > 7973 118181 8819 112703 8177
>>> > > > > > > > > > > > > > > > > > > > 381734 391651 57749 63417
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 64 710831 700712 66792 68864
>>> > > > > > > > > > > > > > > > > > > > 20176 209806 19034 207852
>>> > > > > > > > > > > > > > > > > > > > <tel:19034%20%C2%A0%20207852>
>>> > > > > > > > > > > > > > > > > > > > 21255 589503 601379 104567 105162
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 128 836901 860867 104226
>>> > > > > > > > > > > > > > > > > > > > 100373 40899 358865 40946 360562
>>> > > > > > > > > > > > > > > > > > > > 39415 675968 691538 96086 105695
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 256 798081 828146 107103
>>> > > > > > > > > > > > > > > > > > > > 120433 39084
>>> > > > > > > > > > > > > > > > > > > > <tel:120433%20%C2%A0%2039084>
>>> > > > > > > > > > > > > > > > > > > > 595325 39050 593110 56925 763466
>>> > > > > > > > > > > > > > > > > > > > 797859 109645 113414
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 512 810851 843931 113564
>>> > > > > > > > > > > > > > > > > > > > 106202 35111 714831 46244 745947
>>> > > > > > > > > > > > > > > > > > > > 53636 802902 760172 110492 100879
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 1024 726399 820219 22106
>>> > > > > > > > > > > > > > > > > > > > 22987 53087
>>> > > > > > > > > > > > > > > > > > > > <tel:22987%20%C2%A0%2053087>
>>> > > > > > > > > > > > > > > > > > > > 749053 54781 777705 1075341 772686
>>> > > > > > > > > > > > > > > > > > > > 809723 100349 96619
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 2048 807772 856458 23920
>>> > > > > > > > > > > > > > > > > > > > 23617 66320 829576 72105 740848
>>> > > > > > > > > > > > > > > > > > > > 656379 864539 835446 93499 101714
>>> > > > > > > > > > > > > > > > > > > >
>>> > > > > > > > > > > > > > > > > > > > 16384 4096 797470 840596 27270
>>> > > > > > > > > > > > > > > > > > > > <tel:28132%20%C2%A0%2088784>
>>> > > ...
>>> > >
>>> > > [Message clipped]
>>> >
>>> >
>>>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130311/c337b3f4/attachment.html>
More information about the Gluster-users
mailing list