[Bugs] [Bug 1438706] Sharding: Fix a performance bug
bugzilla at redhat.com
bugzilla at redhat.com
Fri Jun 2 10:25:02 UTC 2017
https://bugzilla.redhat.com/show_bug.cgi?id=1438706
RamaKasturi <knarra at redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Flags| |needinfo?(kdhananj at redhat.c
| |om)
--- Comment #7 from RamaKasturi <knarra at redhat.com> ---
Had HC installed with glusterfs 3.2 bits and observed that lookups sent on the
brick were greater than the one sent on fuse mount. Used fio tools and ran 100
random read workload.
Lookup on fuse mount : 741
Lookup sent over the brick : 15540
Updated 3.2 bits to 3.3 and ran 100 read workload on the vms. I have run the
workload thrice and i have got similar results twice but the first time lookups
sent on the brick were greater.
First run:
=====================
=== Interval 0 stats ===
Duration : 1021 secs
BytesRead : 149981696
BytesWritten : 49664
Block Size : 512B+ 1024B+ 2048B+
Read Count : 29 0 2
Write Count : 59 3 0
Block Size : 4096B+ 8192B+ 16384B+
Read Count : 110 12007 2
Write Count : 0 0 1
Block Size : 32768B+ 65536B+ 131072B+
Read Count : 2 11 384
Write Count : 0 0 0
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 11987 569.88 us 190.00 us 31860.00 us
OPEN 31 52.48 us 27.00 us 104.00 us
READ 12103 12023.84 us 341.00 us 209385.00 us
WRITE 15 7500.80 us 1334.00 us 78435.00 us
STATFS 43 535.09 us 341.00 us 4031.00 us
FLUSH 48 31.56 us 12.00 us 68.00 us
FSYNC 18 912.50 us 423.00 us 1979.00 us
REMOVEXATTR 7 961.29 us 894.00 us 1103.00 us
OPENDIR 31 618.26 us 492.00 us 1432.00 us
FSTAT 42 1106.86 us 256.00 us 29577.00 us
LOOKUP 269 2146.18 us 482.00 us 87316.00 us
SETATTR 7 856.14 us 749.00 us 1078.00 us
READDIRP 62 1465.13 us 16.00 us 43345.00 us
RELEASE 186 0 us 0 us 0 us
RELEASEDIR 189 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
=== Interval 0 stats ===
Duration : 1022 secs
BytesRead : 842203648
BytesWritten : 5270016
Block Size : 512B+ 1024B+ 2048B+
Read Count : 321 0 3
Write Count : 63 3 0
Block Size : 4096B+ 8192B+ 16384B+
Read Count : 409 12008 10
Write Count : 3 0 1
Block Size : 32768B+ 65536B+ 131072B+
Read Count : 6 507 5250
Write Count : 1 7 34
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 12012 688.10 us 191.00 us 29323.00 us
OPEN 177 46.47 us 22.00 us 92.00 us
READ 13041 10628.98 us 315.00 us 279819.00 us
WRITE 14 3515.00 us 1525.00 us 22023.00 us
STATFS 41 467.46 us 297.00 us 782.00 us
FLUSH 280 32.73 us 12.00 us 72.00 us
FSYNC 18 894.11 us 447.00 us 1864.00 us
REMOVEXATTR 7 905.86 us 826.00 us 994.00 us
OPENDIR 33 789.85 us 430.00 us 5189.00 us
FSTAT 59 521.12 us 224.00 us 8359.00 us
LOOKUP 542 3737.25 us 487.00 us 59769.00 us
SETATTR 7 823.14 us 649.00 us 998.00 us
READDIRP 66 857.58 us 15.00 us 36928.00 us
RELEASE 1051 0 us 0 us 0 us
RELEASEDIR 204 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
=== Interval 0 stats ===
Duration : 1022 secs
BytesRead : 151019520
BytesWritten : 2146816
Block Size : 512B+ 1024B+ 2048B+
Read Count : 76 0 1
Write Count : 59 3 0
Block Size : 4096B+ 8192B+ 16384B+
Read Count : 249 12013 5
Write Count : 3 0 1
Block Size : 32768B+ 65536B+ 131072B+
Read Count : 3 13 384
Write Count : 1 4 12
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 11985 678.82 us 198.00 us 24093.00 us
OPEN 29 45.83 us 28.00 us 69.00 us
READ 12099 10853.84 us 330.00 us 218767.00 us
WRITE 15 5502.40 us 1318.00 us 53345.00 us
STATFS 42 444.74 us 349.00 us 537.00 us
FLUSH 46 33.83 us 14.00 us 61.00 us
FSYNC 18 750.39 us 422.00 us 1604.00 us
REMOVEXATTR 6 917.33 us 854.00 us 978.00 us
OPENDIR 30 673.97 us 461.00 us 4076.00 us
FSTAT 56 435.66 us 296.00 us 2610.00 us
LOOKUP 265 1870.12 us 473.00 us 25081.00 us
SETATTR 6 802.50 us 741.00 us 883.00 us
READDIRP 60 1277.52 us 18.00 us 35013.00 us
RELEASE 184 0 us 0 us 0 us
RELEASEDIR 191 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
vol_profile_output:
===========================
Interval 0 Stats:
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 19 0 0
No. of Writes: 0 30 9
Block Size: 8192b+ 16384b+ 65536b+
No. of Reads: 12007 0 6
No. of Writes: 0 3 0
Block Size: 131072b+
No. of Reads: 64
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 93 RELEASEDIR
0.00 108.00 us 108.00 us 108.00 us 1 OPEN
0.00 37.50 us 22.00 us 73.00 us 4 READDIR
0.00 57.42 us 28.00 us 104.00 us 84 FINODELK
0.00 73.95 us 2.00 us 117.00 us 93 OPENDIR
0.01 66.79 us 26.00 us 118.00 us 124 STATFS
0.01 309.07 us 62.00 us 824.00 us 54 FSYNC
0.02 763.58 us 35.00 us 29177.00 us 43 FSTAT
0.04 614.07 us 117.00 us 16973.00 us 84 FXATTROP
0.04 1365.17 us 162.00 us 24131.00 us 42 WRITE
0.06 1575.48 us 36.00 us 43256.00 us 52 READDIRP
0.10 165.39 us 22.00 us 11109.00 us 830 SETATTR
0.13 231.57 us 42.00 us 36982.00 us 811 SETXATTR
0.14 248.70 us 41.00 us 21395.00 us 830 REMOVEXATTR
0.27 328.41 us 22.00 us 37087.00 us 1167 GETXATTR
0.37 456.10 us 113.00 us 30304.00 us 1163 XATTROP
0.79 94.76 us 35.00 us 31630.00 us 11987 STAT
1.01 604.74 us 18.00 us 64453.00 us 2411 INODELK
3.10 1000.71 us 24.00 us 63019.00 us 4462 LOOKUP
93.92 11193.88 us 142.00 us 209114.00 us 12096 READ
Duration: 173 seconds
Data Read: 107151813 bytes
Data Written: 73728 bytes
Lookup sent on fuse mount : 1076
Lookup sent over brick : 4462
Second Run:
====================
=== Interval 1 stats ===
Duration : 404 secs
BytesRead : 119500800
BytesWritten : 10240
Block Size : 512B+ 4096B+ 8192B+
Read Count : 0 41 12007
Write Count : 20 0 0
Block Size : 131072B+
Read Count : 160
Write Count : 0
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 11988 554.58 us 204.00 us 1794.00 us
OPEN 31 46.52 us 28.00 us 75.00 us
READ 12107 10435.64 us 344.00 us 415124.00 us
WRITE 10 3082.40 us 2172.00 us 7542.00 us
STATFS 44 509.98 us 380.00 us 1104.00 us
FLUSH 50 28.78 us 11.00 us 60.00 us
REMOVEXATTR 6 2358.50 us 887.00 us 9236.00 us
OPENDIR 33 1843.15 us 508.00 us 14872.00 us
FSTAT 46 421.09 us 233.00 us 1191.00 us
LOOKUP 277 2001.56 us 514.00 us 30868.00 us
SETATTR 6 912.17 us 837.00 us 953.00 us
READDIRP 66 766.29 us 14.00 us 29581.00 us
RELEASE 68 0 us 0 us 0 us
RELEASEDIR 71 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
=== Interval 1 stats ===
Duration : 404 secs
BytesRead : 395698176
BytesWritten : 10240
Block Size : 512B+ 4096B+ 8192B+
Read Count : 72 41 12002
Write Count : 20 0 0
Block Size : 65536B+ 131072B+
Read Count : 196 2108
Write Count : 0 0
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 12014 717.24 us 202.00 us 36196.00 us
OPEN 188 48.29 us 23.00 us 109.00 us
READ 13119 10006.90 us 275.00 us 269229.00 us
WRITE 9 2375.11 us 2191.00 us 2555.00 us
STATFS 44 1183.20 us 376.00 us 19794.00 us
FLUSH 298 33.81 us 11.00 us 76.00 us
REMOVEXATTR 6 991.00 us 864.00 us 1263.00 us
OPENDIR 31 634.19 us 481.00 us 1094.00 us
FSTAT 60 406.83 us 200.00 us 770.00 us
LOOKUP 583 3492.63 us 504.00 us 32151.00 us
SETATTR 6 809.17 us 754.00 us 917.00 us
READDIRP 62 311.15 us 16.00 us 1003.00 us
RELEASE 408 0 us 0 us 0 us
RELEASEDIR 69 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
=== Interval 1 stats ===
Duration : 404 secs
BytesRead : 119484416
BytesWritten : 10240
Block Size : 512B+ 4096B+ 8192B+
Read Count : 0 41 12005
Write Count : 20 0 0
Block Size : 131072B+
Read Count : 160
Write Count : 0
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 11979 686.02 us 195.00 us 33833.00 us
OPEN 31 38.81 us 25.00 us 70.00 us
READ 12105 10773.69 us 342.00 us 213621.00 us
WRITE 10 2432.90 us 2245.00 us 2622.00 us
STATFS 44 426.73 us 335.00 us 674.00 us
FLUSH 50 31.84 us 12.00 us 60.00 us
REMOVEXATTR 6 857.00 us 803.00 us 932.00 us
OPENDIR 32 573.72 us 443.00 us 1296.00 us
FSTAT 64 484.00 us 276.00 us 3078.00 us
LOOKUP 269 1648.59 us 488.00 us 33870.00 us
SETATTR 6 750.33 us 703.00 us 840.00 us
READDIRP 64 1220.69 us 15.00 us 59162.00 us
RELEASE 68 0 us 0 us 0 us
RELEASEDIR 71 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
volume profile:
=============================
Interval 0 Stats:
Block Size: 256b+ 512b+ 8192b+
No. of Reads: 18 0 12007
No. of Writes: 0 24 0
Block Size: 131072b+
No. of Reads: 64
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 86 RELEASEDIR
0.00 377.33 us 253.00 us 724.00 us 6 READDIR
0.00 58.69 us 29.00 us 93.00 us 48 FINODELK
0.00 74.67 us 38.00 us 144.00 us 40 FSTAT
0.01 73.49 us 3.00 us 143.00 us 86 OPENDIR
0.01 67.92 us 33.00 us 118.00 us 116 STATFS
0.01 168.48 us 103.00 us 234.00 us 48 FXATTROP
0.01 107.80 us 69.00 us 153.00 us 164 REMOVEXATTR
0.02 132.43 us 89.00 us 201.00 us 148 SETXATTR
0.02 122.78 us 31.00 us 193.00 us 164 SETATTR
0.02 1087.46 us 769.00 us 5997.00 us 24 WRITE
0.02 65.09 us 24.00 us 113.00 us 404 INODELK
0.03 178.39 us 24.00 us 457.00 us 175 GETXATTR
0.03 215.85 us 160.00 us 521.00 us 170 XATTROP
0.03 760.12 us 37.00 us 29202.00 us 51 READDIRP
0.21 188.61 us 36.00 us 15933.00 us 1412 LOOKUP
0.76 78.78 us 29.00 us 723.00 us 11987 STAT
98.82 10132.91 us 135.00 us 414844.00 us 12089 READ
Duration: 167 seconds
Data Read: 106758142 bytes
Data Written: 12288 bytes
Lookup sent over fuse : 1129
Lookup sent over brick : 1412
Third run:
==============================
=== Interval 2 stats ===
Duration : 892 secs
BytesRead : 143814656
BytesWritten : 22016
Block Size : 512B+ 4096B+ 8192B+
Read Count : 0 91 12006
Write Count : 43 0 0
Block Size : 131072B+
Read Count : 344
Write Count : 0
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 11981 557.32 us 209.00 us 1811.00 us
OPEN 34 43.24 us 22.00 us 71.00 us
READ 12099 11715.72 us 379.00 us 218366.00 us
WRITE 9 2543.67 us 2284.00 us 2803.00 us
STATFS 45 497.62 us 357.00 us 713.00 us
FLUSH 54 32.30 us 13.00 us 66.00 us
REMOVEXATTR 7 1033.14 us 822.00 us 1228.00 us
OPENDIR 33 616.15 us 513.00 us 741.00 us
FSTAT 61 416.98 us 230.00 us 1320.00 us
LOOKUP 278 1807.43 us 532.00 us 12786.00 us
SETATTR 7 969.14 us 771.00 us 1368.00 us
READDIRP 66 303.11 us 15.00 us 820.00 us
RELEASE 152 0 us 0 us 0 us
RELEASEDIR 155 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
=== Interval 2 stats ===
Duration : 892 secs
BytesRead : 752958464
BytesWritten : 22016
Block Size : 512B+ 4096B+ 8192B+
Read Count : 158 90 12006
Write Count : 43 0 0
Block Size : 65536B+ 131072B+
Read Count : 433 4639
Write Count : 0 0
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 12017 550.86 us 201.00 us 12317.00 us
OPEN 190 45.58 us 22.00 us 129.00 us
READ 13130 10153.46 us 270.00 us 260762.00 us
WRITE 9 3066.78 us 2346.00 us 7145.00 us
STATFS 45 535.87 us 354.00 us 2304.00 us
FLUSH 302 34.14 us 11.00 us 71.00 us
REMOVEXATTR 7 996.14 us 848.00 us 1119.00 us
OPENDIR 36 685.11 us 488.00 us 1982.00 us
FSTAT 63 385.44 us 230.00 us 635.00 us
LOOKUP 578 3320.98 us 489.00 us 36146.00 us
SETATTR 7 878.71 us 615.00 us 1027.00 us
READDIRP 72 297.44 us 17.00 us 895.00 us
RELEASE 902 0 us 0 us 0 us
RELEASEDIR 160 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
=== Interval 2 stats ===
Duration : 892 secs
BytesRead : 143806464
BytesWritten : 22016
Block Size : 512B+ 4096B+ 8192B+
Read Count : 0 91 12005
Write Count : 43 0 0
Block Size : 131072B+
Read Count : 344
Write Count : 0
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 11977 714.93 us 193.00 us 63057.00 us
OPEN 32 39.69 us 22.00 us 71.00 us
READ 12098 10851.17 us 352.00 us 268565.00 us
WRITE 9 2406.11 us 2243.00 us 2701.00 us
STATFS 44 435.57 us 303.00 us 1176.00 us
FLUSH 52 29.65 us 12.00 us 49.00 us
REMOVEXATTR 6 843.67 us 762.00 us 912.00 us
OPENDIR 31 865.48 us 458.00 us 9475.00 us
FSTAT 61 409.67 us 257.00 us 627.00 us
LOOKUP 282 1754.85 us 460.00 us 25927.00 us
SETATTR 6 749.17 us 638.00 us 847.00 us
READDIRP 62 304.05 us 14.00 us 776.00 us
RELEASE 154 0 us 0 us 0 us
RELEASEDIR 155 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
volume profile:
=======================
Interval 0 Stats:
Block Size: 256b+ 512b+ 8192b+
No. of Reads: 18 0 12006
No. of Writes: 0 24 0
Block Size: 131072b+
No. of Reads: 64
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 92 RELEASEDIR
0.00 37.75 us 21.00 us 76.00 us 4 READDIR
0.00 62.33 us 35.00 us 93.00 us 48 FINODELK
0.00 79.50 us 33.00 us 119.00 us 58 FSTAT
0.00 74.51 us 3.00 us 133.00 us 92 OPENDIR
0.01 64.51 us 25.00 us 410.00 us 121 STATFS
0.01 176.33 us 132.00 us 233.00 us 48 FXATTROP
0.01 169.34 us 40.00 us 416.00 us 56 READDIRP
0.01 103.52 us 69.00 us 147.00 us 173 REMOVEXATTR
0.01 124.14 us 83.00 us 185.00 us 155 SETXATTR
0.02 121.36 us 31.00 us 189.00 us 173 SETATTR
0.02 899.67 us 747.00 us 1118.00 us 24 WRITE
0.02 68.47 us 24.00 us 2359.00 us 435 INODELK
0.02 175.49 us 21.00 us 270.00 us 185 GETXATTR
0.03 210.42 us 154.00 us 281.00 us 181 XATTROP
0.17 165.59 us 24.00 us 580.00 us 1451 LOOKUP
0.67 78.49 us 38.00 us 442.00 us 11981 STAT
98.99 11416.08 us 130.00 us 218011.00 us 12088 READ
Duration: 169 seconds
Data Read: 106749950 bytes
Data Written: 12288 bytes
Lookup sent over fuse mount : 1138
Lookup sent over brick : 1451
Any idea why this was more first time? I ran same workload which is 100% random
read.
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=23H1aC3Lte&a=cc_unsubscribe
More information about the Bugs
mailing list