[Bugs] [Bug 1438706] Sharding: Fix a performance bug
bugzilla at redhat.com
bugzilla at redhat.com
Mon Jun 5 05:29:57 UTC 2017
https://bugzilla.redhat.com/show_bug.cgi?id=1438706
Krutika Dhananjay <kdhananj at redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Flags|needinfo?(kdhananj at redhat.c |
|om) |
--- Comment #8 from Krutika Dhananjay <kdhananj at redhat.com> ---
(In reply to RamaKasturi from comment #7)
> Had HC installed with glusterfs 3.2 bits and observed that lookups sent on
> the brick were greater than the one sent on fuse mount. Used fio tools and
> ran 100 random read workload.
>
> Lookup on fuse mount : 741
> Lookup sent over the brick : 15540
>
> Updated 3.2 bits to 3.3 and ran 100 read workload on the vms. I have run the
> workload thrice and i have got similar results twice but the first time
> lookups sent on the brick were greater.
>
> First run:
> =====================
>
> === Interval 0 stats ===
> Duration : 1021 secs
> BytesRead : 149981696
> BytesWritten : 49664
>
> Block Size : 512B+ 1024B+ 2048B+
> Read Count : 29 0 2
> Write Count : 59 3 0
>
> Block Size : 4096B+ 8192B+ 16384B+
> Read Count : 110 12007 2
> Write Count : 0 0 1
>
> Block Size : 32768B+ 65536B+ 131072B+
> Read Count : 2 11 384
> Write Count : 0 0 0
>
> Fop Call Count Avg-Latency Min-Latency Max-Latency
> --- ---------- ----------- ----------- -----------
> STAT 11987 569.88 us 190.00 us 31860.00 us
> OPEN 31 52.48 us 27.00 us 104.00 us
> READ 12103 12023.84 us 341.00 us 209385.00 us
> WRITE 15 7500.80 us 1334.00 us 78435.00 us
> STATFS 43 535.09 us 341.00 us 4031.00 us
> FLUSH 48 31.56 us 12.00 us 68.00 us
> FSYNC 18 912.50 us 423.00 us 1979.00 us
> REMOVEXATTR 7 961.29 us 894.00 us 1103.00 us
> OPENDIR 31 618.26 us 492.00 us 1432.00 us
> FSTAT 42 1106.86 us 256.00 us 29577.00 us
> LOOKUP 269 2146.18 us 482.00 us 87316.00 us
> SETATTR 7 856.14 us 749.00 us 1078.00 us
> READDIRP 62 1465.13 us 16.00 us 43345.00 us
> RELEASE 186 0 us 0 us 0 us
> RELEASEDIR 189 0 us 0 us 0 us
> ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
>
> === Interval 0 stats ===
> Duration : 1022 secs
> BytesRead : 842203648
> BytesWritten : 5270016
>
> Block Size : 512B+ 1024B+ 2048B+
> Read Count : 321 0 3
> Write Count : 63 3 0
>
> Block Size : 4096B+ 8192B+ 16384B+
> Read Count : 409 12008 10
> Write Count : 3 0 1
>
> Block Size : 32768B+ 65536B+ 131072B+
> Read Count : 6 507 5250
> Write Count : 1 7 34
>
> Fop Call Count Avg-Latency Min-Latency Max-Latency
> --- ---------- ----------- ----------- -----------
> STAT 12012 688.10 us 191.00 us 29323.00 us
> OPEN 177 46.47 us 22.00 us 92.00 us
> READ 13041 10628.98 us 315.00 us 279819.00 us
> WRITE 14 3515.00 us 1525.00 us 22023.00 us
> STATFS 41 467.46 us 297.00 us 782.00 us
> FLUSH 280 32.73 us 12.00 us 72.00 us
> FSYNC 18 894.11 us 447.00 us 1864.00 us
> REMOVEXATTR 7 905.86 us 826.00 us 994.00 us
> OPENDIR 33 789.85 us 430.00 us 5189.00 us
> FSTAT 59 521.12 us 224.00 us 8359.00 us
> LOOKUP 542 3737.25 us 487.00 us 59769.00 us
> SETATTR 7 823.14 us 649.00 us 998.00 us
> READDIRP 66 857.58 us 15.00 us 36928.00 us
> RELEASE 1051 0 us 0 us 0 us
> RELEASEDIR 204 0 us 0 us 0 us
> ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
>
> === Interval 0 stats ===
> Duration : 1022 secs
> BytesRead : 151019520
> BytesWritten : 2146816
>
> Block Size : 512B+ 1024B+ 2048B+
> Read Count : 76 0 1
> Write Count : 59 3 0
>
> Block Size : 4096B+ 8192B+ 16384B+
> Read Count : 249 12013 5
> Write Count : 3 0 1
>
> Block Size : 32768B+ 65536B+ 131072B+
> Read Count : 3 13 384
> Write Count : 1 4 12
>
> Fop Call Count Avg-Latency Min-Latency Max-Latency
> --- ---------- ----------- ----------- -----------
> STAT 11985 678.82 us 198.00 us 24093.00 us
> OPEN 29 45.83 us 28.00 us 69.00 us
> READ 12099 10853.84 us 330.00 us 218767.00 us
> WRITE 15 5502.40 us 1318.00 us 53345.00 us
> STATFS 42 444.74 us 349.00 us 537.00 us
> FLUSH 46 33.83 us 14.00 us 61.00 us
> FSYNC 18 750.39 us 422.00 us 1604.00 us
> REMOVEXATTR 6 917.33 us 854.00 us 978.00 us
> OPENDIR 30 673.97 us 461.00 us 4076.00 us
> FSTAT 56 435.66 us 296.00 us 2610.00 us
> LOOKUP 265 1870.12 us 473.00 us 25081.00 us
> SETATTR 6 802.50 us 741.00 us 883.00 us
> READDIRP 60 1277.52 us 18.00 us 35013.00 us
> RELEASE 184 0 us 0 us 0 us
> RELEASEDIR 191 0 us 0 us 0 us
> ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
> vol_profile_output:
> ===========================
> Interval 0 Stats:
> Block Size: 256b+ 512b+
> 1024b+
> No. of Reads: 19 0
> 0
> No. of Writes: 0 30
> 9
>
> Block Size: 8192b+ 16384b+
> 65536b+
> No. of Reads: 12007 0
> 6
> No. of Writes: 0 3
> 0
>
> Block Size: 131072b+
> No. of Reads: 64
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 93
> RELEASEDIR
> 0.00 108.00 us 108.00 us 108.00 us 1
> OPEN
> 0.00 37.50 us 22.00 us 73.00 us 4
> READDIR
> 0.00 57.42 us 28.00 us 104.00 us 84
> FINODELK
> 0.00 73.95 us 2.00 us 117.00 us 93
> OPENDIR
> 0.01 66.79 us 26.00 us 118.00 us 124
> STATFS
> 0.01 309.07 us 62.00 us 824.00 us 54
> FSYNC
> 0.02 763.58 us 35.00 us 29177.00 us 43
> FSTAT
> 0.04 614.07 us 117.00 us 16973.00 us 84
> FXATTROP
> 0.04 1365.17 us 162.00 us 24131.00 us 42
> WRITE
> 0.06 1575.48 us 36.00 us 43256.00 us 52
> READDIRP
> 0.10 165.39 us 22.00 us 11109.00 us 830
> SETATTR
> 0.13 231.57 us 42.00 us 36982.00 us 811
> SETXATTR
> 0.14 248.70 us 41.00 us 21395.00 us 830
> REMOVEXATTR
> 0.27 328.41 us 22.00 us 37087.00 us 1167
> GETXATTR
> 0.37 456.10 us 113.00 us 30304.00 us 1163
> XATTROP
> 0.79 94.76 us 35.00 us 31630.00 us 11987
> STAT
> 1.01 604.74 us 18.00 us 64453.00 us 2411
> INODELK
> 3.10 1000.71 us 24.00 us 63019.00 us 4462
> LOOKUP
> 93.92 11193.88 us 142.00 us 209114.00 us 12096
> READ
>
> Duration: 173 seconds
> Data Read: 107151813 bytes
> Data Written: 73728 bytes
>
> Lookup sent on fuse mount : 1076
> Lookup sent over brick : 4462
>
> Second Run:
> ====================
> === Interval 1 stats ===
> Duration : 404 secs
> BytesRead : 119500800
> BytesWritten : 10240
>
> Block Size : 512B+ 4096B+ 8192B+
> Read Count : 0 41 12007
> Write Count : 20 0 0
>
> Block Size : 131072B+
> Read Count : 160
> Write Count : 0
>
> Fop Call Count Avg-Latency Min-Latency Max-Latency
> --- ---------- ----------- ----------- -----------
> STAT 11988 554.58 us 204.00 us 1794.00 us
> OPEN 31 46.52 us 28.00 us 75.00 us
> READ 12107 10435.64 us 344.00 us 415124.00 us
> WRITE 10 3082.40 us 2172.00 us 7542.00 us
> STATFS 44 509.98 us 380.00 us 1104.00 us
> FLUSH 50 28.78 us 11.00 us 60.00 us
> REMOVEXATTR 6 2358.50 us 887.00 us 9236.00 us
> OPENDIR 33 1843.15 us 508.00 us 14872.00 us
> FSTAT 46 421.09 us 233.00 us 1191.00 us
> LOOKUP 277 2001.56 us 514.00 us 30868.00 us
> SETATTR 6 912.17 us 837.00 us 953.00 us
> READDIRP 66 766.29 us 14.00 us 29581.00 us
> RELEASE 68 0 us 0 us 0 us
> RELEASEDIR 71 0 us 0 us 0 us
> ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
>
> === Interval 1 stats ===
> Duration : 404 secs
> BytesRead : 395698176
> BytesWritten : 10240
>
> Block Size : 512B+ 4096B+ 8192B+
> Read Count : 72 41 12002
> Write Count : 20 0 0
>
> Block Size : 65536B+ 131072B+
> Read Count : 196 2108
> Write Count : 0 0
>
> Fop Call Count Avg-Latency Min-Latency Max-Latency
> --- ---------- ----------- ----------- -----------
> STAT 12014 717.24 us 202.00 us 36196.00 us
> OPEN 188 48.29 us 23.00 us 109.00 us
> READ 13119 10006.90 us 275.00 us 269229.00 us
> WRITE 9 2375.11 us 2191.00 us 2555.00 us
> STATFS 44 1183.20 us 376.00 us 19794.00 us
> FLUSH 298 33.81 us 11.00 us 76.00 us
> REMOVEXATTR 6 991.00 us 864.00 us 1263.00 us
> OPENDIR 31 634.19 us 481.00 us 1094.00 us
> FSTAT 60 406.83 us 200.00 us 770.00 us
> LOOKUP 583 3492.63 us 504.00 us 32151.00 us
> SETATTR 6 809.17 us 754.00 us 917.00 us
> READDIRP 62 311.15 us 16.00 us 1003.00 us
> RELEASE 408 0 us 0 us 0 us
> RELEASEDIR 69 0 us 0 us 0 us
> ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
>
> === Interval 1 stats ===
> Duration : 404 secs
> BytesRead : 119484416
> BytesWritten : 10240
>
> Block Size : 512B+ 4096B+ 8192B+
> Read Count : 0 41 12005
> Write Count : 20 0 0
>
> Block Size : 131072B+
> Read Count : 160
> Write Count : 0
>
> Fop Call Count Avg-Latency Min-Latency Max-Latency
> --- ---------- ----------- ----------- -----------
> STAT 11979 686.02 us 195.00 us 33833.00 us
> OPEN 31 38.81 us 25.00 us 70.00 us
> READ 12105 10773.69 us 342.00 us 213621.00 us
> WRITE 10 2432.90 us 2245.00 us 2622.00 us
> STATFS 44 426.73 us 335.00 us 674.00 us
> FLUSH 50 31.84 us 12.00 us 60.00 us
> REMOVEXATTR 6 857.00 us 803.00 us 932.00 us
> OPENDIR 32 573.72 us 443.00 us 1296.00 us
> FSTAT 64 484.00 us 276.00 us 3078.00 us
> LOOKUP 269 1648.59 us 488.00 us 33870.00 us
> SETATTR 6 750.33 us 703.00 us 840.00 us
> READDIRP 64 1220.69 us 15.00 us 59162.00 us
> RELEASE 68 0 us 0 us 0 us
> RELEASEDIR 71 0 us 0 us 0 us
> ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
>
> volume profile:
> =============================
> Interval 0 Stats:
> Block Size: 256b+ 512b+
> 8192b+
> No. of Reads: 18 0
> 12007
> No. of Writes: 0 24
> 0
>
> Block Size: 131072b+
> No. of Reads: 64
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 86
> RELEASEDIR
> 0.00 377.33 us 253.00 us 724.00 us 6
> READDIR
> 0.00 58.69 us 29.00 us 93.00 us 48
> FINODELK
> 0.00 74.67 us 38.00 us 144.00 us 40
> FSTAT
> 0.01 73.49 us 3.00 us 143.00 us 86
> OPENDIR
> 0.01 67.92 us 33.00 us 118.00 us 116
> STATFS
> 0.01 168.48 us 103.00 us 234.00 us 48
> FXATTROP
> 0.01 107.80 us 69.00 us 153.00 us 164
> REMOVEXATTR
> 0.02 132.43 us 89.00 us 201.00 us 148
> SETXATTR
> 0.02 122.78 us 31.00 us 193.00 us 164
> SETATTR
> 0.02 1087.46 us 769.00 us 5997.00 us 24
> WRITE
> 0.02 65.09 us 24.00 us 113.00 us 404
> INODELK
> 0.03 178.39 us 24.00 us 457.00 us 175
> GETXATTR
> 0.03 215.85 us 160.00 us 521.00 us 170
> XATTROP
> 0.03 760.12 us 37.00 us 29202.00 us 51
> READDIRP
> 0.21 188.61 us 36.00 us 15933.00 us 1412
> LOOKUP
> 0.76 78.78 us 29.00 us 723.00 us 11987
> STAT
> 98.82 10132.91 us 135.00 us 414844.00 us 12089
> READ
>
> Duration: 167 seconds
> Data Read: 106758142 bytes
> Data Written: 12288 bytes
>
>
> Lookup sent over fuse : 1129
> Lookup sent over brick : 1412
>
> Third run:
> ==============================
> === Interval 2 stats ===
> Duration : 892 secs
> BytesRead : 143814656
> BytesWritten : 22016
>
> Block Size : 512B+ 4096B+ 8192B+
> Read Count : 0 91 12006
> Write Count : 43 0 0
>
> Block Size : 131072B+
> Read Count : 344
> Write Count : 0
>
> Fop Call Count Avg-Latency Min-Latency Max-Latency
> --- ---------- ----------- ----------- -----------
> STAT 11981 557.32 us 209.00 us 1811.00 us
> OPEN 34 43.24 us 22.00 us 71.00 us
> READ 12099 11715.72 us 379.00 us 218366.00 us
> WRITE 9 2543.67 us 2284.00 us 2803.00 us
> STATFS 45 497.62 us 357.00 us 713.00 us
> FLUSH 54 32.30 us 13.00 us 66.00 us
> REMOVEXATTR 7 1033.14 us 822.00 us 1228.00 us
> OPENDIR 33 616.15 us 513.00 us 741.00 us
> FSTAT 61 416.98 us 230.00 us 1320.00 us
> LOOKUP 278 1807.43 us 532.00 us 12786.00 us
> SETATTR 7 969.14 us 771.00 us 1368.00 us
> READDIRP 66 303.11 us 15.00 us 820.00 us
> RELEASE 152 0 us 0 us 0 us
> RELEASEDIR 155 0 us 0 us 0 us
> ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
>
> === Interval 2 stats ===
> Duration : 892 secs
> BytesRead : 752958464
> BytesWritten : 22016
>
> Block Size : 512B+ 4096B+ 8192B+
> Read Count : 158 90 12006
> Write Count : 43 0 0
>
> Block Size : 65536B+ 131072B+
> Read Count : 433 4639
> Write Count : 0 0
>
> Fop Call Count Avg-Latency Min-Latency Max-Latency
> --- ---------- ----------- ----------- -----------
> STAT 12017 550.86 us 201.00 us 12317.00 us
> OPEN 190 45.58 us 22.00 us 129.00 us
> READ 13130 10153.46 us 270.00 us 260762.00 us
> WRITE 9 3066.78 us 2346.00 us 7145.00 us
> STATFS 45 535.87 us 354.00 us 2304.00 us
> FLUSH 302 34.14 us 11.00 us 71.00 us
> REMOVEXATTR 7 996.14 us 848.00 us 1119.00 us
> OPENDIR 36 685.11 us 488.00 us 1982.00 us
> FSTAT 63 385.44 us 230.00 us 635.00 us
> LOOKUP 578 3320.98 us 489.00 us 36146.00 us
> SETATTR 7 878.71 us 615.00 us 1027.00 us
> READDIRP 72 297.44 us 17.00 us 895.00 us
> RELEASE 902 0 us 0 us 0 us
> RELEASEDIR 160 0 us 0 us 0 us
> ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
>
> === Interval 2 stats ===
> Duration : 892 secs
> BytesRead : 143806464
> BytesWritten : 22016
>
> Block Size : 512B+ 4096B+ 8192B+
> Read Count : 0 91 12005
> Write Count : 43 0 0
>
> Block Size : 131072B+
> Read Count : 344
> Write Count : 0
>
> Fop Call Count Avg-Latency Min-Latency Max-Latency
> --- ---------- ----------- ----------- -----------
> STAT 11977 714.93 us 193.00 us 63057.00 us
> OPEN 32 39.69 us 22.00 us 71.00 us
> READ 12098 10851.17 us 352.00 us 268565.00 us
> WRITE 9 2406.11 us 2243.00 us 2701.00 us
> STATFS 44 435.57 us 303.00 us 1176.00 us
> FLUSH 52 29.65 us 12.00 us 49.00 us
> REMOVEXATTR 6 843.67 us 762.00 us 912.00 us
> OPENDIR 31 865.48 us 458.00 us 9475.00 us
> FSTAT 61 409.67 us 257.00 us 627.00 us
> LOOKUP 282 1754.85 us 460.00 us 25927.00 us
> SETATTR 6 749.17 us 638.00 us 847.00 us
> READDIRP 62 304.05 us 14.00 us 776.00 us
> RELEASE 154 0 us 0 us 0 us
> RELEASEDIR 155 0 us 0 us 0 us
> ------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
>
> volume profile:
> =======================
> Interval 0 Stats:
> Block Size: 256b+ 512b+
> 8192b+
> No. of Reads: 18 0
> 12006
> No. of Writes: 0 24
> 0
>
> Block Size: 131072b+
> No. of Reads: 64
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 92
> RELEASEDIR
> 0.00 37.75 us 21.00 us 76.00 us 4
> READDIR
> 0.00 62.33 us 35.00 us 93.00 us 48
> FINODELK
> 0.00 79.50 us 33.00 us 119.00 us 58
> FSTAT
> 0.00 74.51 us 3.00 us 133.00 us 92
> OPENDIR
> 0.01 64.51 us 25.00 us 410.00 us 121
> STATFS
> 0.01 176.33 us 132.00 us 233.00 us 48
> FXATTROP
> 0.01 169.34 us 40.00 us 416.00 us 56
> READDIRP
> 0.01 103.52 us 69.00 us 147.00 us 173
> REMOVEXATTR
> 0.01 124.14 us 83.00 us 185.00 us 155
> SETXATTR
> 0.02 121.36 us 31.00 us 189.00 us 173
> SETATTR
> 0.02 899.67 us 747.00 us 1118.00 us 24
> WRITE
> 0.02 68.47 us 24.00 us 2359.00 us 435
> INODELK
> 0.02 175.49 us 21.00 us 270.00 us 185
> GETXATTR
> 0.03 210.42 us 154.00 us 281.00 us 181
> XATTROP
> 0.17 165.59 us 24.00 us 580.00 us 1451
> LOOKUP
> 0.67 78.49 us 38.00 us 442.00 us 11981
> STAT
> 98.99 11416.08 us 130.00 us 218011.00 us 12088
> READ
>
> Duration: 169 seconds
> Data Read: 106749950 bytes
> Data Written: 12288 bytes
>
> Lookup sent over fuse mount : 1138
> Lookup sent over brick : 1451
>
> Any idea why this was more first time? I ran same workload which is 100%
> random read.
Assuming the first run was following a fresh mount of the volume, my guess is
that shard has looked up some shards before reading from them. Earlier it used
to be sending MKNODs before LOOKUP. But ever since we found the add-brick + vm
corruption issue, we had to change the order in this patch -
https://review.gluster.org/17010
Note that this is a one-time lookup. Upon first lookup, shard xl remembers the
shard-to-inode mapping in memory. Subsequent reads/writes in that region won't
require this lookup.
-Krutika
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=zohHCN4ZNb&a=cc_unsubscribe
More information about the Bugs
mailing list