[Gluster-devel] Slow fs
Vijay Bellur
vbellur at redhat.com
Fri Nov 15 05:07:25 UTC 2013
On 11/09/2013 09:36 PM, Emmanuel Dreyfus wrote:
> Hi
>
> On 3.4.1, I had a crash server side because of system limit (heap size). I
> restarted, and now the fs is extremely slow.
>
> Here are profile info. Any hint of where is the issue?
With lookups taking bulk of the time, was there any self heal in progress?
Did the situation recover after a while or has it been the same?
-Vijay
>
> silo# gluster volume profile gfs341 info
> Brick: silo:/export/wd2a
> ------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 0 0 0
> No. of Writes: 119 89 207
>
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 5 5 6
> No. of Writes: 2685 20559 26884
>
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 183 440 333
> No. of Writes: 16666 22414 26828
>
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 162 87 86
> No. of Writes: 47158 39862 46764
>
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 454 454 740
> No. of Writes: 22016 14543 9947
>
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 1897 5631 209278
> No. of Writes: 12471 3710 197
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 279478 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 267232 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 28625 RELEASEDIR
> 0.00 47.67 us 28.00 us 61.00 us 3 STAT
> 0.00 106.33 us 92.00 us 133.00 us 3 TRUNCATE
> 0.00 62.33 us 53.00 us 68.00 us 6 WRITE
> 0.00 97.50 us 65.00 us 124.00 us 4 FSTAT
> 0.00 81.78 us 46.00 us 109.00 us 9 FTRUNCATE
> 0.00 101.38 us 27.00 us 451.00 us 8 STATFS
> 0.00 242.00 us 158.00 us 321.00 us 5 READDIR
> 0.00 1501.00 us 1501.00 us 1501.00 us 1 SETXATTR
> 0.00 531.67 us 49.00 us 1483.00 us 3 OPEN
> 0.03 3865.83 us 1687.00 us 11607.00 us 6 FSYNC
> 0.06 16270.00 us 6346.00 us 36005.00 us 3 RENAME
> 0.18 5054.03 us 2822.00 us 22519.00 us 30 FXATTROP
> 0.23 39790.00 us 21818.00 us 89369.00 us 5 CREATE
> 0.25 6793.66 us 19.00 us 215339.00 us 32 FINODELK
> 0.35 2506.14 us 114.00 us 168464.00 us 120 READ
> 0.43 61041.83 us 3273.00 us 239458.00 us 6 XATTROP
> 3.07 17162.43 us 51.00 us 431101.00 us 153 SETATTR
> 8.95 45045.94 us 30.00 us 2065080.00 us 170 GETXATTR
> 10.82 31385.63 us 17.00 us 2941755.00 us 295 ENTRYLK
> 16.73 14315.53 us 17.00 us 2502697.00 us 1000 INODELK
> 58.89 14729.58 us 86.00 us 3603064.00 us 3420 LOOKUP
>
> Duration: 561493 seconds
> Data Read: 28085570022 bytes
> Data Written: 1503297110 bytes
>
> Interval 3 Stats:
> Block Size: 512b+ 131072b+
> No. of Reads: 0 40
> No. of Writes: 2 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 1 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
> 0.00 61.00 us 61.00 us 61.00 us 1 STAT
> 0.00 63.00 us 63.00 us 63.00 us 1 OPEN
> 0.00 94.00 us 94.00 us 94.00 us 1 TRUNCATE
> 0.00 62.50 us 61.00 us 64.00 us 2 WRITE
> 0.00 68.00 us 44.00 us 92.00 us 2 STATFS
> 0.00 86.33 us 46.00 us 109.00 us 3 FTRUNCATE
> 0.00 30.80 us 23.00 us 41.00 us 10 FINODELK
> 0.00 184.00 us 158.00 us 210.00 us 2 READDIR
> 0.00 97.50 us 65.00 us 124.00 us 4 FSTAT
> 0.03 2557.00 us 1691.00 us 3423.00 us 2 FSYNC
> 0.03 6346.00 us 6346.00 us 6346.00 us 1 RENAME
> 0.03 166.03 us 121.00 us 278.00 us 40 READ
> 0.09 8913.00 us 3273.00 us 14553.00 us 2 XATTROP
> 0.24 4524.00 us 2895.00 us 10112.00 us 10 FXATTROP
> 0.60 57210.50 us 25052.00 us 89369.00 us 2 CREATE
> 2.86 11905.35 us 57.00 us 190800.00 us 46 SETATTR
> 6.60 19771.62 us 396.00 us 608832.00 us 64 GETXATTR
> 9.39 13431.40 us 19.00 us 971690.00 us 134 ENTRYLK
> 11.96 6496.44 us 17.00 us 950230.00 us 353 INODELK
> 68.17 14177.33 us 94.00 us 1380707.00 us 922 LOOKUP
>
> Duration: 28 seconds
> Data Read: 5242880 bytes
> Data Written: 1800 bytes
>
> Brick: debacle:/export/wd1a
> ---------------------------
> Cumulative Stats:
> Block Size: 1b+ 4b+ 8b+
> No. of Reads: 0 3 54
> No. of Writes: 1 0 4
>
> Block Size: 16b+ 32b+ 64b+
> No. of Reads: 1125 1277 837
> No. of Writes: 6 7 18
>
> Block Size: 128b+ 256b+ 512b+
> No. of Reads: 944 1177 1707
> No. of Writes: 31 79 121
>
> Block Size: 1024b+ 2048b+ 4096b+
> No. of Reads: 1436 3085 1828
> No. of Writes: 281 596 317
>
> Block Size: 8192b+ 16384b+ 32768b+
> No. of Reads: 983 3788 461
> No. of Writes: 289 504 4091
>
> Block Size: 65536b+ 131072b+
> No. of Reads: 44771 50582
> No. of Writes: 2242 159
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 288178 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 276611 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 29662 RELEASEDIR
> 0.00 65.50 us 30.00 us 101.00 us 2 STAT
> 0.00 65.00 us 38.00 us 115.00 us 3 OPEN
> 0.00 47.62 us 24.00 us 106.00 us 8 STATFS
> 0.00 53.88 us 32.00 us 72.00 us 8 FTRUNCATE
> 0.00 26.56 us 10.00 us 84.00 us 18 FINODELK
> 0.00 24.23 us 8.00 us 63.00 us 22 ENTRYLK
> 0.00 212.00 us 189.00 us 229.00 us 5 FSYNC
> 0.01 2144.00 us 2144.00 us 2144.00 us 1 READDIR
> 0.01 84.46 us 73.00 us 121.00 us 37 READ
> 0.03 215.22 us 44.00 us 6190.00 us 37 FSTAT
> 0.09 7455.00 us 2237.00 us 13052.00 us 3 RENAME
> 0.23 19000.33 us 62.00 us 56797.00 us 3 TRUNCATE
> 0.47 19886.67 us 1217.00 us 75636.00 us 6 XATTROP
> 0.50 833.23 us 40.00 us 66927.00 us 151 SETATTR
> 0.77 4586.33 us 27.00 us 115632.00 us 42 GETXATTR
> 0.91 14307.44 us 1107.00 us 107253.00 us 16 FXATTROP
> 2.06 103694.20 us 14683.00 us 447120.00 us 5 CREATE
> 12.50 2043.20 us 9.00 us 1030204.00 us 1540 INODELK
> 82.41 3470.41 us 76.00 us 998615.00 us 5976 LOOKUP
>
> Duration: 561508 seconds
> Data Read: 11439890349 bytes
> Data Written: 393336122 bytes
>
> Interval 3 Stats:
> Block Size: 65536b+
> No. of Reads: 11
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 1 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
> 0.00 30.00 us 30.00 us 30.00 us 1 STAT
> 0.00 37.50 us 26.00 us 49.00 us 2 STATFS
> 0.00 115.00 us 115.00 us 115.00 us 1 OPEN
> 0.01 17.12 us 12.00 us 23.00 us 8 ENTRYLK
> 0.01 142.00 us 142.00 us 142.00 us 1 TRUNCATE
> 0.01 56.67 us 39.00 us 69.00 us 3 FTRUNCATE
> 0.01 32.43 us 13.00 us 84.00 us 7 FINODELK
> 0.02 209.00 us 189.00 us 229.00 us 2 FSYNC
> 0.02 48.73 us 48.00 us 51.00 us 11 FSTAT
> 0.04 84.09 us 76.00 us 117.00 us 11 READ
> 0.56 13052.00 us 13052.00 us 13052.00 us 1 RENAME
> 1.59 18707.50 us 14683.00 us 22732.00 us 2 CREATE
> 2.14 172.60 us 12.00 us 23403.00 us 291 INODELK
> 3.28 38566.50 us 1497.00 us 75636.00 us 2 XATTROP
> 4.19 16414.83 us 1176.00 us 85980.00 us 6 FXATTROP
> 5.02 2619.51 us 45.00 us 66927.00 us 45 SETATTR
> 83.10 1549.53 us 80.00 us 171269.00 us 1260 LOOKUP
>
> Duration: 27 seconds
> Data Read: 1146101 bytes
> Data Written: 0 bytes
>
> Brick: hangar:/export/wd3a
> --------------------------
> Cumulative Stats:
> Block Size: 1b+ 4b+ 8b+
> No. of Reads: 0 0 0
> No. of Writes: 1 3 58
>
> Block Size: 16b+ 32b+ 64b+
> No. of Reads: 1 0 0
> No. of Writes: 1130 1284 855
>
> Block Size: 128b+ 256b+ 512b+
> No. of Reads: 0 0 0
> No. of Writes: 975 1256 1827
>
> Block Size: 1024b+ 2048b+ 4096b+
> No. of Reads: 3 3 32
> No. of Writes: 1716 3677 2110
>
> Block Size: 8192b+ 16384b+ 32768b+
> No. of Reads: 25 166 141
> No. of Writes: 1248 1042 4118
>
> Block Size: 65536b+ 131072b+
> No. of Reads: 904 132762
> No. of Writes: 2293 341
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 289743 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 276611 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 29662 RELEASEDIR
> 0.00 127.67 us 103.00 us 150.00 us 3 TRUNCATE
> 0.00 80.00 us 42.00 us 127.00 us 8 FTRUNCATE
> 0.00 276.00 us 78.00 us 800.00 us 4 FSTAT
> 0.00 550.00 us 69.00 us 1512.00 us 3 OPEN
> 0.00 174.13 us 121.00 us 258.00 us 15 READDIR
> 0.00 155.94 us 21.00 us 1508.00 us 18 FINODELK
> 0.03 15207.67 us 6407.00 us 32591.00 us 3 RENAME
> 0.03 9230.33 us 2991.00 us 26321.00 us 6 XATTROP
> 0.07 1779.54 us 150.00 us 33052.00 us 67 READ
> 0.29 65312.50 us 38.00 us 226590.00 us 8 STATFS
> 0.30 106863.80 us 25134.00 us 403476.00 us 5 CREATE
> 0.69 77226.81 us 2748.00 us 773357.00 us 16 FXATTROP
> 0.71 255734.60 us 334.00 us 1032546.00 us 5 FSYNC
> 1.52 18027.92 us 51.00 us 863671.00 us 151 SETATTR
> 3.85 12032.38 us 40.00 us 2023922.00 us 574 GETXATTR
> 11.50 21288.11 us 15.00 us 4791081.00 us 969 ENTRYLK
> 23.47 16990.93 us 18.00 us 3870756.00 us 2479 INODELK
> 57.55 22760.43 us 94.00 us 4753189.00 us 4537 LOOKUP
>
> Duration: 561502 seconds
> Data Read: 17497120878 bytes
> Data Written: 470443381 bytes
>
> Interval 3 Stats:
> Block Size: 131072b+
> No. of Reads: 24
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 1 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
> 0.00 69.00 us 69.00 us 69.00 us 1 OPEN
> 0.00 51.00 us 38.00 us 64.00 us 2 STATFS
> 0.00 150.00 us 150.00 us 150.00 us 1 TRUNCATE
> 0.00 88.33 us 42.00 us 127.00 us 3 FTRUNCATE
> 0.00 200.33 us 143.00 us 254.00 us 3 READDIR
> 0.00 276.00 us 78.00 us 800.00 us 4 FSTAT
> 0.00 346.14 us 24.00 us 1508.00 us 7 FINODELK
> 0.01 6407.00 us 6407.00 us 6407.00 us 1 RENAME
> 0.03 9475.00 us 5680.00 us 13270.00 us 2 XATTROP
> 0.11 2800.33 us 154.00 us 19781.00 us 24 READ
> 0.41 122646.00 us 610.00 us 244682.00 us 2 FSYNC
> 0.67 66579.33 us 2976.00 us 368832.00 us 6 FXATTROP
> 0.72 214305.00 us 25134.00 us 403476.00 us 2 CREATE
> 1.57 20415.13 us 53.00 us 863671.00 us 46 SETATTR
> 7.13 43065.32 us 380.00 us 2023922.00 us 99 GETXATTR
> 17.08 57053.73 us 21.00 us 4791081.00 us 179 ENTRYLK
> 19.27 24833.95 us 18.00 us 1721894.00 us 464 INODELK
> 52.99 31373.74 us 99.00 us 4753189.00 us 1010 LOOKUP
>
> Duration: 27 seconds
> Data Read: 3145728 bytes
> Data Written: 0 bytes
>
> Brick: hangar:/export/wd1a
> --------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 0 1 1
> No. of Writes: 119 89 206
>
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 18 456 638
> No. of Writes: 2674 20109 26254
>
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 653 1114 1047
> No. of Writes: 16225 21757 26094
>
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 815 445 1022
> No. of Writes: 46483 39502 45837
>
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 1188 2113 808
> No. of Writes: 21375 14211 9791
>
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 528 6478 153851
> No. of Writes: 12407 3688 153
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 278772 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 267233 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 28625 RELEASEDIR
> 0.00 66.60 us 59.00 us 79.00 us 5 WRITE
> 0.00 1908.00 us 1908.00 us 1908.00 us 1 READDIR
> 0.01 1413.71 us 35.00 us 9513.00 us 7 STATFS
> 0.01 4227.67 us 63.00 us 12551.00 us 3 OPEN
> 0.01 4651.33 us 75.00 us 13710.00 us 3 TRUNCATE
> 0.01 416.73 us 14.00 us 8592.00 us 37 FSTAT
> 0.02 19443.00 us 19443.00 us 19443.00 us 1 SETXATTR
> 0.02 4094.20 us 1650.00 us 9259.00 us 5 FSYNC
> 0.03 4631.62 us 42.00 us 36533.00 us 8 FTRUNCATE
> 0.05 1433.56 us 35.00 us 48945.00 us 36 READLINK
> 0.13 24155.33 us 5373.00 us 83746.00 us 6 XATTROP
> 0.14 5630.43 us 21.00 us 74418.00 us 28 FINODELK
> 0.15 32752.40 us 26251.00 us 40382.00 us 5 CREATE
> 0.19 69616.33 us 5391.00 us 188134.00 us 3 RENAME
> 0.19 2929.67 us 151.00 us 66289.00 us 72 READ
> 0.28 11943.19 us 2855.00 us 95080.00 us 26 FXATTROP
> 0.95 7093.66 us 53.00 us 418369.00 us 151 SETATTR
> 1.50 76999.59 us 17.00 us 1692792.00 us 22 ENTRYLK
> 3.39 88812.86 us 32.00 us 1730394.00 us 43 GETXATTR
> 23.83 37513.31 us 17.00 us 4552377.00 us 716 INODELK
> 69.10 20612.44 us 88.00 us 5304073.00 us 3778 LOOKUP
>
> Duration: 561502 seconds
> Data Read: 20818299002 bytes
> Data Written: 1477574380 bytes
>
> Interval 3 Stats:
> Block Size: 512b+ 131072b+
> No. of Reads: 0 24
> No. of Writes: 2 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 1 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 3 RELEASE
> 0.00 69.00 us 69.00 us 69.00 us 1 OPEN
> 0.00 62.00 us 39.00 us 85.00 us 2 STATFS
> 0.00 64.00 us 59.00 us 69.00 us 2 WRITE
> 0.00 169.00 us 169.00 us 169.00 us 1 TRUNCATE
> 0.00 76.00 us 52.00 us 100.00 us 3 FTRUNCATE
> 0.00 139.11 us 39.00 us 509.00 us 9 READLINK
> 0.01 2932.50 us 1751.00 us 4114.00 us 2 FSYNC
> 0.03 6897.50 us 5373.00 us 8422.00 us 2 XATTROP
> 0.04 1268.91 us 14.00 us 8592.00 us 11 FSTAT
> 0.14 5535.10 us 2905.00 us 13880.00 us 10 FXATTROP
> 0.15 29488.00 us 26251.00 us 32725.00 us 2 CREATE
> 0.25 4117.08 us 151.00 us 38975.00 us 24 READ
> 0.29 11564.40 us 22.00 us 74418.00 us 10 FINODELK
> 0.47 188134.00 us 188134.00 us 188134.00 us 1 RENAME
> 2.40 20793.83 us 53.00 us 418369.00 us 46 SETATTR
> 4.25 211640.62 us 17.00 us 1692792.00 us 8 ENTRYLK
> 4.41 439321.25 us 640.00 us 1730394.00 us 4 GETXATTR
> 24.88 43685.58 us 20.00 us 2885874.00 us 227 INODELK
> 62.67 22440.90 us 108.00 us 2617203.00 us 1113 LOOKUP
>
> Duration: 27 seconds
> Data Read: 3145728 bytes
> Data Written: 1800 bytes
>
More information about the Gluster-devel
mailing list