[Gluster-users] Performance drop when upgrading from 3.8 to 6.5
Riccardo Murri
riccardo.murri at gmail.com
Mon Nov 4 09:11:42 UTC 2019
Hello Amar,
> Can you please check the profile info [1] ? That may give some hints.
I am attaching the output of `sudo gluster volume profile info` as a text file
to preserve formatting. This covers the time from Friday night to
Monday morning;
during this time the cluster has been the target of a an `rsync` command copying
a large directory tree (14TB; it's taking more than 2 weeks now...).
My take from cursorily reading the profile:
- metadata operations (opendir, create, rename) seem to have very high
latency (up to 25 seconds for a rename!)
- large block size (>128kB) are rare; most blocks seems to be 8kB,
16kB, or 128kB.
I personally do not see any obvious culprit or optimization
opportunities; is there something I'm missing out that catches the eye
of someone more experienced?
Thanks,
R
-------------- next part --------------
$ sudo gluster volume profile glusterfs info
Brick: glusterfs-server-001:/srv/glusterfs
------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 2 18 12
No. of Writes: 47 15 22
Block Size: 16b+ 32b+ 64b+
No. of Reads: 31 77 336
No. of Writes: 65 7869 47979
Block Size: 128b+ 256b+ 512b+
No. of Reads: 291 648 1125
No. of Writes: 8909 125293 62952
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2054 3868 7903
No. of Writes: 11728 293386 87850
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 16084 35981 66646
No. of Writes: 10758668 6852707 289622
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 128659 15220392 0
No. of Writes: 239551 12723330 316
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760721 FORGET
0.00 0.00 us 0.00 us 0.00 us 877203 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 16174.60 us 554.21 us 31794.98 us 2 READDIRP
0.00 185.11 us 82.75 us 660.68 us 377 OPEN
0.00 95974.52 us 95974.52 us 95974.52 us 1 OPENDIR
0.01 59611.16 us 444.39 us 160460.53 us 3 SYMLINK
0.01 217.87 us 72.47 us 15655.59 us 1337 SETXATTR
0.30 79.48 us 16.15 us 6242.48 us 123275 FLUSH
0.31 120.49 us 30.68 us 10641.08 us 83225 FSTAT
0.50 12161.43 us 273.52 us 298385.31 us 1337 MKDIR
0.86 129.11 us 32.64 us 9029.93 us 216678 STATFS
1.35 89.34 us 12.29 us 16695.73 us 491862 ENTRYLK
2.09 91.83 us 11.85 us 20615.78 us 737946 INODELK
2.38 628.28 us 115.01 us 25374396.01 us 122889 RENAME
2.42 2080.96 us 36.39 us 403164.19 us 37810 READ
2.74 179.20 us 49.63 us 1041137.91 us 497183 SETATTR
6.34 154.95 us 33.60 us 17451.12 us 1329171 STAT
11.35 201.91 us 31.14 us 181613.31 us 1824910 LOOKUP
28.65 7568.96 us 169.02 us 1888964.31 us 122898 CREATE
40.68 517.99 us 58.77 us 1076940.31 us 2549418 WRITE
Duration: 2936614 seconds
Data Read: 2012268179193 bytes
Data Written: 2005120204522 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 2 18 12
No. of Writes: 47 15 22
Block Size: 16b+ 32b+ 64b+
No. of Reads: 31 77 336
No. of Writes: 65 7869 47979
Block Size: 128b+ 256b+ 512b+
No. of Reads: 291 648 1125
No. of Writes: 8909 125293 62952
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2054 3868 7903
No. of Writes: 11728 293386 87850
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 16084 35981 66646
No. of Writes: 10758668 6852707 289622
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 128659 15220392 0
No. of Writes: 239551 12723330 316
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760721 FORGET
0.00 0.00 us 0.00 us 0.00 us 877203 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 16174.60 us 554.21 us 31794.98 us 2 READDIRP
0.00 185.11 us 82.75 us 660.68 us 377 OPEN
0.00 95974.52 us 95974.52 us 95974.52 us 1 OPENDIR
0.01 59611.16 us 444.39 us 160460.53 us 3 SYMLINK
0.01 217.87 us 72.47 us 15655.59 us 1337 SETXATTR
0.30 79.48 us 16.15 us 6242.48 us 123275 FLUSH
0.31 120.49 us 30.68 us 10641.08 us 83225 FSTAT
0.50 12161.43 us 273.52 us 298385.31 us 1337 MKDIR
0.86 129.11 us 32.64 us 9029.93 us 216678 STATFS
1.35 89.34 us 12.29 us 16695.73 us 491862 ENTRYLK
2.09 91.83 us 11.85 us 20615.78 us 737946 INODELK
2.38 628.28 us 115.01 us 25374396.01 us 122889 RENAME
2.42 2081.07 us 36.39 us 403164.19 us 37810 READ
2.74 179.20 us 49.63 us 1041137.91 us 497183 SETATTR
6.34 154.95 us 33.60 us 17451.12 us 1329171 STAT
11.35 201.91 us 31.14 us 181613.31 us 1824910 LOOKUP
28.65 7568.96 us 169.02 us 1888964.31 us 122898 CREATE
40.68 517.99 us 58.77 us 1076940.31 us 2549418 WRITE
Duration: 2936614 seconds
Data Read: 2012268179193 bytes
Data Written: 2005120204522 bytes
Brick: glusterfs-server-003:/srv/glusterfs
------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 1 5
No. of Writes: 4 31 9
Block Size: 8b+ 16b+ 32b+
No. of Reads: 9 34 46
No. of Writes: 24 49 7817
Block Size: 64b+ 128b+ 256b+
No. of Reads: 206 267 630
No. of Writes: 47976 8879 125845
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1037 1984 3824
No. of Writes: 63261 11876 295271
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7792 15801 35586
No. of Writes: 88129 10823535 6852962
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 67883 129953 15451758
No. of Writes: 290421 246054 12704001
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 298
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760063 FORGET
0.00 0.00 us 0.00 us 0.00 us 878086 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 23734.89 us 23734.89 us 23734.89 us 1 READLINK
0.00 30289.28 us 30289.28 us 30289.28 us 1 OPENDIR
0.00 13991.48 us 395.61 us 41106.22 us 3 SYMLINK
0.00 208.58 us 97.11 us 696.62 us 369 OPEN
0.00 63746.13 us 323.04 us 127169.22 us 2 READDIRP
0.01 234.57 us 83.38 us 2983.54 us 1337 SETXATTR
0.32 122.63 us 34.38 us 7160.49 us 82807 FSTAT
0.33 84.07 us 16.33 us 1068.90 us 122557 FLUSH
0.58 13565.91 us 180.57 us 859005.00 us 1337 MKDIR
0.93 134.33 us 33.54 us 12274.34 us 216677 STATFS
1.51 96.50 us 14.75 us 18505.08 us 489054 ENTRYLK
1.54 393.63 us 104.09 us 934883.44 us 122178 RENAME
2.24 2067.85 us 45.97 us 300114.83 us 33824 READ
2.31 98.57 us 15.30 us 20739.38 us 733752 INODELK
2.87 181.67 us 48.51 us 362666.29 us 494340 SETATTR
6.33 148.89 us 33.34 us 12370.01 us 1328847 STAT
12.08 207.40 us 29.84 us 185863.09 us 1820818 LOOKUP
28.29 7237.59 us 163.85 us 870990.45 us 122188 CREATE
40.65 501.44 us 53.64 us 1029279.61 us 2534634 WRITE
Duration: 2936613 seconds
Data Read: 2042771607327 bytes
Data Written: 2004299986154 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 1 5
No. of Writes: 4 31 9
Block Size: 8b+ 16b+ 32b+
No. of Reads: 9 34 46
No. of Writes: 24 49 7817
Block Size: 64b+ 128b+ 256b+
No. of Reads: 206 267 630
No. of Writes: 47976 8879 125845
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1037 1984 3824
No. of Writes: 63261 11876 295271
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7792 15801 35586
No. of Writes: 88129 10823535 6852962
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 67883 129953 15451758
No. of Writes: 290421 246054 12704001
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 298
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760063 FORGET
0.00 0.00 us 0.00 us 0.00 us 878086 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 23734.89 us 23734.89 us 23734.89 us 1 READLINK
0.00 30289.28 us 30289.28 us 30289.28 us 1 OPENDIR
0.00 13991.48 us 395.61 us 41106.22 us 3 SYMLINK
0.00 208.58 us 97.11 us 696.62 us 369 OPEN
0.00 63746.13 us 323.04 us 127169.22 us 2 READDIRP
0.01 234.57 us 83.38 us 2983.54 us 1337 SETXATTR
0.32 122.63 us 34.38 us 7160.49 us 82807 FSTAT
0.33 84.07 us 16.33 us 1068.90 us 122557 FLUSH
0.58 13565.91 us 180.57 us 859005.00 us 1337 MKDIR
0.93 134.33 us 33.54 us 12274.34 us 216677 STATFS
1.51 96.50 us 14.75 us 18505.08 us 489054 ENTRYLK
1.54 393.63 us 104.09 us 934883.44 us 122178 RENAME
2.24 2067.85 us 45.97 us 300114.83 us 33824 READ
2.31 98.57 us 15.30 us 20739.38 us 733752 INODELK
2.87 181.67 us 48.51 us 362666.29 us 494340 SETATTR
6.33 148.89 us 33.34 us 12370.01 us 1328847 STAT
12.08 207.40 us 29.84 us 185863.09 us 1820818 LOOKUP
28.29 7237.59 us 163.85 us 870990.45 us 122188 CREATE
40.65 501.44 us 53.64 us 1029279.61 us 2534634 WRITE
Duration: 2936613 seconds
Data Read: 2042771607327 bytes
Data Written: 2004299986154 bytes
Brick: glusterfs-server-007:/srv/glusterfs
------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 1 3
No. of Writes: 2 47 7
Block Size: 8b+ 16b+ 32b+
No. of Reads: 9 48 49
No. of Writes: 11 62 7773
Block Size: 64b+ 128b+ 256b+
No. of Reads: 151 220 509
No. of Writes: 46782 8282 124151
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1051 2040 3960
No. of Writes: 61178 11270 293665
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7421 15965 35941
No. of Writes: 87812 10819486 6800022
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 67354 128628 15259778
No. of Writes: 291441 240144 12720415
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 285
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760010 FORGET
0.00 0.00 us 0.00 us 0.00 us 875188 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 1946.15 us 1946.15 us 1946.15 us 1 READLINK
0.00 200.95 us 102.51 us 707.02 us 335 OPEN
0.00 86756.14 us 86756.14 us 86756.14 us 1 OPENDIR
0.00 44438.95 us 379.52 us 88498.38 us 2 SYMLINK
0.01 88459.57 us 97.37 us 176821.77 us 2 READDIRP
0.01 251.07 us 61.35 us 1497.62 us 1337 SETXATTR
0.31 123.73 us 34.00 us 6948.27 us 82822 FSTAT
0.32 85.83 us 16.27 us 6444.86 us 123010 FLUSH
0.58 14363.95 us 178.44 us 847451.60 us 1337 MKDIR
0.90 137.14 us 32.48 us 9572.80 us 216677 STATFS
1.48 99.27 us 12.80 us 18204.06 us 491038 ENTRYLK
2.03 2069.17 us 39.37 us 305857.53 us 32278 READ
2.28 101.63 us 12.83 us 32912.17 us 736744 INODELK
2.81 186.13 us 44.12 us 531739.19 us 496287 SETATTR
3.82 1024.47 us 107.60 us 78557223.60 us 122666 RENAME
6.26 154.70 us 33.96 us 17842.98 us 1329298 STAT
11.86 213.83 us 29.83 us 287648.96 us 1823022 LOOKUP
28.35 7592.49 us 175.50 us 2709284.15 us 122675 CREATE
38.95 501.36 us 54.27 us 827604.51 us 2552878 WRITE
Duration: 2936614 seconds
Data Read: 2017467234933 bytes
Data Written: 2004757317618 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 1 3
No. of Writes: 2 47 7
Block Size: 8b+ 16b+ 32b+
No. of Reads: 9 48 49
No. of Writes: 11 62 7773
Block Size: 64b+ 128b+ 256b+
No. of Reads: 151 220 509
No. of Writes: 46782 8282 124151
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1051 2040 3960
No. of Writes: 61178 11270 293665
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7421 15965 35941
No. of Writes: 87812 10819486 6800022
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 67354 128628 15259778
No. of Writes: 291441 240144 12720415
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 285
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760010 FORGET
0.00 0.00 us 0.00 us 0.00 us 875188 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 1946.15 us 1946.15 us 1946.15 us 1 READLINK
0.00 200.95 us 102.51 us 707.02 us 335 OPEN
0.00 86756.14 us 86756.14 us 86756.14 us 1 OPENDIR
0.00 44438.95 us 379.52 us 88498.38 us 2 SYMLINK
0.01 88459.57 us 97.37 us 176821.77 us 2 READDIRP
0.01 251.07 us 61.35 us 1497.62 us 1337 SETXATTR
0.31 123.73 us 34.00 us 6948.27 us 82822 FSTAT
0.32 85.83 us 16.27 us 6444.86 us 123010 FLUSH
0.58 14363.95 us 178.44 us 847451.60 us 1337 MKDIR
0.90 137.14 us 32.48 us 9572.80 us 216677 STATFS
1.48 99.27 us 12.80 us 18204.06 us 491038 ENTRYLK
2.03 2069.17 us 39.37 us 305857.53 us 32278 READ
2.28 101.63 us 12.83 us 32912.17 us 736744 INODELK
2.81 186.13 us 44.12 us 531739.19 us 496287 SETATTR
3.82 1024.47 us 107.60 us 78557223.60 us 122666 RENAME
6.26 154.70 us 33.96 us 17842.98 us 1329298 STAT
11.86 213.83 us 29.83 us 287648.96 us 1823022 LOOKUP
28.35 7592.49 us 175.50 us 2709284.15 us 122675 CREATE
38.95 501.36 us 54.27 us 827604.51 us 2552878 WRITE
Duration: 2936614 seconds
Data Read: 2017467234933 bytes
Data Written: 2004757317618 bytes
Brick: glusterfs-server-008:/srv/glusterfs
------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 11
No. of Writes: 4 43 15
Block Size: 8b+ 16b+ 32b+
No. of Reads: 6 32 51
No. of Writes: 21 57 7604
Block Size: 64b+ 128b+ 256b+
No. of Reads: 139 245 601
No. of Writes: 47400 9309 126614
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1017 2053 4007
No. of Writes: 63836 12996 294860
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7724 16240 35266
No. of Writes: 87834 10853140 6829372
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 66488 130659 14994967
No. of Writes: 293071 239905 12697661
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 266
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760943 FORGET
0.00 0.00 us 0.00 us 0.00 us 877102 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 511.87 us 442.51 us 581.23 us 2 SYMLINK
0.00 46748.86 us 46748.86 us 46748.86 us 1 READDIRP
0.00 54050.56 us 54050.56 us 54050.56 us 1 OPENDIR
0.00 228.24 us 113.08 us 6740.27 us 373 OPEN
0.01 242.70 us 90.66 us 25274.55 us 1337 SETXATTR
0.30 124.10 us 37.85 us 6147.92 us 82847 FSTAT
0.31 86.38 us 15.04 us 5570.68 us 123282 FLUSH
0.48 12392.41 us 217.67 us 648607.98 us 1337 MKDIR
0.89 140.06 us 33.20 us 7281.76 us 216677 STATFS
1.33 370.67 us 105.93 us 620038.23 us 122896 RENAME
1.45 101.11 us 14.63 us 16161.03 us 491928 ENTRYLK
2.21 102.51 us 13.86 us 19717.11 us 738064 INODELK
2.31 2165.83 us 39.52 us 456893.32 us 36607 READ
2.69 185.79 us 51.46 us 69209.88 us 497214 SETATTR
6.00 154.61 us 34.94 us 31242.32 us 1329565 STAT
11.39 214.05 us 30.74 us 541340.40 us 1824942 LOOKUP
33.10 9233.79 us 171.40 us 88679990.55 us 122910 CREATE
37.52 500.98 us 53.60 us 800953.46 us 2567932 WRITE
Duration: 2936613 seconds
Data Read: 1982870875754 bytes
Data Written: 2002756853576 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 11
No. of Writes: 4 43 15
Block Size: 8b+ 16b+ 32b+
No. of Reads: 6 32 51
No. of Writes: 21 57 7604
Block Size: 64b+ 128b+ 256b+
No. of Reads: 139 245 601
No. of Writes: 47400 9309 126614
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1017 2053 4007
No. of Writes: 63836 12996 294860
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7724 16240 35266
No. of Writes: 87834 10853140 6829372
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 66488 130659 14994967
No. of Writes: 293071 239905 12697661
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 266
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760943 FORGET
0.00 0.00 us 0.00 us 0.00 us 877102 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 511.87 us 442.51 us 581.23 us 2 SYMLINK
0.00 46748.86 us 46748.86 us 46748.86 us 1 READDIRP
0.00 54050.56 us 54050.56 us 54050.56 us 1 OPENDIR
0.00 228.24 us 113.08 us 6740.27 us 373 OPEN
0.01 242.70 us 90.66 us 25274.55 us 1337 SETXATTR
0.30 124.10 us 37.85 us 6147.92 us 82847 FSTAT
0.31 86.38 us 15.04 us 5570.68 us 123282 FLUSH
0.48 12392.41 us 217.67 us 648607.98 us 1337 MKDIR
0.89 140.06 us 33.20 us 7281.76 us 216677 STATFS
1.33 370.67 us 105.93 us 620038.23 us 122896 RENAME
1.45 101.11 us 14.63 us 16161.03 us 491928 ENTRYLK
2.21 102.51 us 13.86 us 19717.11 us 738064 INODELK
2.31 2165.83 us 39.52 us 456893.32 us 36607 READ
2.69 185.79 us 51.46 us 69209.88 us 497214 SETATTR
6.00 154.61 us 34.94 us 31242.32 us 1329565 STAT
11.39 214.05 us 30.74 us 541340.40 us 1824942 LOOKUP
33.10 9233.79 us 171.40 us 88679990.55 us 122910 CREATE
37.52 500.98 us 53.60 us 800953.46 us 2567932 WRITE
Duration: 2936613 seconds
Data Read: 1982870875754 bytes
Data Written: 2002756853576 bytes
Brick: glusterfs-server-002:/srv/glusterfs
------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 4 5
No. of Writes: 6 44 8
Block Size: 8b+ 16b+ 32b+
No. of Reads: 10 29 63
No. of Writes: 14 54 7727
Block Size: 64b+ 128b+ 256b+
No. of Reads: 217 256 614
No. of Writes: 47383 8560 124718
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 999 2174 4160
No. of Writes: 62177 11292 293569
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7867 16054 35775
No. of Writes: 87716 10789835 6831300
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 66988 129409 15315216
No. of Writes: 291029 240118 12700481
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 286
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760249 FORGET
0.00 0.00 us 0.00 us 0.00 us 878245 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 550.43 us 550.43 us 550.43 us 1 OPENDIR
0.00 46921.21 us 46921.21 us 46921.21 us 1 READDIRP
0.00 199.36 us 78.62 us 858.20 us 358 OPEN
0.01 227.48 us 63.02 us 1406.38 us 1337 SETXATTR
0.01 106036.10 us 379.00 us 407187.84 us 4 SYMLINK
0.31 122.72 us 34.01 us 6661.97 us 82581 FSTAT
0.32 83.14 us 16.66 us 6999.97 us 122609 FLUSH
0.56 13474.57 us 225.70 us 831509.75 us 1337 MKDIR
0.91 136.00 us 32.69 us 8138.46 us 216677 STATFS
1.45 382.74 us 114.83 us 906798.58 us 122241 RENAME
1.46 96.79 us 14.17 us 20609.26 us 489278 ENTRYLK
2.27 100.04 us 11.86 us 24366.52 us 734074 INODELK
2.36 2380.77 us 38.86 us 596570.69 us 32071 READ
2.93 191.83 us 52.47 us 1098174.76 us 494594 SETATTR
6.40 155.75 us 34.02 us 15763.39 us 1329122 STAT
11.85 210.48 us 30.96 us 216326.61 us 1821062 LOOKUP
28.77 7612.00 us 163.44 us 1490314.45 us 122251 CREATE
40.38 513.68 us 56.02 us 1426464.65 us 2543049 WRITE
Duration: 2936613 seconds
Data Read: 2024792095578 bytes
Data Written: 2002323867368 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 4 5
No. of Writes: 6 44 8
Block Size: 8b+ 16b+ 32b+
No. of Reads: 10 29 63
No. of Writes: 14 54 7727
Block Size: 64b+ 128b+ 256b+
No. of Reads: 217 256 614
No. of Writes: 47383 8560 124718
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 999 2174 4160
No. of Writes: 62177 11292 293569
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7867 16054 35775
No. of Writes: 87716 10789835 6831300
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 66988 129409 15315216
No. of Writes: 291029 240118 12700481
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 286
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 760249 FORGET
0.00 0.00 us 0.00 us 0.00 us 878245 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 550.43 us 550.43 us 550.43 us 1 OPENDIR
0.00 46921.21 us 46921.21 us 46921.21 us 1 READDIRP
0.00 199.36 us 78.62 us 858.20 us 358 OPEN
0.01 227.48 us 63.02 us 1406.38 us 1337 SETXATTR
0.01 106036.10 us 379.00 us 407187.84 us 4 SYMLINK
0.31 122.72 us 34.01 us 6661.97 us 82581 FSTAT
0.32 83.14 us 16.66 us 6999.97 us 122609 FLUSH
0.56 13474.57 us 225.70 us 831509.75 us 1337 MKDIR
0.91 136.00 us 32.69 us 8138.46 us 216677 STATFS
1.45 382.74 us 114.83 us 906798.58 us 122241 RENAME
1.46 96.79 us 14.17 us 20609.26 us 489278 ENTRYLK
2.27 100.04 us 11.86 us 24366.52 us 734074 INODELK
2.36 2380.71 us 38.86 us 596570.69 us 32071 READ
2.93 191.83 us 52.47 us 1098174.76 us 494594 SETATTR
6.40 155.75 us 34.02 us 15763.39 us 1329122 STAT
11.85 210.48 us 30.96 us 216326.61 us 1821062 LOOKUP
28.77 7612.00 us 163.44 us 1490314.45 us 122251 CREATE
40.38 513.68 us 56.02 us 1426464.65 us 2543049 WRITE
Duration: 2936613 seconds
Data Read: 2024792095578 bytes
Data Written: 2002323867368 bytes
Brick: glusterfs-server-005:/srv/glusterfs
------------------------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 4 3 7
No. of Writes: 52 16 19
Block Size: 16b+ 32b+ 64b+
No. of Reads: 33 53 169
No. of Writes: 49 7913 48095
Block Size: 128b+ 256b+ 512b+
No. of Reads: 215 571 1019
No. of Writes: 9528 125591 63575
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2026 3756 7697
No. of Writes: 11496 293277 87768
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 15833 35386 67097
No. of Writes: 10764933 6849111 289233
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 129816 15330275 0
No. of Writes: 239469 12697465 324
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 761089 FORGET
0.00 0.00 us 0.00 us 0.00 us 877250 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 27055.56 us 27055.56 us 27055.56 us 1 OPENDIR
0.00 14055.61 us 418.00 us 53534.63 us 4 SYMLINK
0.00 205.56 us 99.61 us 988.66 us 348 OPEN
0.00 53888.73 us 272.35 us 107505.10 us 2 READDIRP
0.01 352.22 us 95.36 us 146156.03 us 1337 SETXATTR
0.28 82.27 us 14.07 us 5471.66 us 123111 FLUSH
0.28 122.98 us 36.31 us 10769.50 us 83127 FSTAT
0.50 13730.50 us 249.06 us 1510813.17 us 1337 MKDIR
0.81 135.81 us 25.37 us 8437.55 us 216677 STATFS
1.22 363.33 us 115.19 us 702658.13 us 122751 RENAME
1.30 96.66 us 15.12 us 18893.15 us 491322 ENTRYLK
1.96 2094.11 us 42.38 us 359704.68 us 34240 READ
1.99 98.62 us 14.67 us 16834.50 us 737142 INODELK
2.49 183.33 us 49.59 us 293925.11 us 496634 SETATTR
5.51 151.39 us 34.18 us 19410.12 us 1329114 STAT
10.98 208.37 us 30.23 us 305491.07 us 1924121 LOOKUP
12.32 133.79 us 24.19 us 24796.27 us 3362310 READLINK
24.78 7366.92 us 172.39 us 3216981.76 us 122762 CREATE
35.55 511.07 us 56.96 us 765927.63 us 2538760 WRITE
Duration: 2936613 seconds
Data Read: 2026789649490 bytes
Data Written: 2001490811195 bytes
Interval 0 Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 4 3 7
No. of Writes: 52 16 19
Block Size: 16b+ 32b+ 64b+
No. of Reads: 33 53 169
No. of Writes: 49 7913 48095
Block Size: 128b+ 256b+ 512b+
No. of Reads: 215 571 1019
No. of Writes: 9528 125591 63575
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2026 3756 7697
No. of Writes: 11496 293277 87768
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 15833 35386 67097
No. of Writes: 10764933 6849111 289233
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 129816 15330275 0
No. of Writes: 239469 12697465 324
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 761089 FORGET
0.00 0.00 us 0.00 us 0.00 us 877250 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 27055.56 us 27055.56 us 27055.56 us 1 OPENDIR
0.00 14055.61 us 418.00 us 53534.63 us 4 SYMLINK
0.00 205.56 us 99.61 us 988.66 us 348 OPEN
0.00 53888.73 us 272.35 us 107505.10 us 2 READDIRP
0.01 352.22 us 95.36 us 146156.03 us 1337 SETXATTR
0.28 82.27 us 14.07 us 5471.66 us 123111 FLUSH
0.28 122.98 us 36.31 us 10769.50 us 83127 FSTAT
0.50 13730.50 us 249.06 us 1510813.17 us 1337 MKDIR
0.81 135.81 us 25.37 us 8437.55 us 216677 STATFS
1.22 363.33 us 115.19 us 702658.13 us 122751 RENAME
1.30 96.66 us 15.12 us 18893.15 us 491322 ENTRYLK
1.96 2094.16 us 42.38 us 359704.68 us 34240 READ
1.99 98.62 us 14.67 us 16834.50 us 737142 INODELK
2.49 183.33 us 49.59 us 293925.11 us 496634 SETATTR
5.51 151.39 us 34.18 us 19410.12 us 1329114 STAT
10.98 208.37 us 30.23 us 305491.07 us 1924121 LOOKUP
12.32 133.79 us 24.19 us 24796.27 us 3362310 READLINK
24.78 7366.92 us 172.39 us 3216981.76 us 122762 CREATE
35.55 511.07 us 56.96 us 765927.63 us 2538760 WRITE
Duration: 2936613 seconds
Data Read: 2026789649490 bytes
Data Written: 2001490811195 bytes
Brick: glusterfs-server-006:/srv/glusterfs
------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 12
No. of Writes: 2 59 8
Block Size: 8b+ 16b+ 32b+
No. of Reads: 19 20 61
No. of Writes: 16 52 7775
Block Size: 64b+ 128b+ 256b+
No. of Reads: 152 249 667
No. of Writes: 46572 8270 124912
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1029 1806 3777
No. of Writes: 62658 12107 293760
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7591 16284 35374
No. of Writes: 87794 10811902 6825757
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 66785 129681 15196967
No. of Writes: 290003 238724 12681970
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 312
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 759997 FORGET
0.00 0.00 us 0.00 us 0.00 us 876383 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 25546.82 us 25546.82 us 25546.82 us 1 OPENDIR
0.00 19223.13 us 181.54 us 38264.73 us 2 READDIRP
0.00 212.54 us 98.56 us 725.51 us 343 OPEN
0.00 41228.94 us 17427.73 us 84573.17 us 3 SYMLINK
0.01 228.90 us 69.37 us 1174.57 us 1337 SETXATTR
0.29 125.67 us 30.08 us 6781.36 us 82251 FSTAT
0.29 85.99 us 14.01 us 5756.84 us 122256 FLUSH
0.47 12732.09 us 251.34 us 557837.65 us 1337 MKDIR
0.82 136.64 us 29.19 us 5954.76 us 216678 STATFS
1.36 403.90 us 99.30 us 960987.11 us 121904 RENAME
1.38 102.06 us 14.64 us 18850.86 us 487950 ENTRYLK
1.92 1963.09 us 35.94 us 450094.97 us 35237 READ
2.10 103.57 us 14.68 us 19286.41 us 732092 INODELK
5.59 151.77 us 31.52 us 21707.17 us 1329145 STAT
6.04 138.64 us 30.37 us 56900.52 us 1571940 READLINK
7.05 515.99 us 45.21 us 80443768.38 us 493241 SETATTR
11.23 214.75 us 27.75 us 583654.68 us 1885952 LOOKUP
25.98 7688.85 us 153.38 us 16089338.93 us 121913 CREATE
35.45 503.91 us 47.07 us 969118.07 us 2538136 WRITE
Duration: 2936612 seconds
Data Read: 2009273726518 bytes
Data Written: 1999857803405 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 12
No. of Writes: 2 59 8
Block Size: 8b+ 16b+ 32b+
No. of Reads: 19 20 61
No. of Writes: 16 52 7775
Block Size: 64b+ 128b+ 256b+
No. of Reads: 152 249 667
No. of Writes: 46572 8270 124912
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1029 1806 3777
No. of Writes: 62658 12107 293760
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7591 16284 35374
No. of Writes: 87794 10811902 6825757
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 66785 129681 15196967
No. of Writes: 290003 238724 12681970
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 312
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 759997 FORGET
0.00 0.00 us 0.00 us 0.00 us 876383 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 25546.82 us 25546.82 us 25546.82 us 1 OPENDIR
0.00 19223.13 us 181.54 us 38264.73 us 2 READDIRP
0.00 212.54 us 98.56 us 725.51 us 343 OPEN
0.00 41228.94 us 17427.73 us 84573.17 us 3 SYMLINK
0.01 228.90 us 69.37 us 1174.57 us 1337 SETXATTR
0.29 125.67 us 30.08 us 6781.36 us 82251 FSTAT
0.29 85.99 us 14.01 us 5756.84 us 122256 FLUSH
0.47 12732.09 us 251.34 us 557837.65 us 1337 MKDIR
0.82 136.64 us 29.19 us 5954.76 us 216678 STATFS
1.36 403.90 us 99.30 us 960987.11 us 121904 RENAME
1.38 102.06 us 14.64 us 18850.86 us 487950 ENTRYLK
1.92 1963.09 us 35.94 us 450094.97 us 35237 READ
2.10 103.57 us 14.68 us 19286.41 us 732092 INODELK
5.59 151.77 us 31.52 us 21707.17 us 1329145 STAT
6.04 138.64 us 30.37 us 56900.52 us 1571940 READLINK
7.05 515.99 us 45.21 us 80443768.38 us 493241 SETATTR
11.23 214.75 us 27.75 us 583654.68 us 1885952 LOOKUP
25.98 7688.85 us 153.38 us 16089338.93 us 121913 CREATE
35.45 503.91 us 47.07 us 969118.07 us 2538136 WRITE
Duration: 2936612 seconds
Data Read: 2009273726518 bytes
Data Written: 1999857803405 bytes
Brick: glusterfs-server-004:/srv/glusterfs
------------------------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 2 9
No. of Writes: 3 47 11
Block Size: 8b+ 16b+ 32b+
No. of Reads: 15 34 51
No. of Writes: 24 39 7729
Block Size: 64b+ 128b+ 256b+
No. of Reads: 150 206 586
No. of Writes: 49078 8897 125797
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1017 2103 3927
No. of Writes: 62556 11244 294675
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7994 15937 35538
No. of Writes: 88183 10816399 6833859
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 67465 129676 15270559
No. of Writes: 294684 241097 12687722
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 309
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 759877 FORGET
0.00 0.00 us 0.00 us 0.00 us 876212 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 20372.47 us 222.50 us 40522.44 us 2 READDIRP
0.00 190.68 us 100.72 us 487.51 us 374 OPEN
0.00 48639.83 us 9423.20 us 87856.46 us 2 SYMLINK
0.00 108883.61 us 108883.61 us 108883.61 us 1 OPENDIR
0.01 234.64 us 63.55 us 14273.03 us 1337 SETXATTR
0.30 122.28 us 35.47 us 5785.74 us 82730 FSTAT
0.31 82.91 us 14.66 us 7258.28 us 122582 FLUSH
0.61 15288.20 us 206.38 us 476600.42 us 1337 MKDIR
0.88 134.62 us 30.75 us 8385.37 us 216677 STATFS
1.40 95.11 us 9.77 us 18536.12 us 489146 ENTRYLK
2.15 97.48 us 13.42 us 20998.88 us 733890 INODELK
2.23 2053.85 us 45.72 us 346922.27 us 36151 READ
2.26 616.26 us 112.17 us 25253859.69 us 122201 RENAME
2.75 185.47 us 51.28 us 968572.94 us 494427 SETATTR
6.20 155.41 us 34.07 us 13948.71 us 1329009 STAT
11.17 204.32 us 30.59 us 257985.38 us 1819858 LOOKUP
30.76 8381.50 us 128.57 us 47936543.81 us 122208 CREATE
38.96 511.20 us 55.63 us 1132665.78 us 2538043 WRITE
Duration: 2936612 seconds
Data Read: 2018962698577 bytes
Data Written: 2001437401423 bytes
Interval 0 Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 2 9
No. of Writes: 3 47 11
Block Size: 8b+ 16b+ 32b+
No. of Reads: 15 34 51
No. of Writes: 24 39 7729
Block Size: 64b+ 128b+ 256b+
No. of Reads: 150 206 586
No. of Writes: 49078 8897 125797
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 1017 2103 3927
No. of Writes: 62556 11244 294675
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 7994 15937 35538
No. of Writes: 88183 10816399 6833859
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 67465 129676 15270559
No. of Writes: 294684 241097 12687722
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 309
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 759877 FORGET
0.00 0.00 us 0.00 us 0.00 us 876212 RELEASE
0.00 0.00 us 0.00 us 0.00 us 16092 RELEASEDIR
0.00 20372.47 us 222.50 us 40522.44 us 2 READDIRP
0.00 190.68 us 100.72 us 487.51 us 374 OPEN
0.00 48639.83 us 9423.20 us 87856.46 us 2 SYMLINK
0.00 108883.61 us 108883.61 us 108883.61 us 1 OPENDIR
0.01 234.64 us 63.55 us 14273.03 us 1337 SETXATTR
0.30 122.28 us 35.47 us 5785.74 us 82730 FSTAT
0.31 82.91 us 14.66 us 7258.28 us 122582 FLUSH
0.61 15288.20 us 206.38 us 476600.42 us 1337 MKDIR
0.88 134.62 us 30.75 us 8385.37 us 216677 STATFS
1.40 95.11 us 9.77 us 18536.12 us 489146 ENTRYLK
2.15 97.48 us 13.42 us 20998.88 us 733890 INODELK
2.23 2053.85 us 45.72 us 346922.27 us 36151 READ
2.26 616.26 us 112.17 us 25253859.69 us 122201 RENAME
2.75 185.47 us 51.28 us 968572.94 us 494427 SETATTR
6.20 155.41 us 34.07 us 13948.71 us 1329009 STAT
11.17 204.32 us 30.59 us 257985.38 us 1819858 LOOKUP
30.76 8381.50 us 128.57 us 47936543.81 us 122208 CREATE
38.96 511.20 us 55.63 us 1132665.78 us 2538043 WRITE
Duration: 2936612 seconds
Data Read: 2018962698577 bytes
Data Written: 2001437401423 bytes
More information about the Gluster-users
mailing list