[Gluster-users] glusterfs and glusterfsd process utilization

Mon Dec 22 04:19:50 UTC 2014

Hi, 

Could you provide the following pieces of information? 

1. The brick and client logs 
2. output of `gluster volume info` 
3. average number of entries under directories 

Just to be clear, you don't have quota enabled on the volume, do you? 

-Krutika 
----- Original Message -----

> From: "Kyle Harris" <kyle.harris98 at gmail.com>
> To: gluster-users at gluster.org
> Sent: Monday, December 22, 2014 8:27:09 AM
> Subject: Re: [Gluster-users] glusterfs and glusterfsd process utilization

> I haven't heard anything from anyone re this post yet so I have a bit more
> info to add. The last time I posted about extremely high process utilization
> for glusterfs and glusterfsd, I was asked to see if I could capture the
> output of a profile. I did so and here is the output of 2 captures (they
> appear very similar). Perhaps this will assist. Note that previously I was
> mounting with the fuse client. This time I tried NFS.

> Profile Output 1:
> Brick: 192.168.10.92:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 0 0 0
> No. of Writes: 133 6 37
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 28 93 31
> No. of Writes: 4576 6494 22228
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 217 216 254
> No. of Writes: 37627 41711 20519
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 484 978 684
> No. of Writes: 39094 42589 203747
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 1073 1800 2838
> No. of Writes: 220154 229725 215083
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 4224 14941 57812
> No. of Writes: 60938 61967 793400
> Block Size: 262144b+ 524288b+ 1048576b+
> No. of Reads: 4 4 2
> No. of Writes: 5 0 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 1835064 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 2639321 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1439759 RELEASEDIR
> 0.00 153.50 us 142.00 us 165.00 us 2 XATTROP
> 0.00 102.67 us 97.00 us 110.00 us 3 SETATTR
> 0.00 93.20 us 53.00 us 129.00 us 5 GETXATTR
> 0.00 55.07 us 33.00 us 100.00 us 15 INODELK
> 0.00 46.61 us 35.00 us 69.00 us 18 STATFS
> 0.01 48.40 us 31.00 us 64.00 us 25 LK
> 0.01 50.72 us 32.00 us 88.00 us 47 FINODELK
> 0.01 35.49 us 17.00 us 68.00 us 86 FLUSH
> 0.02 50.59 us 27.00 us 76.00 us 68 ENTRYLK
> 0.03 75.72 us 47.00 us 112.00 us 74 OPEN
> 0.06 58.79 us 32.00 us 100.00 us 222 OPENDIR
> 0.08 1535.92 us 129.00 us 16583.00 us 12 UNLINK
> 0.11 24814.00 us 24814.00 us 24814.00 us 1 TRUNCATE
> 0.12 34.25 us 11.00 us 1950.00 us 784 ACCESS
> 0.13 603.50 us 90.00 us 21829.00 us 46 FXATTROP
> 0.16 7225.40 us 63.00 us 14563.00 us 5 READLINK
> 0.17 437.90 us 54.00 us 19042.00 us 83 READ
> 0.36 85.92 us 56.00 us 313.00 us 912 FSTAT
> 0.42 59.86 us 22.00 us 1443.00 us 1533 STAT
> 0.45 4318.65 us 70.00 us 56839.00 us 23 WRITE
> 0.84 8405.18 us 161.00 us 37688.00 us 22 CREATE
> 8.33 2005.77 us 361.00 us 2796.00 us 912 READDIRP
> 88.68 921.10 us 34.00 us 159393.00 us 21137 LOOKUP
> Duration: 338151 seconds
> Data Read: 9290348715 bytes
> Data Written: 122127653133 bytes
> Interval 1 Stats:
> Block Size: 8b+ 32b+ 64b+
> No. of Reads: 2 0 1
> No. of Writes: 0 1 4
> Block Size: 128b+ 256b+ 512b+
> No. of Reads: 1 2 5
> No. of Writes: 3 0 0
> Block Size: 1024b+ 2048b+ 4096b+
> No. of Reads: 7 13 11
> No. of Writes: 6 0 0
> Block Size: 8192b+ 16384b+ 32768b+
> No. of Reads: 10 2 9
> No. of Writes: 0 0 0
> Block Size: 65536b+ 131072b+ 262144b+
> No. of Reads: 1 1 1
> No. of Writes: 0 0 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 7 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 45 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 177 RELEASEDIR
> 0.00 109.00 us 109.00 us 109.00 us 1 GETXATTR
> 0.00 110.00 us 110.00 us 110.00 us 1 SETATTR
> 0.00 52.78 us 33.00 us 81.00 us 9 INODELK
> 0.00 48.92 us 36.00 us 69.00 us 12 STATFS
> 0.01 146.80 us 129.00 us 171.00 us 5 UNLINK
> 0.01 47.89 us 31.00 us 63.00 us 18 LK
> 0.01 34.25 us 17.00 us 61.00 us 40 FLUSH
> 0.01 49.89 us 35.00 us 64.00 us 28 FINODELK
> 0.02 49.12 us 27.00 us 76.00 us 42 ENTRYLK
> 0.02 78.14 us 47.00 us 99.00 us 29 OPEN
> 0.08 58.01 us 34.00 us 100.00 us 177 OPENDIR
> 0.13 241.08 us 54.00 us 10214.00 us 66 READ
> 0.18 34.76 us 11.00 us 1950.00 us 649 ACCESS
> 0.21 906.82 us 94.00 us 21829.00 us 28 FXATTROP
> 0.29 9014.75 us 63.00 us 14563.00 us 4 READLINK
> 0.32 85.78 us 56.00 us 313.00 us 455 FSTAT
> 0.59 60.01 us 22.00 us 1443.00 us 1215 STAT
> 0.67 5906.21 us 70.00 us 56839.00 us 14 WRITE
> 1.02 7854.81 us 161.00 us 27303.00 us 16 CREATE
> 7.42 2008.99 us 361.00 us 2796.00 us 455 READDIRP
> 89.00 1052.36 us 37.00 us 132924.00 us 10418 LOOKUP
> Duration: 18 seconds
> Data Read: 1285736 bytes
> Data Written: 8465 bytes
> Brick: 192.168.10.90:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 2b+ 8b+ 16b+
> No. of Reads: 0 6 32
> No. of Writes: 1 1928 2604
> Block Size: 32b+ 64b+ 128b+
> No. of Reads: 21 112 169
> No. of Writes: 6983 12066 26008
> Block Size: 256b+ 512b+ 1024b+
> No. of Reads: 85 299 809
> No. of Writes: 743 18584 7209
> Block Size: 2048b+ 4096b+ 8192b+
> No. of Reads: 790 1418 2884
> No. of Writes: 1650 10647 16901
> Block Size: 16384b+ 32768b+ 65536b+
> No. of Reads: 4588 5856 32428
> No. of Writes: 2974 5758 3387
> Block Size: 131072b+
> No. of Reads: 455199
> No. of Writes: 4601
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 604780 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1410794 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 989700 RELEASEDIR
> 0.00 45.50 us 29.00 us 62.00 us 2 READDIR
> 0.00 58.00 us 44.00 us 77.00 us 4 GETXATTR
> 0.00 92.00 us 86.00 us 101.00 us 3 SETATTR
> 0.00 147.50 us 136.00 us 159.00 us 2 XATTROP
> 0.00 55.89 us 41.00 us 84.00 us 9 FSTAT
> 0.00 54.22 us 33.00 us 76.00 us 18 STATFS
> 0.00 46.04 us 30.00 us 60.00 us 25 LK
> 0.01 172.17 us 115.00 us 222.00 us 12 UNLINK
> 0.01 38.75 us 15.00 us 150.00 us 56 FLUSH
> 0.01 54.46 us 30.00 us 101.00 us 46 FINODELK
> 0.01 52.78 us 27.00 us 91.00 us 68 ENTRYLK
> 0.01 93.74 us 49.00 us 352.00 us 43 OPEN
> 0.01 204.50 us 150.00 us 511.00 us 22 CREATE
> 0.03 58.25 us 3.00 us 94.00 us 183 OPENDIR
> 0.03 247.33 us 73.00 us 5551.00 us 45 FXATTROP
> 0.11 35786.00 us 35786.00 us 35786.00 us 1 TRUNCATE
> 0.15 669.42 us 41.00 us 23631.00 us 76 READLINK
> 0.17 3501.75 us 60.00 us 27637.00 us 16 READ
> 0.34 5093.86 us 60.00 us 69136.00 us 22 WRITE
> 4.56 100248.87 us 25.00 us 1502989.00 us 15 INODELK
> 9.64 1930.07 us 30.00 us 3404.00 us 1646 READDIRP
> 84.89 1367.04 us 34.00 us 828308.00 us 20463 LOOKUP
> Duration: 76033 seconds
> Data Read: 63112532714 bytes
> Data Written: 1489661828 bytes
> Interval 1 Stats:
> Block Size: 32b+ 64b+ 128b+
> No. of Reads: 0 0 0
> No. of Writes: 1 4 3
> Block Size: 1024b+ 65536b+ 131072b+
> No. of Reads: 0 2 1
> No. of Writes: 6 0 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 9 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 45 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 177 RELEASEDIR
> 0.00 86.00 us 86.00 us 86.00 us 1 SETATTR
> 0.00 55.89 us 41.00 us 84.00 us 9 FSTAT
> 0.00 53.50 us 40.00 us 76.00 us 12 STATFS
> 0.00 160.00 us 130.00 us 184.00 us 5 UNLINK
> 0.00 46.83 us 30.00 us 60.00 us 18 LK
> 0.01 51.11 us 30.00 us 88.00 us 28 FINODELK
> 0.01 37.88 us 15.00 us 150.00 us 40 FLUSH
> 0.01 52.14 us 27.00 us 83.00 us 42 ENTRYLK
> 0.01 101.34 us 52.00 us 352.00 us 29 OPEN
> 0.02 209.44 us 150.00 us 511.00 us 16 CREATE
> 0.04 319.57 us 73.00 us 5551.00 us 28 FXATTROP
> 0.05 58.58 us 29.00 us 94.00 us 177 OPENDIR
> 0.23 669.42 us 41.00 us 23631.00 us 76 READLINK
> 0.26 3728.27 us 60.00 us 27637.00 us 15 READ
> 0.37 5742.93 us 60.00 us 69136.00 us 14 WRITE
> 6.88 167043.89 us 25.00 us 1502989.00 us 9 INODELK
> 10.95 1908.32 us 30.00 us 3404.00 us 1254 READDIRP
> 81.16 1701.62 us 34.00 us 828308.00 us 10421 LOOKUP
> Duration: 18 seconds
> Data Read: 310621 bytes
> Data Written: 8465 bytes
> Brick: 192.168.10.91:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 0 0 0
> No. of Writes: 3 3 6
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 13 173 58
> No. of Writes: 3370 5495 13458
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 194 280 500
> No. of Writes: 33703 34265 7020
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 1302 1888 3596
> No. of Writes: 26022 17086 12495
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 5031 13204 21838
> No. of Writes: 21889 32294 18289
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 73578 190508 2710663
> No. of Writes: 17017 13980 144272
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 817553 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1663792 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1438610 RELEASEDIR
> 0.00 85.00 us 85.00 us 85.00 us 1 READDIR
> 0.00 58.50 us 56.00 us 60.00 us 4 FSTAT
> 0.00 34.86 us 24.00 us 46.00 us 7 STAT
> 0.00 89.67 us 81.00 us 100.00 us 3 SETATTR
> 0.00 136.50 us 108.00 us 165.00 us 2 XATTROP
> 0.00 63.20 us 35.00 us 87.00 us 5 GETXATTR
> 0.00 54.73 us 39.00 us 66.00 us 15 INODELK
> 0.01 53.39 us 33.00 us 88.00 us 18 STATFS
> 0.01 58.00 us 32.00 us 259.00 us 25 LK
> 0.01 208.50 us 112.00 us 691.00 us 12 UNLINK
> 0.02 60.76 us 40.00 us 125.00 us 46 FINODELK
> 0.02 176.63 us 74.00 us 302.00 us 19 READ
> 0.02 52.51 us 23.00 us 97.00 us 68 ENTRYLK
> 0.02 48.00 us 18.00 us 579.00 us 81 FLUSH
> 0.03 206.95 us 163.00 us 497.00 us 22 CREATE
> 0.04 85.16 us 51.00 us 168.00 us 70 OPEN
> 0.04 145.02 us 92.00 us 344.00 us 45 FXATTROP
> 0.07 59.91 us 3.00 us 145.00 us 184 OPENDIR
> 0.13 1024.41 us 61.00 us 12251.00 us 22 WRITE
> 0.14 444.38 us 44.00 us 11184.00 us 52 READLINK
> 0.27 45104.00 us 45104.00 us 45104.00 us 1 TRUNCATE
> 20.33 1895.33 us 27.00 us 6083.00 us 1819 READDIRP
> 78.84 645.59 us 35.00 us 209082.00 us 20706 LOOKUP
> Duration: 269212 seconds
> Data Read: 376538217873 bytes
> Data Written: 21968843126 bytes
> Interval 1 Stats:
> Block Size: 32b+ 64b+ 128b+
> No. of Reads: 0 0 0
> No. of Writes: 1 4 3
> Block Size: 1024b+ 32768b+ 131072b+
> No. of Reads: 0 1 10
> No. of Writes: 6 0 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 8 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 45 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 178 RELEASEDIR
> 0.00 81.00 us 81.00 us 81.00 us 1 SETATTR
> 0.00 85.00 us 85.00 us 85.00 us 1 READDIR
> 0.00 63.33 us 35.00 us 87.00 us 3 GETXATTR
> 0.00 58.50 us 56.00 us 60.00 us 4 FSTAT
> 0.00 34.86 us 24.00 us 46.00 us 7 STAT
> 0.00 54.00 us 47.00 us 66.00 us 9 INODELK
> 0.01 51.25 us 33.00 us 78.00 us 12 STATFS
> 0.01 156.20 us 112.00 us 202.00 us 5 UNLINK
> 0.01 61.33 us 34.00 us 259.00 us 18 LK
> 0.02 61.64 us 40.00 us 125.00 us 28 FINODELK
> 0.02 54.65 us 21.00 us 579.00 us 40 FLUSH
> 0.02 54.12 us 28.00 us 97.00 us 42 ENTRYLK
> 0.02 161.93 us 74.00 us 302.00 us 15 READ
> 0.02 83.97 us 68.00 us 107.00 us 29 OPEN
> 0.03 214.88 us 163.00 us 497.00 us 16 CREATE
> 0.04 144.21 us 92.00 us 344.00 us 28 FXATTROP
> 0.10 60.01 us 3.00 us 145.00 us 178 OPENDIR
> 0.10 316.62 us 44.00 us 8512.00 us 34 READLINK
> 0.15 1121.14 us 61.00 us 12251.00 us 14 WRITE
> 32.97 1895.33 us 27.00 us 6083.00 us 1819 READDIRP
> 66.46 667.09 us 35.00 us 67882.00 us 10418 LOOKUP
> Duration: 18 seconds
> Data Read: 1352163 bytes
> Data Written: 8465 bytes
> Profile 2:
> Brick: 192.168.10.92:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 0 0 0
> No. of Writes: 133 6 37
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 28 93 31
> No. of Writes: 4577 6495 22231
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 219 218 254
> No. of Writes: 37641 41723 20519
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 484 981 696
> No. of Writes: 39094 42613 203748
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 1079 1802 2840
> No. of Writes: 220154 229725 215083
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 4227 14944 57816
> No. of Writes: 60938 61967 793400
> Block Size: 262144b+ 524288b+ 1048576b+
> No. of Reads: 6 5 2
> No. of Writes: 5 0 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 1835089 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 2639439 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1440418 RELEASEDIR
> 0.00 53.50 us 43.00 us 64.00 us 2 READDIR
> 0.00 153.50 us 142.00 us 165.00 us 2 XATTROP
> 0.00 113.00 us 97.00 us 144.00 us 4 SETATTR
> 0.00 114.60 us 49.00 us 339.00 us 10 GETXATTR
> 0.00 55.00 us 33.00 us 100.00 us 24 INODELK
> 0.00 48.85 us 25.00 us 81.00 us 71 STATFS
> 0.01 49.47 us 31.00 us 76.00 us 92 LK
> 0.01 36.07 us 16.00 us 68.00 us 180 FLUSH
> 0.01 52.45 us 32.00 us 88.00 us 159 FINODELK
> 0.02 79.16 us 47.00 us 112.00 us 137 OPEN
> 0.02 51.06 us 26.00 us 95.00 us 216 ENTRYLK
> 0.03 24814.00 us 24814.00 us 24814.00 us 1 TRUNCATE
> 0.06 271.36 us 86.00 us 21829.00 us 158 FXATTROP
> 0.07 59.02 us 2.00 us 100.00 us 881 OPENDIR
> 0.07 5910.00 us 56.00 us 16873.00 us 9 READLINK
> 0.11 33.81 us 11.00 us 1950.00 us 2348 ACCESS
> 0.17 4181.53 us 129.00 us 41541.00 us 30 UNLINK
> 0.31 2788.77 us 59.00 us 56839.00 us 79 WRITE
> 0.37 2096.99 us 54.00 us 61234.00 us 125 READ
> 0.38 84.58 us 53.00 us 313.00 us 3191 FSTAT
> 0.50 61.21 us 20.00 us 14904.00 us 5873 STAT
> 0.76 7023.69 us 145.00 us 102353.00 us 78 CREATE
> 8.90 2002.80 us 361.00 us 3545.00 us 3191 READDIRP
> 88.19 1112.81 us 34.00 us 307904.00 us 56876 LOOKUP
> Duration: 338222 seconds
> Data Read: 9292898491 bytes
> Data Written: 122127690284 bytes
> Interval 4 Stats:
> Block Size: 16b+ 32b+ 64b+
> No. of Reads: 0 0 2
> No. of Writes: 1 3 12
> Block Size: 128b+ 1024b+ 2048b+
> No. of Reads: 2 3 10
> No. of Writes: 8 16 0
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 4 2 1
> No. of Writes: 0 0 0
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 2 3 1
> No. of Writes: 0 0 0
> Block Size: 262144b+
> No. of Reads: 2
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 21 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 76 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 407 RELEASEDIR
> 0.00 144.00 us 144.00 us 144.00 us 1 SETATTR
> 0.00 63.33 us 56.00 us 69.00 us 3 READLINK
> 0.00 226.00 us 113.00 us 339.00 us 2 GETXATTR
> 0.00 54.89 us 39.00 us 70.00 us 9 INODELK
> 0.01 48.03 us 25.00 us 72.00 us 37 STATFS
> 0.01 36.75 us 16.00 us 65.00 us 59 FLUSH
> 0.01 50.44 us 33.00 us 76.00 us 48 LK
> 0.01 83.73 us 55.00 us 111.00 us 37 OPEN
> 0.01 53.79 us 32.00 us 80.00 us 80 FINODELK
> 0.02 50.55 us 26.00 us 83.00 us 104 ENTRYLK
> 0.03 135.21 us 86.00 us 228.00 us 80 FXATTROP
> 0.07 58.75 us 35.00 us 98.00 us 407 OPENDIR
> 0.10 1044.22 us 55.00 us 30268.00 us 32 READ
> 0.14 33.74 us 13.00 us 1044.00 us 1384 ACCESS
> 0.19 5076.42 us 141.00 us 41541.00 us 12 UNLINK
> 0.36 2860.70 us 59.00 us 29520.00 us 40 WRITE
> 0.36 84.16 us 53.00 us 182.00 us 1369 FSTAT
> 0.65 5225.18 us 145.00 us 28783.00 us 40 CREATE
> 0.73 61.98 us 20.00 us 14904.00 us 3771 STAT
> 8.53 2009.18 us 518.00 us 2532.00 us 1369 READDIRP
> 88.77 1156.14 us 36.00 us 232288.00 us 24747 LOOKUP
> Duration: 48 seconds
> Data Read: 1424095 bytes
> Data Written: 22682 bytes
> Brick: 192.168.10.91:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+ 4b+
> No. of Reads: 0 0 0
> No. of Writes: 3 3 6
> Block Size: 8b+ 16b+ 32b+
> No. of Reads: 13 173 58
> No. of Writes: 3371 5496 13461
> Block Size: 64b+ 128b+ 256b+
> No. of Reads: 194 280 500
> No. of Writes: 33717 34277 7020
> Block Size: 512b+ 1024b+ 2048b+
> No. of Reads: 1302 1888 3596
> No. of Writes: 26022 17110 12496
> Block Size: 4096b+ 8192b+ 16384b+
> No. of Reads: 5031 13204 21838
> No. of Writes: 21889 32294 18289
> Block Size: 32768b+ 65536b+ 131072b+
> No. of Reads: 73579 190511 2710676
> No. of Writes: 17017 13980 144272
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 817592 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1663910 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1439269 RELEASEDIR
> 0.00 61.67 us 39.00 us 85.00 us 3 READDIR
> 0.00 136.50 us 108.00 us 165.00 us 2 XATTROP
> 0.00 91.00 us 81.00 us 100.00 us 4 SETATTR
> 0.00 58.86 us 35.00 us 87.00 us 7 GETXATTR
> 0.00 60.57 us 55.00 us 69.00 us 7 FSTAT
> 0.00 35.53 us 24.00 us 53.00 us 15 STAT
> 0.00 57.12 us 39.00 us 72.00 us 24 INODELK
> 0.01 51.25 us 33.00 us 90.00 us 71 STATFS
> 0.01 52.09 us 32.00 us 259.00 us 92 LK
> 0.01 175.87 us 112.00 us 691.00 us 30 UNLINK
> 0.01 41.84 us 16.00 us 579.00 us 175 FLUSH
> 0.02 56.23 us 25.00 us 125.00 us 158 FINODELK
> 0.02 51.59 us 23.00 us 97.00 us 216 ENTRYLK
> 0.02 85.10 us 51.00 us 168.00 us 133 OPEN
> 0.03 200.55 us 150.00 us 554.00 us 78 CREATE
> 0.06 163.03 us 38.00 us 11184.00 us 204 READLINK
> 0.07 1022.60 us 67.00 us 24446.00 us 40 READ
> 0.08 45104.00 us 45104.00 us 45104.00 us 1 TRUNCATE
> 0.09 59.52 us 3.00 us 145.00 us 843 OPENDIR
> 0.13 936.73 us 56.00 us 12251.00 us 78 WRITE
> 0.19 687.17 us 88.00 us 85925.00 us 157 FXATTROP
> 22.66 1881.29 us 23.00 us 6083.00 us 6698 READDIRP
> 76.59 754.63 us 33.00 us 209082.00 us 56449 LOOKUP
> Duration: 269283 seconds
> Data Read: 376540270509 bytes
> Data Written: 21968880277 bytes
> Interval 4 Stats:
> Block Size: 16b+ 32b+ 64b+
> No. of Reads: 0 0 0
> No. of Writes: 1 3 12
> Block Size: 128b+ 1024b+ 32768b+
> No. of Reads: 0 0 1
> No. of Writes: 8 16 0
> Block Size: 65536b+ 131072b+
> No. of Reads: 2 9
> No. of Writes: 0 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 36 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 76 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 408 RELEASEDIR
> 0.00 95.00 us 95.00 us 95.00 us 1 SETATTR
> 0.00 48.00 us 39.00 us 57.00 us 2 GETXATTR
> 0.00 50.00 us 39.00 us 61.00 us 2 READDIR
> 0.00 35.50 us 27.00 us 53.00 us 4 STAT
> 0.00 63.33 us 55.00 us 69.00 us 3 FSTAT
> 0.00 61.11 us 50.00 us 72.00 us 9 INODELK
> 0.01 50.41 us 34.00 us 90.00 us 37 STATFS
> 0.01 160.08 us 125.00 us 213.00 us 12 UNLINK
> 0.01 37.19 us 18.00 us 71.00 us 59 FLUSH
> 0.01 48.46 us 32.00 us 78.00 us 48 LK
> 0.01 84.92 us 57.00 us 118.00 us 37 OPEN
> 0.02 54.89 us 25.00 us 89.00 us 80 FINODELK
> 0.02 50.57 us 27.00 us 74.00 us 104 ENTRYLK
> 0.03 66.88 us 38.00 us 100.00 us 113 READLINK
> 0.03 196.90 us 150.00 us 554.00 us 40 CREATE
> 0.04 139.06 us 88.00 us 233.00 us 80 FXATTROP
> 0.10 58.95 us 8.00 us 93.00 us 408 OPENDIR
> 0.11 1675.56 us 67.00 us 24446.00 us 16 READ
> 0.17 1066.62 us 62.00 us 8506.00 us 40 WRITE
> 23.14 1889.02 us 23.00 us 3775.00 us 3073 READDIRP
> 76.29 773.40 us 33.00 us 135928.00 us 24746 LOOKUP
> Duration: 48 seconds
> Data Read: 1446611 bytes
> Data Written: 22682 bytes
> Brick: 192.168.10.90:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 2b+ 8b+ 16b+
> No. of Reads: 0 6 32
> No. of Writes: 1 1929 2605
> Block Size: 32b+ 64b+ 128b+
> No. of Reads: 21 112 169
> No. of Writes: 6986 12080 26020
> Block Size: 256b+ 512b+ 1024b+
> No. of Reads: 85 299 809
> No. of Writes: 743 18584 7233
> Block Size: 2048b+ 4096b+ 8192b+
> No. of Reads: 790 1419 2885
> No. of Writes: 1651 10647 16901
> Block Size: 16384b+ 32768b+ 65536b+
> No. of Reads: 4588 5858 32434
> No. of Writes: 2974 5758 3387
> Block Size: 131072b+
> No. of Reads: 455218
> No. of Writes: 4601
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 604813 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1410912 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 990360 RELEASEDIR
> 0.00 44.33 us 16.00 us 62.00 us 6 READDIR
> 0.00 147.50 us 136.00 us 159.00 us 2 XATTROP
> 0.00 100.25 us 86.00 us 125.00 us 4 SETATTR
> 0.00 60.11 us 39.00 us 92.00 us 9 GETXATTR
> 0.00 60.19 us 41.00 us 84.00 us 16 FSTAT
> 0.00 45.50 us 17.00 us 182.00 us 38 STAT
> 0.00 51.37 us 31.00 us 81.00 us 71 STATFS
> 0.00 49.05 us 30.00 us 79.00 us 92 LK
> 0.00 168.87 us 115.00 us 287.00 us 30 UNLINK
> 0.01 38.74 us 15.00 us 150.00 us 150 FLUSH
> 0.01 54.76 us 30.00 us 236.00 us 158 FINODELK
> 0.01 87.17 us 49.00 us 352.00 us 106 OPEN
> 0.01 52.56 us 27.00 us 91.00 us 216 ENTRYLK
> 0.02 174.25 us 73.00 us 5551.00 us 157 FXATTROP
> 0.03 35786.00 us 35786.00 us 35786.00 us 1 TRUNCATE
> 0.05 62.12 us 3.00 us 1628.00 us 843 OPENDIR
> 0.10 685.58 us 41.00 us 51432.00 us 157 READLINK
> 0.36 6483.54 us 59.00 us 47225.00 us 63 READ
> 0.58 8304.77 us 60.00 us 185503.00 us 78 WRITE
> 0.67 9646.06 us 150.00 us 686579.00 us 78 CREATE
> 1.82 85338.88 us 25.00 us 1502989.00 us 24 INODELK
> 14.62 1920.48 us 23.00 us 9822.00 us 8567 READDIRP
> 81.71 1635.58 us 30.00 us 828308.00 us 56208 LOOKUP
> Duration: 76104 seconds
> Data Read: 63115678019 bytes
> Data Written: 1489698979 bytes
> Interval 4 Stats:
> Block Size: 16b+ 32b+ 64b+
> No. of Reads: 0 0 0
> No. of Writes: 1 3 12
> Block Size: 128b+ 1024b+ 4096b+
> No. of Reads: 0 0 1
> No. of Writes: 8 16 0
> Block Size: 8192b+ 32768b+ 65536b+
> No. of Reads: 1 1 6
> No. of Writes: 0 0 0
> Block Size: 131072b+
> No. of Reads: 10
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
> --------- ----------- ----------- ----------- ------------ ----
> 0.00 0.00 us 0.00 us 0.00 us 29 FORGET
> 0.00 0.00 us 0.00 us 0.00 us 76 RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 409 RELEASEDIR
> 0.00 125.00 us 125.00 us 125.00 us 1 SETATTR
> 0.00 43.75 us 16.00 us 60.00 us 4 READDIR
> 0.00 61.80 us 39.00 us 92.00 us 5 GETXATTR
> 0.00 65.71 us 44.00 us 76.00 us 7 FSTAT
> 0.00 48.50 us 24.00 us 182.00 us 28 STAT
> 0.00 50.08 us 31.00 us 70.00 us 37 STATFS
> 0.00 165.25 us 119.00 us 238.00 us 12 UNLINK
> 0.00 39.66 us 19.00 us 62.00 us 59 FLUSH
> 0.00 51.12 us 33.00 us 79.00 us 48 LK
> 0.01 82.78 us 64.00 us 104.00 us 37 OPEN
> 0.01 56.45 us 30.00 us 236.00 us 80 FINODELK
> 0.01 52.17 us 28.00 us 84.00 us 104 ENTRYLK
> 0.02 154.12 us 88.00 us 1577.00 us 80 FXATTROP
> 0.05 65.00 us 3.00 us 1628.00 us 409 OPENDIR
> 0.09 909.64 us 43.00 us 51432.00 us 61 READLINK
> 0.60 9748.22 us 59.00 us 47225.00 us 36 READ
> 0.86 12688.17 us 60.00 us 185503.00 us 40 WRITE
> 0.92 60488.89 us 50.00 us 543898.00 us 9 INODELK
> 1.26 18623.40 us 161.00 us 686579.00 us 40 CREATE
> 19.51 1951.60 us 25.00 us 9822.00 us 5889 READDIRP
> 76.64 1823.96 us 30.00 us 318395.00 us 24747 LOOKUP
> Duration: 48 seconds
> Data Read: 1931486 bytes
> Data Written: 22682 bytes
> Again, any assistance on this issue would be much appreciated.

> Thank you.

> On Sat, Dec 20, 2014 at 11:58 PM, Kyle Harris < kyle.harris98 at gmail.com >
> wrote:

> > Hello All,
> 

> > I have been experiencing difficulty with high process utilization on the
> > glusterfs and glusterfsd processes. I thought I had it tracked down to
> > Apache looking for files on the cluster drive that don't exist. I have been
> > working to clean that up and now I have discovered another problem.
> 

> > When I reboot one of the nodes in my 3-node replicated cluster, as soon as
> > it
> > comes back up I see a bunch of entries such as follows:
> 

> > [2014-12-21 05:33:49.873989] I [dict.c:370:dict_get] (-->
> > /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fe3bd0d2420] (-->
> > /usr/lib64/libglusterfs.so.0(dict_get+0x4f)[0x7fe3bd0cbbdf] (-->
> > /usr/lib64/glusterfs/3.6.1/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e1)[0x7fe3aad3ada1]
> > (-->
> > /usr/lib64/glusterfs/3.6.1/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113)[0x7fe3aaf524e3]
> > (-->
> > /usr/lib64/glusterfs/3.6.1/xlator/performance/md-cache.so(mdc_lookup+0x300)[0x7fe3ab167310]
> > ))))) 0-dict: !this || key=system.posix_acl_access
> 

> > While this is occurring, the process utilization for glusterfs and
> > glusterfsd
> > go sky high. Eventually they calm down on their own but the servers are
> > virtually unusable until they do. Note this is on Centos 6.6 with Gluster
> > 3.6.1 and kernel 2.6.32 using a fuse mount with each of the 3 servers
> > having
> > the client mounted for hosting web sites via Apache. There are many small
> > files and some are php. I have not done any tuning to the default gluster
> > parameters.
> 

> > Can someone tell me why this occurs after one of the nodes is rebooted and
> > perhaps lend some guidance on what can be done to resolve this issue?
> 

> > Thank you.
> 

> > --
> 
> > Kyle
> 

> --
> Kyle A. Harris
> Kyle at TheHarrisHome.com
> 615-364-6752

> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20141221/fabac4bf/attachment.html>