[Gluster-users] glusterfs and glusterfsd process utilization
Kyle Harris
kyle.harris98 at gmail.com
Mon Dec 22 05:00:14 UTC 2014
Hi Krutika,
Thank you so much for the response. I'm starting to get a bit desperate :-)
I'm not sure which file is the brick log but the client log (I assume that
is the one that start with mnt-) is far too large to include. If you can
tell me what it is, perhaps we can arrange to get it to you using some
other method?
Anyway, I primarily see the following messages in what I believe to be the
client log.
The first is as mentioned earlier and repeated again here:
[2014-12-21 05:33:49.873989] I [dict.c:370:dict_get] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fe3bd0d2420] (-->
/usr/lib64/libglusterfs.so.0(dict_get+0x4f)[0x7fe3bd0cbbdf] (-->
/usr/lib64/glusterfs/3.6.1/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e1)[0x7fe3aad3ada1]
(-->
/usr/lib64/glusterfs/3.6.1/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113)[0x7fe3aaf524e3]
(-->
/usr/lib64/glusterfs/3.6.1/xlator/performance/md-cache.so(mdc_lookup+0x300)[0x7fe3ab167310]
))))) 0-dict: !this || key=system.posix_acl_access
This seems to be the one I see the most when it is starting up and having
this issue but I can't be certain. By the way, can you tell me what that
means?
The second entry I see often in the log files is in regards to missing
files such as follows:
[2014-12-22 02:33:32.062741] I [dht-common.c:1822:dht_lookup_cbk]
0-gv0-dht: Entry /html/some_dir/some_file/asdf.php missing on subvol
gv0-replicate-0
I thought these might be causing my problems but I see them all the time,
including right now and process utilization on the servers in this cluster
is currently way down next to nothing so I no longer think that is my
problem. I also checked while to see what, if any files needed healing
during the last time this issue occurred using the following command:
gluster volume heal gv0 info
It showed no files in need of healing.
In the log file etc-glusterfs-glusterd.vol.log I see many of the following
entries:
[2014-12-21 19:13:54.137263] W [socket.c:611:__socket_rwv] 0-management:
readv on /var/run/095271447d0b71369a479e6211bbe6e5.socket failed (Invalid
argument)
I have a bricks dir with the file export-brick1.log and it is empty.
gluster volume info as requested:
Volume Name: gv0
Type: Replicate
Volume ID: 3565c78b-2d23-42e7-8cd2-1724fe671610
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.10.90:/export/brick1
Brick2: 192.168.10.91:/export/brick1
Brick3: 192.168.10.92:/export/brick1
Options Reconfigured:
cluster.data-self-heal-algorithm: full
cluster.entry-self-heal: off
I also found the following in a log file and perhaps it will help:
Final graph:
+------------------------------------------------------------------------------+
1: volume gv0-client-0
2: type protocol/client
3: option ping-timeout 42
4: option remote-host 192.168.10.90
5: option remote-subvolume /export/brick1
6: option transport-type socket
7: option username {removed}
8: option password {removed}
9: option send-gids true
10: end-volume
11:
12: volume gv0-client-1
13: type protocol/client
14: option ping-timeout 42
15: option remote-host 192.168.10.91
16: option remote-subvolume /export/brick1
17: option transport-type socket
18: option username {removed}
19: option password {removed}
20: option send-gids true
21: end-volume
22:
23: volume gv0-client-2
24: type protocol/client
25: option ping-timeout 42
26: option remote-host 192.168.10.92
27: option remote-subvolume /export/brick1
28: option transport-type socket
29: option username {removed}
30: option password {removed}
31: option send-gids true
32: end-volume
33:
34: volume gv0-replicate-0
35: type cluster/replicate
36: option entry-self-heal off
37: option data-self-heal-algorithm full
38: subvolumes gv0-client-0 gv0-client-1 gv0-client-2
39: end-volume
40:
41: volume gv0-dht
42: type cluster/distribute
43: subvolumes gv0-replicate-0
44: end-volume
45:
46: volume gv0-write-behind
47: type performance/write-behind
48: subvolumes gv0-dht
49: end-volume
50:
51: volume gv0
52: type debug/io-stats
53: option latency-measurement off
54: option count-fop-hits off
55: subvolumes gv0-write-behind
56: end-volume
57:
58: volume nfs-server
59: type nfs/server
60: option rpc-auth.auth-glusterfs on
61: option rpc-auth.auth-unix on
62: option rpc-auth.auth-null on
63: option rpc-auth.ports.insecure on
64: option rpc-auth-allow-insecure on
65: option transport-type socket
66: option transport.socket.listen-port 2049
67: option nfs.dynamic-volumes on
68: option nfs.nlm on
69: option nfs.drc off
70: option rpc-auth.addr.gv0.allow *
71: option nfs3.gv0.volume-id 3565c78b-2d23-42e7-8cd2-1724fe671610
72: subvolumes gv0
73: end-volume
74:
With regard to the average number of directory entries, I don't know for
sure but I'm sure some of them get pretty deep.
Finally, I do not have quota enabled.
Again, thank you very much and regards,
Kyle
On Sun, Dec 21, 2014 at 10:19 PM, Krutika Dhananjay <kdhananj at redhat.com>
wrote:
> Hi,
>
> Could you provide the following pieces of information?
>
> 1. The brick and client logs
> 2. output of `gluster volume info`
> 3. average number of entries under directories
>
> Just to be clear, you don't have quota enabled on the volume, do you?
>
> -Krutika
> ------------------------------
>
> *From: *"Kyle Harris" <kyle.harris98 at gmail.com>
> *To: *gluster-users at gluster.org
> *Sent: *Monday, December 22, 2014 8:27:09 AM
> *Subject: *Re: [Gluster-users] glusterfs and glusterfsd process
> utilization
>
> I haven't heard anything from anyone re this post yet so I have a bit more
> info to add. The last time I posted about extremely high process
> utilization for glusterfs and glusterfsd, I was asked to see if I could
> capture the output of a profile. I did so and here is the output of 2
> captures (they appear very similar). Perhaps this will assist. Note that
> previously I was mounting with the fuse client. This time I tried NFS.
>
> Profile Output 1:
> Brick: 192.168.10.92:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 133 6
> 37
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 28 93
> 31
> No. of Writes: 4576 6494
> 22228
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 217 216
> 254
> No. of Writes: 37627 41711
> 20519
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 484 978
> 684
> No. of Writes: 39094 42589
> 203747
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 1073 1800
> 2838
> No. of Writes: 220154 229725
> 215083
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 4224 14941
> 57812
> No. of Writes: 60938 61967
> 793400
>
> Block Size: 262144b+ 524288b+
> 1048576b+
> No. of Reads: 4 4
> 2
> No. of Writes: 5 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 1835064
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 2639321
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1439759
> RELEASEDIR
> 0.00 153.50 us 142.00 us 165.00 us 2
> XATTROP
> 0.00 102.67 us 97.00 us 110.00 us 3
> SETATTR
> 0.00 93.20 us 53.00 us 129.00 us 5
> GETXATTR
> 0.00 55.07 us 33.00 us 100.00 us 15
> INODELK
> 0.00 46.61 us 35.00 us 69.00 us 18
> STATFS
> 0.01 48.40 us 31.00 us 64.00 us 25
> LK
> 0.01 50.72 us 32.00 us 88.00 us 47
> FINODELK
> 0.01 35.49 us 17.00 us 68.00 us 86
> FLUSH
> 0.02 50.59 us 27.00 us 76.00 us 68
> ENTRYLK
> 0.03 75.72 us 47.00 us 112.00 us 74
> OPEN
> 0.06 58.79 us 32.00 us 100.00 us 222
> OPENDIR
> 0.08 1535.92 us 129.00 us 16583.00 us 12
> UNLINK
> 0.11 24814.00 us 24814.00 us 24814.00 us 1
> TRUNCATE
> 0.12 34.25 us 11.00 us 1950.00 us 784
> ACCESS
> 0.13 603.50 us 90.00 us 21829.00 us 46
> FXATTROP
> 0.16 7225.40 us 63.00 us 14563.00 us 5
> READLINK
> 0.17 437.90 us 54.00 us 19042.00 us 83
> READ
> 0.36 85.92 us 56.00 us 313.00 us 912
> FSTAT
> 0.42 59.86 us 22.00 us 1443.00 us 1533
> STAT
> 0.45 4318.65 us 70.00 us 56839.00 us 23
> WRITE
> 0.84 8405.18 us 161.00 us 37688.00 us 22
> CREATE
> 8.33 2005.77 us 361.00 us 2796.00 us 912
> READDIRP
> 88.68 921.10 us 34.00 us 159393.00 us 21137
> LOOKUP
>
> Duration: 338151 seconds
> Data Read: 9290348715 bytes
> Data Written: 122127653133 bytes
>
> Interval 1 Stats:
> Block Size: 8b+ 32b+
> 64b+
> No. of Reads: 2 0
> 1
> No. of Writes: 0 1
> 4
>
> Block Size: 128b+ 256b+
> 512b+
> No. of Reads: 1 2
> 5
> No. of Writes: 3 0
> 0
>
> Block Size: 1024b+ 2048b+
> 4096b+
> No. of Reads: 7 13
> 11
> No. of Writes: 6 0
> 0
>
> Block Size: 8192b+ 16384b+
> 32768b+
> No. of Reads: 10 2
> 9
> No. of Writes: 0 0
> 0
>
> Block Size: 65536b+ 131072b+
> 262144b+
> No. of Reads: 1 1
> 1
> No. of Writes: 0 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 7
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 45
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 177
> RELEASEDIR
> 0.00 109.00 us 109.00 us 109.00 us 1
> GETXATTR
> 0.00 110.00 us 110.00 us 110.00 us 1
> SETATTR
> 0.00 52.78 us 33.00 us 81.00 us 9
> INODELK
> 0.00 48.92 us 36.00 us 69.00 us 12
> STATFS
> 0.01 146.80 us 129.00 us 171.00 us 5
> UNLINK
> 0.01 47.89 us 31.00 us 63.00 us 18
> LK
> 0.01 34.25 us 17.00 us 61.00 us 40
> FLUSH
> 0.01 49.89 us 35.00 us 64.00 us 28
> FINODELK
> 0.02 49.12 us 27.00 us 76.00 us 42
> ENTRYLK
> 0.02 78.14 us 47.00 us 99.00 us 29
> OPEN
> 0.08 58.01 us 34.00 us 100.00 us 177
> OPENDIR
> 0.13 241.08 us 54.00 us 10214.00 us 66
> READ
> 0.18 34.76 us 11.00 us 1950.00 us 649
> ACCESS
> 0.21 906.82 us 94.00 us 21829.00 us 28
> FXATTROP
> 0.29 9014.75 us 63.00 us 14563.00 us 4
> READLINK
> 0.32 85.78 us 56.00 us 313.00 us 455
> FSTAT
> 0.59 60.01 us 22.00 us 1443.00 us 1215
> STAT
> 0.67 5906.21 us 70.00 us 56839.00 us 14
> WRITE
> 1.02 7854.81 us 161.00 us 27303.00 us 16
> CREATE
> 7.42 2008.99 us 361.00 us 2796.00 us 455
> READDIRP
> 89.00 1052.36 us 37.00 us 132924.00 us 10418
> LOOKUP
>
> Duration: 18 seconds
> Data Read: 1285736 bytes
> Data Written: 8465 bytes
>
> Brick: 192.168.10.90:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 2b+ 8b+
> 16b+
> No. of Reads: 0 6
> 32
> No. of Writes: 1 1928
> 2604
>
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 21 112
> 169
> No. of Writes: 6983 12066
> 26008
>
> Block Size: 256b+ 512b+
> 1024b+
> No. of Reads: 85 299
> 809
> No. of Writes: 743 18584
> 7209
>
> Block Size: 2048b+ 4096b+
> 8192b+
> No. of Reads: 790 1418
> 2884
> No. of Writes: 1650 10647
> 16901
>
> Block Size: 16384b+ 32768b+
> 65536b+
> No. of Reads: 4588 5856
> 32428
> No. of Writes: 2974 5758
> 3387
>
> Block Size: 131072b+
> No. of Reads: 455199
> No. of Writes: 4601
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 604780
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1410794
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 989700
> RELEASEDIR
> 0.00 45.50 us 29.00 us 62.00 us 2
> READDIR
> 0.00 58.00 us 44.00 us 77.00 us 4
> GETXATTR
> 0.00 92.00 us 86.00 us 101.00 us 3
> SETATTR
> 0.00 147.50 us 136.00 us 159.00 us 2
> XATTROP
> 0.00 55.89 us 41.00 us 84.00 us 9
> FSTAT
> 0.00 54.22 us 33.00 us 76.00 us 18
> STATFS
> 0.00 46.04 us 30.00 us 60.00 us 25
> LK
> 0.01 172.17 us 115.00 us 222.00 us 12
> UNLINK
> 0.01 38.75 us 15.00 us 150.00 us 56
> FLUSH
> 0.01 54.46 us 30.00 us 101.00 us 46
> FINODELK
> 0.01 52.78 us 27.00 us 91.00 us 68
> ENTRYLK
> 0.01 93.74 us 49.00 us 352.00 us 43
> OPEN
> 0.01 204.50 us 150.00 us 511.00 us 22
> CREATE
> 0.03 58.25 us 3.00 us 94.00 us 183
> OPENDIR
> 0.03 247.33 us 73.00 us 5551.00 us 45
> FXATTROP
> 0.11 35786.00 us 35786.00 us 35786.00 us 1
> TRUNCATE
> 0.15 669.42 us 41.00 us 23631.00 us 76
> READLINK
> 0.17 3501.75 us 60.00 us 27637.00 us 16
> READ
> 0.34 5093.86 us 60.00 us 69136.00 us 22
> WRITE
> 4.56 100248.87 us 25.00 us 1502989.00 us 15
> INODELK
> 9.64 1930.07 us 30.00 us 3404.00 us 1646
> READDIRP
> 84.89 1367.04 us 34.00 us 828308.00 us 20463
> LOOKUP
>
> Duration: 76033 seconds
> Data Read: 63112532714 bytes
> Data Written: 1489661828 bytes
>
> Interval 1 Stats:
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 0 0
> 0
> No. of Writes: 1 4
> 3
>
> Block Size: 1024b+ 65536b+
> 131072b+
> No. of Reads: 0 2
> 1
> No. of Writes: 6 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 9
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 45
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 177
> RELEASEDIR
> 0.00 86.00 us 86.00 us 86.00 us 1
> SETATTR
> 0.00 55.89 us 41.00 us 84.00 us 9
> FSTAT
> 0.00 53.50 us 40.00 us 76.00 us 12
> STATFS
> 0.00 160.00 us 130.00 us 184.00 us 5
> UNLINK
> 0.00 46.83 us 30.00 us 60.00 us 18
> LK
> 0.01 51.11 us 30.00 us 88.00 us 28
> FINODELK
> 0.01 37.88 us 15.00 us 150.00 us 40
> FLUSH
> 0.01 52.14 us 27.00 us 83.00 us 42
> ENTRYLK
> 0.01 101.34 us 52.00 us 352.00 us 29
> OPEN
> 0.02 209.44 us 150.00 us 511.00 us 16
> CREATE
> 0.04 319.57 us 73.00 us 5551.00 us 28
> FXATTROP
> 0.05 58.58 us 29.00 us 94.00 us 177
> OPENDIR
> 0.23 669.42 us 41.00 us 23631.00 us 76
> READLINK
> 0.26 3728.27 us 60.00 us 27637.00 us 15
> READ
> 0.37 5742.93 us 60.00 us 69136.00 us 14
> WRITE
> 6.88 167043.89 us 25.00 us 1502989.00 us 9
> INODELK
> 10.95 1908.32 us 30.00 us 3404.00 us 1254
> READDIRP
> 81.16 1701.62 us 34.00 us 828308.00 us 10421
> LOOKUP
>
> Duration: 18 seconds
> Data Read: 310621 bytes
> Data Written: 8465 bytes
>
> Brick: 192.168.10.91:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 3 3
> 6
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 13 173
> 58
> No. of Writes: 3370 5495
> 13458
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 194 280
> 500
> No. of Writes: 33703 34265
> 7020
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 1302 1888
> 3596
> No. of Writes: 26022 17086
> 12495
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 5031 13204
> 21838
> No. of Writes: 21889 32294
> 18289
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 73578 190508
> 2710663
> No. of Writes: 17017 13980
> 144272
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 817553
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1663792
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1438610
> RELEASEDIR
> 0.00 85.00 us 85.00 us 85.00 us 1
> READDIR
> 0.00 58.50 us 56.00 us 60.00 us 4
> FSTAT
> 0.00 34.86 us 24.00 us 46.00 us 7
> STAT
> 0.00 89.67 us 81.00 us 100.00 us 3
> SETATTR
> 0.00 136.50 us 108.00 us 165.00 us 2
> XATTROP
> 0.00 63.20 us 35.00 us 87.00 us 5
> GETXATTR
> 0.00 54.73 us 39.00 us 66.00 us 15
> INODELK
> 0.01 53.39 us 33.00 us 88.00 us 18
> STATFS
> 0.01 58.00 us 32.00 us 259.00 us 25
> LK
> 0.01 208.50 us 112.00 us 691.00 us 12
> UNLINK
> 0.02 60.76 us 40.00 us 125.00 us 46
> FINODELK
> 0.02 176.63 us 74.00 us 302.00 us 19
> READ
> 0.02 52.51 us 23.00 us 97.00 us 68
> ENTRYLK
> 0.02 48.00 us 18.00 us 579.00 us 81
> FLUSH
> 0.03 206.95 us 163.00 us 497.00 us 22
> CREATE
> 0.04 85.16 us 51.00 us 168.00 us 70
> OPEN
> 0.04 145.02 us 92.00 us 344.00 us 45
> FXATTROP
> 0.07 59.91 us 3.00 us 145.00 us 184
> OPENDIR
> 0.13 1024.41 us 61.00 us 12251.00 us 22
> WRITE
> 0.14 444.38 us 44.00 us 11184.00 us 52
> READLINK
> 0.27 45104.00 us 45104.00 us 45104.00 us 1
> TRUNCATE
> 20.33 1895.33 us 27.00 us 6083.00 us 1819
> READDIRP
> 78.84 645.59 us 35.00 us 209082.00 us 20706
> LOOKUP
>
> Duration: 269212 seconds
> Data Read: 376538217873 bytes
> Data Written: 21968843126 bytes
>
> Interval 1 Stats:
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 0 0
> 0
> No. of Writes: 1 4
> 3
>
> Block Size: 1024b+ 32768b+
> 131072b+
> No. of Reads: 0 1
> 10
> No. of Writes: 6 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 8
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 45
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 178
> RELEASEDIR
> 0.00 81.00 us 81.00 us 81.00 us 1
> SETATTR
> 0.00 85.00 us 85.00 us 85.00 us 1
> READDIR
> 0.00 63.33 us 35.00 us 87.00 us 3
> GETXATTR
> 0.00 58.50 us 56.00 us 60.00 us 4
> FSTAT
> 0.00 34.86 us 24.00 us 46.00 us 7
> STAT
> 0.00 54.00 us 47.00 us 66.00 us 9
> INODELK
> 0.01 51.25 us 33.00 us 78.00 us 12
> STATFS
> 0.01 156.20 us 112.00 us 202.00 us 5
> UNLINK
> 0.01 61.33 us 34.00 us 259.00 us 18
> LK
> 0.02 61.64 us 40.00 us 125.00 us 28
> FINODELK
> 0.02 54.65 us 21.00 us 579.00 us 40
> FLUSH
> 0.02 54.12 us 28.00 us 97.00 us 42
> ENTRYLK
> 0.02 161.93 us 74.00 us 302.00 us 15
> READ
> 0.02 83.97 us 68.00 us 107.00 us 29
> OPEN
> 0.03 214.88 us 163.00 us 497.00 us 16
> CREATE
> 0.04 144.21 us 92.00 us 344.00 us 28
> FXATTROP
> 0.10 60.01 us 3.00 us 145.00 us 178
> OPENDIR
> 0.10 316.62 us 44.00 us 8512.00 us 34
> READLINK
> 0.15 1121.14 us 61.00 us 12251.00 us 14
> WRITE
> 32.97 1895.33 us 27.00 us 6083.00 us 1819
> READDIRP
> 66.46 667.09 us 35.00 us 67882.00 us 10418
> LOOKUP
>
> Duration: 18 seconds
> Data Read: 1352163 bytes
> Data Written: 8465 bytes
>
> Profile 2:
> Brick: 192.168.10.92:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 133 6
> 37
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 28 93
> 31
> No. of Writes: 4577 6495
> 22231
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 219 218
> 254
> No. of Writes: 37641 41723
> 20519
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 484 981
> 696
> No. of Writes: 39094 42613
> 203748
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 1079 1802
> 2840
> No. of Writes: 220154 229725
> 215083
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 4227 14944
> 57816
> No. of Writes: 60938 61967
> 793400
>
> Block Size: 262144b+ 524288b+
> 1048576b+
> No. of Reads: 6 5
> 2
> No. of Writes: 5 0
> 0
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 1835089
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 2639439
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1440418
> RELEASEDIR
> 0.00 53.50 us 43.00 us 64.00 us 2
> READDIR
> 0.00 153.50 us 142.00 us 165.00 us 2
> XATTROP
> 0.00 113.00 us 97.00 us 144.00 us 4
> SETATTR
> 0.00 114.60 us 49.00 us 339.00 us 10
> GETXATTR
> 0.00 55.00 us 33.00 us 100.00 us 24
> INODELK
> 0.00 48.85 us 25.00 us 81.00 us 71
> STATFS
> 0.01 49.47 us 31.00 us 76.00 us 92
> LK
> 0.01 36.07 us 16.00 us 68.00 us 180
> FLUSH
> 0.01 52.45 us 32.00 us 88.00 us 159
> FINODELK
> 0.02 79.16 us 47.00 us 112.00 us 137
> OPEN
> 0.02 51.06 us 26.00 us 95.00 us 216
> ENTRYLK
> 0.03 24814.00 us 24814.00 us 24814.00 us 1
> TRUNCATE
> 0.06 271.36 us 86.00 us 21829.00 us 158
> FXATTROP
> 0.07 59.02 us 2.00 us 100.00 us 881
> OPENDIR
> 0.07 5910.00 us 56.00 us 16873.00 us 9
> READLINK
> 0.11 33.81 us 11.00 us 1950.00 us 2348
> ACCESS
> 0.17 4181.53 us 129.00 us 41541.00 us 30
> UNLINK
> 0.31 2788.77 us 59.00 us 56839.00 us 79
> WRITE
> 0.37 2096.99 us 54.00 us 61234.00 us 125
> READ
> 0.38 84.58 us 53.00 us 313.00 us 3191
> FSTAT
> 0.50 61.21 us 20.00 us 14904.00 us 5873
> STAT
> 0.76 7023.69 us 145.00 us 102353.00 us 78
> CREATE
> 8.90 2002.80 us 361.00 us 3545.00 us 3191
> READDIRP
> 88.19 1112.81 us 34.00 us 307904.00 us 56876
> LOOKUP
>
> Duration: 338222 seconds
> Data Read: 9292898491 bytes
> Data Written: 122127690284 bytes
>
> Interval 4 Stats:
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0 0
> 2
> No. of Writes: 1 3
> 12
>
> Block Size: 128b+ 1024b+
> 2048b+
> No. of Reads: 2 3
> 10
> No. of Writes: 8 16
> 0
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 4 2
> 1
> No. of Writes: 0 0
> 0
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 2 3
> 1
> No. of Writes: 0 0
> 0
>
> Block Size: 262144b+
> No. of Reads: 2
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 21
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 76
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 407
> RELEASEDIR
> 0.00 144.00 us 144.00 us 144.00 us 1
> SETATTR
> 0.00 63.33 us 56.00 us 69.00 us 3
> READLINK
> 0.00 226.00 us 113.00 us 339.00 us 2
> GETXATTR
> 0.00 54.89 us 39.00 us 70.00 us 9
> INODELK
> 0.01 48.03 us 25.00 us 72.00 us 37
> STATFS
> 0.01 36.75 us 16.00 us 65.00 us 59
> FLUSH
> 0.01 50.44 us 33.00 us 76.00 us 48
> LK
> 0.01 83.73 us 55.00 us 111.00 us 37
> OPEN
> 0.01 53.79 us 32.00 us 80.00 us 80
> FINODELK
> 0.02 50.55 us 26.00 us 83.00 us 104
> ENTRYLK
> 0.03 135.21 us 86.00 us 228.00 us 80
> FXATTROP
> 0.07 58.75 us 35.00 us 98.00 us 407
> OPENDIR
> 0.10 1044.22 us 55.00 us 30268.00 us 32
> READ
> 0.14 33.74 us 13.00 us 1044.00 us 1384
> ACCESS
> 0.19 5076.42 us 141.00 us 41541.00 us 12
> UNLINK
> 0.36 2860.70 us 59.00 us 29520.00 us 40
> WRITE
> 0.36 84.16 us 53.00 us 182.00 us 1369
> FSTAT
> 0.65 5225.18 us 145.00 us 28783.00 us 40
> CREATE
> 0.73 61.98 us 20.00 us 14904.00 us 3771
> STAT
> 8.53 2009.18 us 518.00 us 2532.00 us 1369
> READDIRP
> 88.77 1156.14 us 36.00 us 232288.00 us 24747
> LOOKUP
>
> Duration: 48 seconds
> Data Read: 1424095 bytes
> Data Written: 22682 bytes
>
> Brick: 192.168.10.91:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 1b+ 2b+
> 4b+
> No. of Reads: 0 0
> 0
> No. of Writes: 3 3
> 6
>
> Block Size: 8b+ 16b+
> 32b+
> No. of Reads: 13 173
> 58
> No. of Writes: 3371 5496
> 13461
>
> Block Size: 64b+ 128b+
> 256b+
> No. of Reads: 194 280
> 500
> No. of Writes: 33717 34277
> 7020
>
> Block Size: 512b+ 1024b+
> 2048b+
> No. of Reads: 1302 1888
> 3596
> No. of Writes: 26022 17110
> 12496
>
> Block Size: 4096b+ 8192b+
> 16384b+
> No. of Reads: 5031 13204
> 21838
> No. of Writes: 21889 32294
> 18289
>
> Block Size: 32768b+ 65536b+
> 131072b+
> No. of Reads: 73579 190511
> 2710676
> No. of Writes: 17017 13980
> 144272
>
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 817592
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1663910
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 1439269
> RELEASEDIR
> 0.00 61.67 us 39.00 us 85.00 us 3
> READDIR
> 0.00 136.50 us 108.00 us 165.00 us 2
> XATTROP
> 0.00 91.00 us 81.00 us 100.00 us 4
> SETATTR
> 0.00 58.86 us 35.00 us 87.00 us 7
> GETXATTR
> 0.00 60.57 us 55.00 us 69.00 us 7
> FSTAT
> 0.00 35.53 us 24.00 us 53.00 us 15
> STAT
> 0.00 57.12 us 39.00 us 72.00 us 24
> INODELK
> 0.01 51.25 us 33.00 us 90.00 us 71
> STATFS
> 0.01 52.09 us 32.00 us 259.00 us 92
> LK
> 0.01 175.87 us 112.00 us 691.00 us 30
> UNLINK
> 0.01 41.84 us 16.00 us 579.00 us 175
> FLUSH
> 0.02 56.23 us 25.00 us 125.00 us 158
> FINODELK
> 0.02 51.59 us 23.00 us 97.00 us 216
> ENTRYLK
> 0.02 85.10 us 51.00 us 168.00 us 133
> OPEN
> 0.03 200.55 us 150.00 us 554.00 us 78
> CREATE
> 0.06 163.03 us 38.00 us 11184.00 us 204
> READLINK
> 0.07 1022.60 us 67.00 us 24446.00 us 40
> READ
> 0.08 45104.00 us 45104.00 us 45104.00 us 1
> TRUNCATE
> 0.09 59.52 us 3.00 us 145.00 us 843
> OPENDIR
> 0.13 936.73 us 56.00 us 12251.00 us 78
> WRITE
> 0.19 687.17 us 88.00 us 85925.00 us 157
> FXATTROP
> 22.66 1881.29 us 23.00 us 6083.00 us 6698
> READDIRP
> 76.59 754.63 us 33.00 us 209082.00 us 56449
> LOOKUP
>
> Duration: 269283 seconds
> Data Read: 376540270509 bytes
> Data Written: 21968880277 bytes
>
> Interval 4 Stats:
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0 0
> 0
> No. of Writes: 1 3
> 12
>
> Block Size: 128b+ 1024b+
> 32768b+
> No. of Reads: 0 0
> 1
> No. of Writes: 8 16
> 0
>
> Block Size: 65536b+ 131072b+
> No. of Reads: 2 9
> No. of Writes: 0 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 36
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 76
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 408
> RELEASEDIR
> 0.00 95.00 us 95.00 us 95.00 us 1
> SETATTR
> 0.00 48.00 us 39.00 us 57.00 us 2
> GETXATTR
> 0.00 50.00 us 39.00 us 61.00 us 2
> READDIR
> 0.00 35.50 us 27.00 us 53.00 us 4
> STAT
> 0.00 63.33 us 55.00 us 69.00 us 3
> FSTAT
> 0.00 61.11 us 50.00 us 72.00 us 9
> INODELK
> 0.01 50.41 us 34.00 us 90.00 us 37
> STATFS
> 0.01 160.08 us 125.00 us 213.00 us 12
> UNLINK
> 0.01 37.19 us 18.00 us 71.00 us 59
> FLUSH
> 0.01 48.46 us 32.00 us 78.00 us 48
> LK
> 0.01 84.92 us 57.00 us 118.00 us 37
> OPEN
> 0.02 54.89 us 25.00 us 89.00 us 80
> FINODELK
> 0.02 50.57 us 27.00 us 74.00 us 104
> ENTRYLK
> 0.03 66.88 us 38.00 us 100.00 us 113
> READLINK
> 0.03 196.90 us 150.00 us 554.00 us 40
> CREATE
> 0.04 139.06 us 88.00 us 233.00 us 80
> FXATTROP
> 0.10 58.95 us 8.00 us 93.00 us 408
> OPENDIR
> 0.11 1675.56 us 67.00 us 24446.00 us 16
> READ
> 0.17 1066.62 us 62.00 us 8506.00 us 40
> WRITE
> 23.14 1889.02 us 23.00 us 3775.00 us 3073
> READDIRP
> 76.29 773.40 us 33.00 us 135928.00 us 24746
> LOOKUP
>
> Duration: 48 seconds
> Data Read: 1446611 bytes
> Data Written: 22682 bytes
>
> Brick: 192.168.10.90:/export/brick1
> -----------------------------------
> Cumulative Stats:
> Block Size: 2b+ 8b+
> 16b+
> No. of Reads: 0 6
> 32
> No. of Writes: 1 1929
> 2605
>
> Block Size: 32b+ 64b+
> 128b+
> No. of Reads: 21 112
> 169
> No. of Writes: 6986 12080
> 26020
>
> Block Size: 256b+ 512b+
> 1024b+
> No. of Reads: 85 299
> 809
> No. of Writes: 743 18584
> 7233
>
> Block Size: 2048b+ 4096b+
> 8192b+
> No. of Reads: 790 1419
> 2885
> No. of Writes: 1651 10647
> 16901
>
> Block Size: 16384b+ 32768b+
> 65536b+
> No. of Reads: 4588 5858
> 32434
> No. of Writes: 2974 5758
> 3387
>
> Block Size: 131072b+
> No. of Reads: 455218
> No. of Writes: 4601
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 604813
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 1410912
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 990360
> RELEASEDIR
> 0.00 44.33 us 16.00 us 62.00 us 6
> READDIR
> 0.00 147.50 us 136.00 us 159.00 us 2
> XATTROP
> 0.00 100.25 us 86.00 us 125.00 us 4
> SETATTR
> 0.00 60.11 us 39.00 us 92.00 us 9
> GETXATTR
> 0.00 60.19 us 41.00 us 84.00 us 16
> FSTAT
> 0.00 45.50 us 17.00 us 182.00 us 38
> STAT
> 0.00 51.37 us 31.00 us 81.00 us 71
> STATFS
> 0.00 49.05 us 30.00 us 79.00 us 92
> LK
> 0.00 168.87 us 115.00 us 287.00 us 30
> UNLINK
> 0.01 38.74 us 15.00 us 150.00 us 150
> FLUSH
> 0.01 54.76 us 30.00 us 236.00 us 158
> FINODELK
> 0.01 87.17 us 49.00 us 352.00 us 106
> OPEN
> 0.01 52.56 us 27.00 us 91.00 us 216
> ENTRYLK
> 0.02 174.25 us 73.00 us 5551.00 us 157
> FXATTROP
> 0.03 35786.00 us 35786.00 us 35786.00 us 1
> TRUNCATE
> 0.05 62.12 us 3.00 us 1628.00 us 843
> OPENDIR
> 0.10 685.58 us 41.00 us 51432.00 us 157
> READLINK
> 0.36 6483.54 us 59.00 us 47225.00 us 63
> READ
> 0.58 8304.77 us 60.00 us 185503.00 us 78
> WRITE
> 0.67 9646.06 us 150.00 us 686579.00 us 78
> CREATE
> 1.82 85338.88 us 25.00 us 1502989.00 us 24
> INODELK
> 14.62 1920.48 us 23.00 us 9822.00 us 8567
> READDIRP
> 81.71 1635.58 us 30.00 us 828308.00 us 56208
> LOOKUP
>
> Duration: 76104 seconds
> Data Read: 63115678019 bytes
> Data Written: 1489698979 bytes
>
> Interval 4 Stats:
> Block Size: 16b+ 32b+
> 64b+
> No. of Reads: 0 0
> 0
> No. of Writes: 1 3
> 12
>
> Block Size: 128b+ 1024b+
> 4096b+
> No. of Reads: 0 0
> 1
> No. of Writes: 8 16
> 0
>
> Block Size: 8192b+ 32768b+
> 65536b+
> No. of Reads: 1 1
> 6
> No. of Writes: 0 0
> 0
>
> Block Size: 131072b+
> No. of Reads: 10
> No. of Writes: 0
> %-latency Avg-latency Min-Latency Max-Latency No. of calls
> Fop
> --------- ----------- ----------- ----------- ------------
> ----
> 0.00 0.00 us 0.00 us 0.00 us 29
> FORGET
> 0.00 0.00 us 0.00 us 0.00 us 76
> RELEASE
> 0.00 0.00 us 0.00 us 0.00 us 409
> RELEASEDIR
> 0.00 125.00 us 125.00 us 125.00 us 1
> SETATTR
> 0.00 43.75 us 16.00 us 60.00 us 4
> READDIR
> 0.00 61.80 us 39.00 us 92.00 us 5
> GETXATTR
> 0.00 65.71 us 44.00 us 76.00 us 7
> FSTAT
> 0.00 48.50 us 24.00 us 182.00 us 28
> STAT
> 0.00 50.08 us 31.00 us 70.00 us 37
> STATFS
> 0.00 165.25 us 119.00 us 238.00 us 12
> UNLINK
> 0.00 39.66 us 19.00 us 62.00 us 59
> FLUSH
> 0.00 51.12 us 33.00 us 79.00 us 48
> LK
> 0.01 82.78 us 64.00 us 104.00 us 37
> OPEN
> 0.01 56.45 us 30.00 us 236.00 us 80
> FINODELK
> 0.01 52.17 us 28.00 us 84.00 us 104
> ENTRYLK
> 0.02 154.12 us 88.00 us 1577.00 us 80
> FXATTROP
> 0.05 65.00 us 3.00 us 1628.00 us 409
> OPENDIR
> 0.09 909.64 us 43.00 us 51432.00 us 61
> READLINK
> 0.60 9748.22 us 59.00 us 47225.00 us 36
> READ
> 0.86 12688.17 us 60.00 us 185503.00 us 40
> WRITE
> 0.92 60488.89 us 50.00 us 543898.00 us 9
> INODELK
> 1.26 18623.40 us 161.00 us 686579.00 us 40
> CREATE
> 19.51 1951.60 us 25.00 us 9822.00 us 5889
> READDIRP
> 76.64 1823.96 us 30.00 us 318395.00 us 24747
> LOOKUP
>
> Duration: 48 seconds
> Data Read: 1931486 bytes
> Data Written: 22682 bytes
>
> Again, any assistance on this issue would be much appreciated.
>
> Thank you.
>
>
>
> On Sat, Dec 20, 2014 at 11:58 PM, Kyle Harris <kyle.harris98 at gmail.com>
> wrote:
>
>> Hello All,
>>
>> I have been experiencing difficulty with high process utilization on the
>> glusterfs and glusterfsd processes. I thought I had it tracked down to
>> Apache looking for files on the cluster drive that don't exist. I have
>> been working to clean that up and now I have discovered another problem.
>>
>> When I reboot one of the nodes in my 3-node replicated cluster, as soon
>> as it comes back up I see a bunch of entries such as follows:
>>
>> [2014-12-21 05:33:49.873989] I [dict.c:370:dict_get] (-->
>> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1e0)[0x7fe3bd0d2420] (-->
>> /usr/lib64/libglusterfs.so.0(dict_get+0x4f)[0x7fe3bd0cbbdf] (-->
>> /usr/lib64/glusterfs/3.6.1/xlator/system/posix-acl.so(posix_acl_lookup_cbk+0x1e1)[0x7fe3aad3ada1]
>> (-->
>> /usr/lib64/glusterfs/3.6.1/xlator/debug/io-stats.so(io_stats_lookup_cbk+0x113)[0x7fe3aaf524e3]
>> (-->
>> /usr/lib64/glusterfs/3.6.1/xlator/performance/md-cache.so(mdc_lookup+0x300)[0x7fe3ab167310]
>> ))))) 0-dict: !this || key=system.posix_acl_access
>>
>> While this is occurring, the process utilization for glusterfs and
>> glusterfsd go sky high. Eventually they calm down on their own but the
>> servers are virtually unusable until they do. Note this is on Centos 6.6
>> with Gluster 3.6.1 and kernel 2.6.32 using a fuse mount with each of the 3
>> servers having the client mounted for hosting web sites via Apache. There
>> are many small files and some are php. I have not done any tuning to the
>> default gluster parameters.
>>
>> Can someone tell me why this occurs after one of the nodes is rebooted
>> and perhaps lend some guidance on what can be done to resolve this issue?
>>
>> Thank you.
>>
>> --
>> Kyle
>>
>>
>
>
> --
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
>
>
--
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20141221/22b6efbe/attachment.html>
More information about the Gluster-users
mailing list