[Bugs] [Bug 1275323] [Tier]: CPU shoots up to 200-300%age when a 1MB file is written from fuse client every 10 secs
bugzilla at redhat.com
bugzilla at redhat.com
Thu Nov 26 08:58:03 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1275323
Rahul Hinduja <rhinduja at redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|ON_QA |ASSIGNED
--- Comment #4 from Rahul Hinduja <rhinduja at redhat.com> ---
Verified with build: glusterfs-3.7.5-7.el7rhgs.x86_64
I still see CPU usage sometime shooting upto 350+ %age. Below are some stats
from few machines in cluster.
Steps to reproduce remains same.
Watermarks {low and Hi} are set to 1 and 2 respectively
CPU information for machines:
=============================
[root at dhcp37-199 ~]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel Xeon E312xx (Sandy Bridge)
stepping : 1
microcode : 0x1
cpu MHz : 2199.998
cache size : 4096 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc nopl eagerfpu pni
pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave
avx hypervisor lahf_lm xsaveopt
bogomips : 4399.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel Xeon E312xx (Sandy Bridge)
stepping : 1
microcode : 0x1
cpu MHz : 2199.998
cache size : 4096 KB
physical id : 1
siblings : 1
core id : 0
cpu cores : 1
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc nopl eagerfpu pni
pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave
avx hypervisor lahf_lm xsaveopt
bogomips : 4399.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel Xeon E312xx (Sandy Bridge)
stepping : 1
microcode : 0x1
cpu MHz : 2199.998
cache size : 4096 KB
physical id : 2
siblings : 1
core id : 0
cpu cores : 1
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc nopl eagerfpu pni
pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave
avx hypervisor lahf_lm xsaveopt
bogomips : 4399.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 42
model name : Intel Xeon E312xx (Sandy Bridge)
stepping : 1
microcode : 0x1
cpu MHz : 2199.998
cache size : 4096 KB
physical id : 3
siblings : 1
core id : 0
cpu cores : 1
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush mmx fxsr sse sse2 syscall nx lm constant_tsc nopl eagerfpu pni
pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave
avx hypervisor lahf_lm xsaveopt
bogomips : 4399.99
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
[root at dhcp37-199 ~]#
Machine 1:
%Cpu(s): 33.6 us, 43.5 sy, 0.0 ni, 13.3 id, 0.4 wa, 0.0 hi, 7.7 si, 1.5 st
KiB Mem : 3882192 total, 125020 free, 1462676 used, 2294496 buff/cache
KiB Swap: 2097148 total, 2095600 free, 1548 used. 2059368 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1920 root 20 0 2405516 103084 4740 S 208.0 2.7 95:54.44 glusterfsd
2002 root 20 0 2541716 83748 4708 S 91.0 2.2 25:55.28 glusterfsd
2072 root 20 0 1514536 731836 4352 S 44.5 18.9 14:27.88 glusterfs
21 root 20 0 0 0 0 S 0.3 0.0 3:46.50 ksoftirqd/1
p - 14:18:40 up 27 days, 1:43, 3 users, load average: 6.19, 5.90, 5.82
Tasks: 191 total, 2 running, 189 sleeping, 0 stopped, 0 zombie
%Cpu(s): 14.7 us, 18.4 sy, 0.0 ni, 59.1 id, 1.8 wa, 0.0 hi, 4.6 si, 1.4 st
KiB Mem : 3882192 total, 108352 free, 1465076 used, 2308764 buff/cache
KiB Swap: 2097148 total, 2095600 free, 1548 used. 2056752 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1920 root 20 0 2405516 103308 4740 S 90.0 2.7 96:07.46 glusterfsd
2072 root 20 0 1514536 733420 4352 S 44.2 18.9 14:29.90 glusterfs
2002 root 20 0 2541716 83748 4708 S 38.2 2.2 25:58.43 glusterfsd
21 root 20 0 0 0 0 S 0.3 0.0 3:46.52 ksoftirqd/1
2235 root 20 0 146144 2060 1420 R 0.3 0.1 0:13.65 top
[root at dhcp37-110 ~]#
[root at dhcp37-110 ~]#
[root at dhcp37-110 ~]#
[root at dhcp37-110 ~]# top
top - 14:18:58 up 27 days, 1:43, 3 users, load average: 6.79, 6.04, 5.86
Tasks: 191 total, 1 running, 190 sleeping, 0 stopped, 0 zombie
%Cpu(s): 23.9 us, 42.4 sy, 0.0 ni, 28.7 id, 0.3 wa, 0.0 hi, 3.7 si, 0.9 st
KiB Mem : 3882192 total, 101892 free, 1468804 used, 2311496 buff/cache
KiB Swap: 2097148 total, 2095600 free, 1548 used. 2053184 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2002 root 20 0 2541716 83748 4708 S 148.7 2.2 26:07.19 glusterfsd
1920 root 20 0 2405516 103592 4740 S 129.0 2.7 96:38.33 glusterfsd
2072 root 20 0 1514536 736588 4352 S 11.3 19.0 14:34.09 glusterfs
2150 root 20 0 0 0 0 S 0.3 0.0 0:01.51
kworker/u8:3
2235 root 20 0 146144 2060 1420 R 0.3 0.1 0:13.69 top
top - 14:19:10 up 27 days, 1:43, 3 users, load average: 7.25, 6.16, 5.90
Tasks: 191 total, 1 running, 190 sleeping, 0 stopped, 0 zombie
%Cpu(s): 27.0 us, 35.3 sy, 0.0 ni, 28.6 id, 0.7 wa, 0.0 hi, 7.2 si, 1.3 st
KiB Mem : 3882192 total, 126084 free, 1471560 used, 2284548 buff/cache
KiB Swap: 2097148 total, 2095600 free, 1548 used. 2050360 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1920 root 20 0 2405516 103812 4740 S 185.7 2.7 96:51.05 glusterfsd
2072 root 20 0 1514536 739492 4352 S 51.2 19.0 14:38.71 glusterfs
2002 root 20 0 2541716 83748 4708 S 49.8 2.2 26:10.70 glusterfsd
21 root 20 0 0 0 0 S 0.3 0.0 3:46.58 ksoftirqd/1
2235 root 20 0 146144 2060 1420 R 0.3 0.1 0:13.71 top
%Cpu(s): 24.0 us, 70.1 sy, 0.0 ni, 2.1 id, 0.8 wa, 0.0 hi, 2.2 si, 0.7 st
KiB Mem : 3882192 total, 123940 free, 1473136 used, 2285116 buff/cache
KiB Swap: 2097148 total, 2095600 free, 1548 used. 2048916 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1920 root 20 0 2405516 103812 4740 S 357.8 2.7 97:09.47 glusterfsd
2072 root 20 0 1514536 741340 4352 S 21.6 19.1 14:40.86 glusterfs
2002 root 20 0 2541716 83748 4708 S 4.7 2.2 26:11.65 glusterfsd
2030 root 20 0 2089052 105416 3032 S 0.7 2.7 0:02.72 glusterfs
21 root 20 0 0 0 0 S 0.3 0.0 3:46.59 ksoftirqd/1
2235 root 20 0 146144 2060 1420 R 0.3 0.1 0:13.72 top
u(s): 18.8 us, 33.8 sy, 0.0 ni, 40.3 id, 0.8 wa, 0.0 hi, 4.5 si, 1.8 st
KiB Mem : 3882192 total, 119936 free, 1475432 used, 2286824 buff/cache
KiB Swap: 2097148 total, 2095600 free, 1548 used. 2046400 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1920 root 20 0 2405516 103812 4740 S 182.7 2.7 97:24.35 glusterfsd
2072 root 20 0 1514536 743452 4352 S 30.2 19.2 14:43.63 glusterfs
2002 root 20 0 2541716 83748 4708 S 29.9 2.2 26:13.33 glusterfsd
2235 root 20 0 146144 2060 1420 R 0.7 0.1 0:13.74 top
1 root 20 0 191632 6156 3296 S 0.0 0.2 0:57.54 systemd
root at 10.70.37.110's password:
Last login: Thu Nov 26 12:13:21 2015 from dhcp35-201.lab.eng.blr.redhat.com
[root at dhcp37-110 ~]#
[root at dhcp37-110 ~]#
[root at dhcp37-110 ~]#
[root at dhcp37-110 ~]# top
top - 14:19:31 up 27 days, 1:44, 3 users, load average: 7.60, 6.35, 5.97
Tasks: 191 total, 1 running, 190 sleeping, 0 stopped, 0 zombie
%Cpu(s): 15.1 us, 29.1 sy, 0.0 ni, 50.2 id, 1.5 wa, 0.0 hi, 2.8 si, 1.3 st
KiB Mem : 3882192 total, 116716 free, 1476928 used, 2288548 buff/cache
KiB Swap: 2097148 total, 2095600 free, 1548 used. 2045128 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1920 root 20 0 2405516 103820 4740 S 155.8 2.7 97:34.55 glusterfsd
2002 root 20 0 2541716 83748 4708 S 30.9 2.2 26:15.69 glusterfsd
2072 root 20 0 1514536 745036 4352 S 19.9 19.2 14:45.68 glusterfs
21 root 20 0 0 0 0 S 0.3 0.0 3:46.61 ksoftirqd/1
2150 root 20 0 0 0 0 S 0.3 0.0 0:01.52 kworker/u8
Machine 2:
==========
Tasks: 192 total, 2 running, 190 sleeping, 0 stopped, 0 zombie
%Cpu(s): 19.8 us, 28.9 sy, 0.0 ni, 45.7 id, 1.0 wa, 0.0 hi, 3.0 si, 1.6 st
KiB Mem : 3882184 total, 1044460 free, 846392 used, 1991332 buff/cache
KiB Swap: 2097148 total, 2094432 free, 2716 used. 2679944 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23336 root 20 0 2263148 106660 4756 S 136.5 2.7 55:59.39 glusterfsd
23417 root 20 0 2499916 80312 4720 S 83.1 2.1 32:36.07 glusterfsd
23489 root 20 0 990248 117672 4608 S 5.6 3.0 7:33.37 glusterfs
21 root 20 0 0 0 0 S 0.3 0.0 6:13.23 ksoftirqd/1
Tasks: 192 total, 2 running, 190 sleeping, 0 stopped, 0 zombie
%Cpu(s): 14.7 us, 19.9 sy, 0.0 ni, 59.9 id, 0.7 wa, 0.0 hi, 3.1 si, 1.6 st
KiB Mem : 3882184 total, 1036836 free, 849884 used, 1995464 buff/cache
KiB Swap: 2097148 total, 2094432 free, 2716 used. 2676772 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23336 root 20 0 2263148 106660 4756 S 98.0 2.7 56:13.10 glusterfsd
23417 root 20 0 2499916 80312 4720 S 65.8 2.1 32:45.94 glusterfsd
23489 root 20 0 990248 119480 4608 S 10.0 3.1 7:34.68 glusterfs
23650 root 20 0 146144 2048 1420 R 0.7 0.1 0:15.20 top
21 root 20 0 0 0 0 S 0.3 0.0 6:13.28 ksoftirqd/1
4:21:23 up 27 days, 1:45, 3 users, load average: 5.38, 3.92, 3.13
Tasks: 192 total, 1 running, 191 sleeping, 0 stopped, 0 zombie
%Cpu(s): 12.6 us, 19.5 sy, 0.0 ni, 62.3 id, 0.8 wa, 0.0 hi, 3.2 si, 1.5 st
KiB Mem : 3882184 total, 1033952 free, 850020 used, 1998212 buff/cache
KiB Swap: 2097148 total, 2094432 free, 2716 used. 2676260 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23336 root 20 0 2263148 106656 4756 S 108.6 2.7 56:21.09 glusterfsd
23417 root 20 0 2499916 80312 4720 S 38.5 2.1 32:50.68 glusterfsd
23489 root 20 0 990248 119676 4608 S 15.3 3.1 7:35.23 glusterfs
21 root 20 0 0 0 0 S 0.7 0.0 6:13.32 ksoftirqd/1
14 root 20 0 0 0 0 S 0.3 0.0 0:45.28 rcuos/0
%Cpu(s): 20.9 us, 36.1 sy, 0.0 ni, 34.6 id, 1.4 wa, 0.0 hi, 5.1 si, 1.9 st
KiB Mem : 3882184 total, 1020060 free, 850860 used, 2011264 buff/cache
KiB Swap: 2097148 total, 2094432 free, 2716 used. 2675808 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23336 root 20 0 2263148 106660 4756 S 191.0 2.7 56:32.08 glusterfsd
23417 root 20 0 2499916 80312 4720 S 63.8 2.1 32:54.16 glusterfsd
23489 root 20 0 990248 119764 4608 S 7.0 3.1 7:35.87 glusterfs
23650 root 20 0 146144 2048 1420 R 0.7 0.1 0:15.24 top
13 root 20 0 0 0 0 S 0.3 0.0 2:28.62 rcu_sched
Machine 3:
==========
%Cpu(s): 13.6 us, 49.5 sy, 0.0 ni, 34.6 id, 0.6 wa, 0.0 hi, 0.4 si, 1.3 st
KiB Mem : 3882184 total, 1539628 free, 836604 used, 1505952 buff/cache
KiB Swap: 2097148 total, 2095184 free, 1964 used. 2690168 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24195 root 20 0 2191488 99144 4736 S 265.4 2.6 60:19.25 glusterfsd
24341 root 20 0 990248 121524 4592 S 1.3 3.1 7:34.59 glusterfs
24273 root 20 0 2490484 78552 4716 S 1.0 2.0 35:19.10 glusterfsd
23823 root 20 0 0 0 0 S 0.3 0.0 0:04.22
xfsaild/dm-6
24509 root 20 0 146144 2048 1420 R 0.3 0.1 0:15.28 top
KiB Mem : 3882184 total, 1537992 free, 834152 used, 1510040 buff/cache
KiB Swap: 2097148 total, 2095184 free, 1964 used. 2692604 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24195 root 20 0 2191488 99184 4736 S 105.6 2.6 60:32.09 glusterfsd
24273 root 20 0 2490484 78552 4716 S 21.9 2.0 35:27.23 glusterfsd
24341 root 20 0 990248 119868 4592 S 10.0 3.1 7:35.41 glusterfs
13 root 20 0 0 0 0 S 0.3 0.0 2:28.20 rcu_sched
Tasks: 190 total, 2 running, 188 sleeping, 0 stopped, 0 zombie
%Cpu(s): 20.1 us, 54.7 sy, 0.0 ni, 19.7 id, 2.5 wa, 0.0 hi, 1.8 si, 1.1 st
KiB Mem : 3882184 total, 1531364 free, 836868 used, 1513952 buff/cache
KiB Swap: 2097148 total, 2095184 free, 1964 used. 2689920 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24195 root 20 0 2191488 99184 4736 S 291.3 2.6 60:47.16 glusterfsd
24273 root 20 0 2490484 78552 4716 S 12.7 2.0 35:30.64 glusterfsd
24341 root 20 0 990248 121956 4592 S 7.0 3.1 7:36.24 glusterfs
21 root 20 0 0 0 0 S 0.7 0.0 7:29.87 ksoftirqd/1
bash-4.3$ ssh root at 10.70.37.199
root at 10.70.37.199's password:
Last login: Thu Nov 26 12:13:21 2015 from dhcp35-201.lab.eng.blr.redhat.com
[root at dhcp37-199 ~]# top
top - 14:22:14 up 27 days, 1:46, 3 users, load average: 5.01, 3.79, 3.23
Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie
%Cpu(s): 19.4 us, 32.8 sy, 0.0 ni, 42.0 id, 1.1 wa, 0.0 hi, 3.5 si, 1.3 st
KiB Mem : 3882184 total, 1524516 free, 836252 used, 1521416 buff/cache
KiB Swap: 2097148 total, 2095184 free, 1964 used. 2690504 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24195 root 20 0 2191488 99184 4736 S 187.0 2.6 61:13.39 glusterfsd
24273 root 20 0 2490484 78552 4716 S 48.7 2.0 35:35.11 glusterfsd
24341 root 20 0 990248 121640 4592 S 3.7 3.1 7:37.60 glusterfs
13 root 20 0 0 0 0 S 0.3 0.0 2:28.22 rcu_sched
top - 14:24:06 up 27 days, 1:48, 3 users, load average: 5.85, 4.36, 3.49
Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie
%Cpu(s): 21.9 us, 71.8 sy, 0.0 ni, 3.0 id, 0.0 wa, 0.0 hi, 2.0 si, 1.3 st
KiB Mem : 3882184 total, 1441996 free, 837428 used, 1602760 buff/cache
KiB Swap: 2097148 total, 2095184 free, 1964 used. 2689160 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
24195 root 20 0 2191488 99556 4736 S 368.8 2.6 63:11.97 glusterfsd
24273 root 20 0 2490484 78552 4716 S 17.6 2.0 36:25.21 glusterfsd
21 root 20 0 0 0 0 S 0.3 0.0 7:30.40 ksoftirqd/1
1 root 20 0 191616 6244 3328 S 0.0 0.2 0:52.04 systemd
Moving bug to assigned state.
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=3o1JldxGjm&a=cc_unsubscribe
More information about the Bugs
mailing list