<div dir="ltr"><div><div>CPU load in some of brick servers are very high and write performance is very slow.<br></div><div>dd one file to the volume, the result is only 10+KB/sec<br><br>any comments?<br></div><div><br></div>
<div>more infomation >>>>><br></div><div>Volume Name: prodvolume<br>Type: Distributed-Replicate<br>Volume ID: f3fc24b3-23c7-430d-8ab1-81a646b1ce06<br>Status: Started<br>Number of Bricks: 17 x 3 = 51 (I have 51 servers)<br>
Transport-type: tcp<br>Bricks:<br> ....<br>Options Reconfigured:<br>performance.io-thread-count: 32<br>auth.allow: *,10.121.48.244,10.121.48.82<br>features.limit-usage: /:400TB<br>features.quota: on<br>server.allow-insecure: on<br>
features.quota-timeout: 5<br><br></div><br></div>most of cpu utilization from system/kernel mode<br><div><div><br>top - 14:47:13 up 219 days, 23:36, 2 users, load average: 17.76, 20.98, 24.74<br>Tasks: 493 total, 1 running, 491 sleeping, 0 stopped, 1 zombie<br>
Cpu(s): 8.2%us, 49.0%sy, 0.0%ni, 42.2%id, 0.1%wa, 0.0%hi, 0.4%si, 0.0%st<br>Mem: 132112276k total, 131170760k used, 941516k free, 71224k buffers<br>Swap: 4194296k total, 867216k used, 3327080k free, 110888216k cached<br>
<br> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br><b> 6226 root 20 0 2677m 496m 2268 S 1183.4 0.4 89252:09 glusterfsd</b><br>27994 root 20 0 1691m 77m 2000 S 111.6 0.1 324333:47 glusterfsd<br>
14169 root 20 0 14.9g 23m 1984 S 51.3 0.0 3700:30 glusterfsd<br>20582 root 20 0 2129m 1.4g 1708 S 12.6 1.1 198:03.53 glusterfs<br>24528 root 20 0 0 0 0 S 6.3 0.0 14:18.60 flush-8:16<br>
17717 root 20 0 21416 11m 8268 S 5.0 0.0 14:51.18 oprofiled<br><br><br></div><div>use perf top -p 6226, are casusd by spin_lock<br><br>Events: 49K cycles<br> 72.51% [kernel] [k] _spin_lock<br>
4.00% <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> [.] pthread_mutex_lock<br> 2.63% [kernel] [k] _spin_unlock_irqrestore<br> 1.61% <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> [.] pthread_mutex_unlock<br>
1.59% [unknown] [.] 0xffffffffff600157<br> 1.57% [xfs] [k] xfs_inobt_get_rec<br> 1.41% [xfs] [k] xfs_btree_increment<br> 1.27% [xfs] [k] xfs_btree_get_rec<br>
1.17% <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> [.] __lll_lock_wait<br> 0.96% [xfs] [k] _xfs_buf_find<br> 0.95% [xfs] [k] xfs_btree_get_block<br>
0.88% [kernel] [k] copy_user_generic_string<br> 0.50% [xfs] [k] xfs_dialloc<br> 0.48% [xfs] [k] xfs_btree_rec_offset<br> 0.47% [xfs] [k] xfs_btree_readahead<br>
0.41% [kernel] [k] futex_wait_setup<br> 0.41% [kernel] [k] futex_wake<br> 0.35% [kernel] [k] system_call_after_swapgs<br> 0.33% [xfs] [k] xfs_btree_rec_addr<br>
0.30% [kernel] [k] __link_path_walk<br> 0.29% io-threads.so.0.0.0 [.] __iot_dequeue<br> 0.29% io-threads.so.0.0.0 [.] iot_worker<br> 0.25% [kernel] [k] __d_lookup<br>
0.21% <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> [.] __lll_unlock_wake<br> 0.20% [kernel] [k] get_futex_key<br> 0.18% [kernel] [k] hash_futex<br>
0.17% [kernel] [k] do_futex<br> 0.15% [kernel] [k] thread_return<br> 0.15% <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> [.] pthread_spin_lock<br> 0.14% <a href="http://libc-2.12.so">libc-2.12.so</a> [.] _int_malloc<br>
0.14% [kernel] [k] sys_futex<br> 0.14% [kernel] [k] wake_futex<br> 0.14% [kernel] [k] _atomic_dec_and_lock<br> 0.12% [kernel] [k] kmem_cache_free<br>
0.12% [xfs] [k] xfs_trans_buf_item_match<br> 0.12% [xfs] [k] xfs_btree_check_sblock<br> 0.11% <a href="http://libc-2.12.so">libc-2.12.so</a> [.] vfprintf<br>
0.11% [kernel] [k] futex_wait<br> 0.11% [kernel] [k] kmem_cache_alloc<br> 0.09% [kernel] [k] acl_permission_check<br><br></div><div>use oprifile, I found the cpu are almost caused breakdown into:<br>
<br>CPU: Intel Sandy Bridge microarchitecture, speed 2000.02 MHz (estimated)<br>Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a unit mask of 0x00 (No unit mask) count 100000<br>samples % linenr info image name app name symbol name<br>
-------------------------------------------------------------------------------<br>288683303 41.2321 clocksource.c:828 vmlinux vmlinux <b> sysfs_show_available_clocksources</b><br>
288683303 100.000 clocksource.c:828 vmlinux vmlinux sysfs_show_available_clocksources [self]<br>-------------------------------------------------------------------------------<br>
203797076 29.1079 clocksource.c:236 vmlinux vmlinux <b>clocksource_mark_unstable</b><br> 203797076 100.000 clocksource.c:236 vmlinux vmlinux clocksource_mark_unstable [self]<br>
-------------------------------------------------------------------------------<br>42321053 6.0446 (no location information) xfs xfs /xfs<br> 42321053 100.000 (no location information) xfs xfs /xfs [self]<br>
-------------------------------------------------------------------------------<br>23662768 3.3797 (no location information) <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> pthread_mutex_lock<br>
23662768 100.000 (no location information) <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> pthread_mutex_lock [self]<br>-------------------------------------------------------------------------------<br>
10867915 1.5522 (no location information) <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> pthread_mutex_unlock<br> 10867915 100.000 (no location information) <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> pthread_mutex_unlock [self]<br>
-------------------------------------------------------------------------------<br>7727828 1.1038 (no location information) <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> __lll_lock_wait<br>
7727828 100.000 (no location information) <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> <a href="http://libpthread-2.12.so">libpthread-2.12.so</a> __lll_lock_wait [self]<br>-------------------------------------------------------------------------------<br>
6296394 0.8993 blk-sysfs.c:260 vmlinux vmlinux queue_rq_affinity_store<br> 6296394 100.000 blk-sysfs.c:260 vmlinux vmlinux queue_rq_affinity_store [self]<br>
-------------------------------------------------------------------------------<br>3543413 0.5061 sched.h:293 vmlinux vmlinux ftrace_profile_templ_sched_stat_template<br>
3543413 100.000 sched.h:293 vmlinux vmlinux ftrace_profile_templ_sched_stat_template [self]<br>-------------------------------------------------------------------------------<br>
2960958 0.4229 msi.c:82 vmlinux vmlinux msi_set_enable<br> 2960958 100.000 msi.c:82 vmlinux vmlinux msi_set_enable [self]<br>
-------------------------------------------------------------------------------<br>2814515 0.4020 clocksource.c:249 vmlinux vmlinux clocksource_watchdog<br> 2814515 100.000 clocksource.c:249 vmlinux vmlinux clocksource_watchdog [self]<br>
<br></div></div></div>