[Bugs] [Bug 1349953] thread CPU saturation limiting throughput on write workloads
bugzilla at redhat.com
bugzilla at redhat.com
Fri Jul 8 08:03:41 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1349953
--- Comment #17 from Manoj Pillai <mpillai at redhat.com> ---
Results with 3x2 volume (6 servers) with bricks on 12-disk RAID-6:
Benchmark: distributed iozone (write test) on 6 clients:
iozone -+m ${IOZONE_CONF} -i 0 -w -+n -c -C -e -s 10g -r 64k -t 48
With standard tuning options:
cluster.lookup-optimize: on
server.event-threads: 4
client.event-threads: 4
iozone output:
throughput for 48 initial writers = 2052985.17 kB/sec
iozone output after adding option performance.client-io-threads on:
throughput for 48 initial writers = 3180628.89 kB/sec
So we see a big bump in the case of replica-2 as well with client-io-threads.
So we really want this option on irrespective of volume type.
Without client-io-threads, we see the following in top output on the clients:
<quote>
top - 02:49:26 up 21:08, 0 users, load average: 0.24, 0.47, 0.47
Threads: 296 total, 4 running, 292 sleeping, 0 stopped, 0 zombie
%Cpu(s): 7.5 us, 9.2 sy, 0.0 ni, 82.6 id, 0.0 wa, 0.0 hi, 0.8 si, 0.0 st
KiB Mem : 65728904 total, 59729480 free, 758744 used, 5240680 buff/cache
KiB Swap: 32964604 total, 32964604 free, 0 used. 64684248 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5156 root 20 0 736788 62396 3512 R 99.7 0.1 0:10.38 glusterfs
5152 root 20 0 736788 62396 3512 R 20.3 0.1 0:02.10 glusterfs
5150 root 20 0 736788 62396 3512 S 20.2 0.1 0:02.16 glusterfs
5153 root 20 0 736788 62396 3512 S 19.9 0.1 0:02.08 glusterfs
5151 root 20 0 736788 62396 3512 S 19.4 0.1 0:02.03 glusterfs
5198 root 20 0 53752 19484 816 S 4.4 0.0 0:00.45 iozone
5184 root 20 0 53752 19488 816 S 4.3 0.0 0:00.44 iozone
[...]
</quote>
pstat shows that the hot thread is the fuse thread:
<quote>
[...]
37 0x00007eff9f18c3c0 in meta_writev () from
/usr/lib64/glusterfs/3.8.0/xlator/meta.so
#38 0x00007effa90aea55 in fuse_write_resume () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#39 0x00007effa90a55d5 in fuse_resolve_all () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#40 0x00007effa90a52e8 in fuse_resolve () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#41 0x00007effa90a561e in fuse_resolve_all () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#42 0x00007effa90a4903 in fuse_resolve_continue () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#43 0x00007effa90a5266 in fuse_resolve () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#44 0x00007effa90a55fe in fuse_resolve_all () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#45 0x00007effa90a5640 in fuse_resolve_and_resume () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#46 0x00007effa90bb7be in fuse_thread_proc () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#47 0x00007effb1659df5 in start_thread () from /lib64/libpthread.so.0
#48 0x00007effb0fa01ad in clone () from /lib64/libc.so.6
</quote>
--
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=kKLty0FOFr&a=cc_unsubscribe
More information about the Bugs
mailing list