[Bugs] [Bug 1349953] thread CPU saturation limiting throughput on write workloads

bugzilla at redhat.com bugzilla at redhat.com
Fri Jul 8 08:03:41 UTC 2016


https://bugzilla.redhat.com/show_bug.cgi?id=1349953



--- Comment #17 from Manoj Pillai <mpillai at redhat.com> ---

Results with 3x2 volume (6 servers) with bricks on 12-disk RAID-6:

Benchmark: distributed iozone (write test) on 6 clients:
iozone -+m ${IOZONE_CONF} -i 0 -w -+n -c -C -e -s 10g -r 64k -t 48

With standard tuning options:
cluster.lookup-optimize: on
server.event-threads: 4
client.event-threads: 4

iozone output:
throughput for 48 initial writers  = 2052985.17 kB/sec

iozone output after adding option performance.client-io-threads on:
throughput for 48 initial writers  = 3180628.89 kB/sec

So we see a big bump in the case of replica-2 as well with client-io-threads.
So we really want this option on irrespective of volume type.

Without client-io-threads, we see the following in top output on the clients:
<quote>
top - 02:49:26 up 21:08,  0 users,  load average: 0.24, 0.47, 0.47
Threads: 296 total,   4 running, 292 sleeping,   0 stopped,   0 zombie
%Cpu(s):  7.5 us,  9.2 sy,  0.0 ni, 82.6 id,  0.0 wa,  0.0 hi,  0.8 si,  0.0 st
KiB Mem : 65728904 total, 59729480 free,   758744 used,  5240680 buff/cache
KiB Swap: 32964604 total, 32964604 free,        0 used. 64684248 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
 5156 root      20   0  736788  62396   3512 R 99.7  0.1   0:10.38 glusterfs
 5152 root      20   0  736788  62396   3512 R 20.3  0.1   0:02.10 glusterfs
 5150 root      20   0  736788  62396   3512 S 20.2  0.1   0:02.16 glusterfs
 5153 root      20   0  736788  62396   3512 S 19.9  0.1   0:02.08 glusterfs
 5151 root      20   0  736788  62396   3512 S 19.4  0.1   0:02.03 glusterfs
 5198 root      20   0   53752  19484    816 S  4.4  0.0   0:00.45 iozone
 5184 root      20   0   53752  19488    816 S  4.3  0.0   0:00.44 iozone
[...]
</quote>

pstat shows that the hot thread is the fuse thread:
<quote>
[...]
37 0x00007eff9f18c3c0 in meta_writev () from
/usr/lib64/glusterfs/3.8.0/xlator/meta.so
#38 0x00007effa90aea55 in fuse_write_resume () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#39 0x00007effa90a55d5 in fuse_resolve_all () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#40 0x00007effa90a52e8 in fuse_resolve () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#41 0x00007effa90a561e in fuse_resolve_all () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#42 0x00007effa90a4903 in fuse_resolve_continue () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#43 0x00007effa90a5266 in fuse_resolve () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#44 0x00007effa90a55fe in fuse_resolve_all () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#45 0x00007effa90a5640 in fuse_resolve_and_resume () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#46 0x00007effa90bb7be in fuse_thread_proc () from
/usr/lib64/glusterfs/3.8.0/xlator/mount/fuse.so
#47 0x00007effb1659df5 in start_thread () from /lib64/libpthread.so.0
#48 0x00007effb0fa01ad in clone () from /lib64/libc.so.6
</quote>

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=kKLty0FOFr&a=cc_unsubscribe


More information about the Bugs mailing list