[Bugs] [Bug 1349953] thread CPU saturation limiting throughput on write workloads

Wed Jun 29 03:32:53 UTC 2016

https://bugzilla.redhat.com/show_bug.cgi?id=1349953

--- Comment #5 from Manoj Pillai <mpillai at redhat.com> ---
Results for distributed iozone test, with 6 clients. Command line:
iozone -+m ${IOZONE_CONF} -i 0 -w -+n -c -C -e -s 10g -r 64k -t 48

Volume same as in comment #0.
6 servers, 12x(4+2).
Volume options set:
cluster.lookup-optimize: on
server.event-threads: 4
client.event-threads: 4

iozone output:
Children see throughput for 48 initial writers  = 1092648.42 kB/sec

Now, adding option: performance.client-io-threads on:
iozone output:
Children see throughput for 48 initial writers  = 3746335.84 kB/sec

So, enabling client-io-threads gave us ~3.5x bump in throughput here. 

top output (on a client) without client-io-threads is given in comment #0.
Here's what it looks like after.

<body>
top - 12:07:08 up 11:37,  5 users,  load average: 1.66, 0.40, 0.17
Threads: 314 total,   7 running, 307 sleeping,   0 stopped,   0 zombie
%Cpu(s): 27.1 us, 15.9 sy,  0.0 ni, 54.5 id,  0.0 wa,  0.0 hi,  2.4 si,  0.0 st
KiB Mem : 65728904 total, 45210228 free,   973552 used, 19545124 buff/cache
KiB Swap: 32964604 total, 32964604 free,        0 used. 64288252 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
22074 root      20   0 1168900 239896   3708 R 88.0  0.4   0:21.89 glusterfs
22078 root      20   0 1168900 239896   3708 R 88.0  0.4   0:21.68 glusterfs
22076 root      20   0 1168900 239896   3708 S 87.9  0.4   0:21.62 glusterfs
22077 root      20   0 1168900 239896   3708 R 87.9  0.4   0:21.68 glusterfs
22081 root      20   0 1168900 239896   3708 R 75.5  0.4   0:17.93 glusterfs
22075 root      20   0 1168900 239896   3708 S 19.1  0.4   0:04.56 glusterfs
22273 root      20   0 1168900 239896   3708 R 18.9  0.4   0:04.48 glusterfs
22272 root      20   0 1168900 239896   3708 R 18.8  0.4   0:04.56 glusterfs
22274 root      20   0 1168900 239896   3708 S 18.7  0.4   0:04.48 glusterfs
22199 root      20   0   53752  19484    816 S  6.2  0.0   0:01.63 iozone
22186 root      20   0   53752  18536    820 S  6.1  0.0   0:01.52 iozone
[...]
</body>

The single hot thread is gone. Overall CPU utilization has gone up quite a bit
(idle is down to ~54%), and that may be a concern for some deployments.

Network stats (below is from a client) for the run with client-io-threads
enabled show that the 10gE link is now being pushed to near its limit, which is
what we want:
12:07:07 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s  
txcmp/s  rxmcst/s
12:07:17 PM      p3p1  76644.80 150502.30  15037.46 1197410.19      0.00     
0.00      0.00
12:07:17 PM      p3p2      0.00      0.00      0.00      0.00      0.00     
0.00      0.00
12:07:17 PM        lo      0.00      0.00      0.00      0.00      0.00     
0.00      0.00
12:07:17 PM       em3      0.00      0.00      0.00      0.00      0.00     
0.00      0.00
12:07:17 PM       em1      6.80      0.00      0.45      0.00      0.00     
0.00      3.00
12:07:17 PM       em2      0.00      0.00      0.00      0.00      0.00     
0.00      0.00
12:07:17 PM       em4      0.00      0.00      0.00      0.00      0.00     
0.00      0.00

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=MOKWMcHBee&a=cc_unsubscribe