[Bugs] [Bug 1349953] thread CPU saturation limiting throughput on write workloads

Wed Jun 29 03:45:04 UTC 2016

https://bugzilla.redhat.com/show_bug.cgi?id=1349953

Pranith Kumar K <pkarampu at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |xhernandez at datalab.es

--- Comment #6 from Pranith Kumar K <pkarampu at redhat.com> ---
(In reply to Manoj Pillai from comment #5)
> Results for distributed iozone test, with 6 clients. Command line:
> iozone -+m ${IOZONE_CONF} -i 0 -w -+n -c -C -e -s 10g -r 64k -t 48
> 
> Volume same as in comment #0.
> 6 servers, 12x(4+2).
> Volume options set:
> cluster.lookup-optimize: on
> server.event-threads: 4
> client.event-threads: 4
> 
> iozone output:
> Children see throughput for 48 initial writers  = 1092648.42 kB/sec
> 
> Now, adding option: performance.client-io-threads on:
> iozone output:
> Children see throughput for 48 initial writers  = 3746335.84 kB/sec
> 
> So, enabling client-io-threads gave us ~3.5x bump in throughput here. 
> 
> top output (on a client) without client-io-threads is given in comment #0.
> Here's what it looks like after.
> 
> <body>
> top - 12:07:08 up 11:37,  5 users,  load average: 1.66, 0.40, 0.17
> Threads: 314 total,   7 running, 307 sleeping,   0 stopped,   0 zombie
> %Cpu(s): 27.1 us, 15.9 sy,  0.0 ni, 54.5 id,  0.0 wa,  0.0 hi,  2.4 si,  0.0
> st
> KiB Mem : 65728904 total, 45210228 free,   973552 used, 19545124 buff/cache
> KiB Swap: 32964604 total, 32964604 free,        0 used. 64288252 avail Mem
> 
>   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
> 22074 root      20   0 1168900 239896   3708 R 88.0  0.4   0:21.89 glusterfs
> 22078 root      20   0 1168900 239896   3708 R 88.0  0.4   0:21.68 glusterfs
> 22076 root      20   0 1168900 239896   3708 S 87.9  0.4   0:21.62 glusterfs
> 22077 root      20   0 1168900 239896   3708 R 87.9  0.4   0:21.68 glusterfs
> 22081 root      20   0 1168900 239896   3708 R 75.5  0.4   0:17.93 glusterfs
> 22075 root      20   0 1168900 239896   3708 S 19.1  0.4   0:04.56 glusterfs
> 22273 root      20   0 1168900 239896   3708 R 18.9  0.4   0:04.48 glusterfs
> 22272 root      20   0 1168900 239896   3708 R 18.8  0.4   0:04.56 glusterfs
> 22274 root      20   0 1168900 239896   3708 S 18.7  0.4   0:04.48 glusterfs
> 22199 root      20   0   53752  19484    816 S  6.2  0.0   0:01.63 iozone
> 22186 root      20   0   53752  18536    820 S  6.1  0.0   0:01.52 iozone
> [...]
> </body>
> 
> The single hot thread is gone. Overall CPU utilization has gone up quite a
> bit (idle is down to ~54%), and that may be a concern for some deployments.

CPU utilization will go up because more threads are doing encoding now. Do you
suggest for us to have a way to throttle that? I also added Xavi CCed Xavi to
the bug to get his inputs.

> 
> Network stats (below is from a client) for the run with client-io-threads
> enabled show that the 10gE link is now being pushed to near its limit, which
> is what we want:
> 12:07:07 PM     IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s   rxcmp/s  
> txcmp/s  rxmcst/s
> 12:07:17 PM      p3p1  76644.80 150502.30  15037.46 1197410.19      0.00    
> 0.00      0.00
> 12:07:17 PM      p3p2      0.00      0.00      0.00      0.00      0.00     
> 0.00      0.00
> 12:07:17 PM        lo      0.00      0.00      0.00      0.00      0.00     
> 0.00      0.00
> 12:07:17 PM       em3      0.00      0.00      0.00      0.00      0.00     
> 0.00      0.00
> 12:07:17 PM       em1      6.80      0.00      0.45      0.00      0.00     
> 0.00      3.00
> 12:07:17 PM       em2      0.00      0.00      0.00      0.00      0.00     
> 0.00      0.00
> 12:07:17 PM       em4      0.00      0.00      0.00      0.00      0.00     
> 0.00      0.00

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=GLrLPrQ0jo&a=cc_unsubscribe