[Bugs] [Bug 1499649] New: Need option to control number of client-io-threads

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1499649

            Bug ID: 1499649
           Summary: Need option to control number of client-io-threads
           Product: GlusterFS
           Version: 3.12
         Component: io-threads
          Assignee: bugs at gluster.org
          Reporter: mpillai at redhat.com
                CC: bugs at gluster.org



Description of problem:

The option "performance.client-io-threads on" is useful in situations where the
fuse-thread bottlenecks. We see such situations often with disperse volumes>
But we also see it with other volume types, esp. on small random I/O.

The number of client io-threads started is controlled by the option
performance.io-thread-count, which by default is 16. That number is generally
too high for client-io-threads. However, reducing io-thread-count can introduce
bottlenecks on the brick side, since the same option controls number of
io-threads on both client-side and brick. 

Ideally, we would have a separate option for controlling the number of
client-side io-threads.

Version-Release number of selected component (if applicable):
glusterfs*-3.12.1-2.el7.x86_64


Actual results:

I have a single-brick distribute volume created on ramdisk. On this I run a
random I/O read workload with 4k block size.

Results with different options:

performance.client-io-threads off: 63.9MiB/s i.e approx. 16k iops
[in the above, the fuse thread is the bottleneck at 99% CPU utilization]


[impact of turning client-io-threads on, without tuning io-thread-count, which
by default is 16. mutrace seemed to report contention among the
client-io-threads in this case; need to dig more into that]
performance.client-io-threads on : 98.5MiB/s

performance.client-io-threads on,
performance.io-thread-count 2    : 104MiB/s

performance.client-io-threads on,
performance.io-thread-count 1    : 69.7MiB/s

In the run with io-thread-count=1, there seems to be a bottleneck at the brick:
  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND      
 9263 root      20   0  682268  20792   3872 R 99.9  0.0   1:00.43
glusteriotwr0                              
 9166 root      20   0 1248736  28304   4956 R 50.0  0.1   0:29.33
glusteriotwr0 

Ideally, we would like to set number of client-io-threads to 1 or 2: just
enough to eliminate the fuse-thread bottleneck, but not so high that it
introduces contention among the client-io-threads. But depending on the nature
of the brick device we may not want to reduce io-thread-count.


Additional info:

Thanks to Ravi, for clarifying io-threads behavior and providing the pointer:
https://github.com/gluster/glusterfs/commit/09232fd6855f288c47b5396dcd4d4245a154576f

Krutika also reports that a high number of client-io-threads is not needed:
https://bugzilla.redhat.com/show_bug.cgi?id=1467614#c16. Some of the other
observations in this bz differs from hers though.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.