[Gluster-devel] I/O performance

[Vijay Bellur]

On Tue, Feb 5, 2019 at 10:57 PM Xavi Hernandez <xhernandez at redhat.com>
wrote:

> On Wed, Feb 6, 2019 at 7:00 AM Poornima Gurusiddaiah <pgurusid at redhat.com>
> wrote:
>
>>
>>
>> On Tue, Feb 5, 2019, 10:53 PM Xavi Hernandez <xhernandez at redhat.com
>> wrote:
>>
>>> On Fri, Feb 1, 2019 at 1:51 PM Xavi Hernandez <xhernandez at redhat.com>
>>> wrote:
>>>
>>>> On Fri, Feb 1, 2019 at 1:25 PM Poornima Gurusiddaiah <
>>>> pgurusid at redhat.com> wrote:
>>>>
>>>>> Can the threads be categorised to do certain kinds of fops?
>>>>>
>>>>
>>>> Could be, but creating multiple thread groups for different tasks is
>>>> generally bad because many times you end up with lots of idle threads which
>>>> waste resources and could increase contention. I think we should only
>>>> differentiate threads if it's absolutely necessary.
>>>>
>>>>
>>>>> Read/write affinitise to certain set of threads, the other metadata
>>>>> fops to other set of threads. So we limit the read/write threads and not
>>>>> the metadata threads? Also if aio is enabled in the backend the threads
>>>>> will not be blocked on disk IO right?
>>>>>
>>>>
>>>> If we don't block the thread but we don't prevent more requests to go
>>>> to the disk, then we'll probably have the same problem. Anyway, I'll try to
>>>> run some tests with AIO to see if anything changes.
>>>>
>>>
>>> I've run some simple tests with AIO enabled and results are not good. A
>>> simple dd takes >25% more time. Multiple parallel dd take 35% more time to
>>> complete.
>>>
>>
>>
>> Thank you. That is strange! Had few questions, what tests are you running
>> for measuring the io-threads performance(not particularly aoi)? is it dd
>> from multiple clients?
>>
>
> Yes, it's a bit strange. What I see is that many threads from the thread
> pool are active but using very little CPU. I also see an AIO thread for
> each brick, but its CPU usage is not big either. Wait time is always 0 (I
> think this is a side effect of AIO activity). However system load grows
> very high. I've seen around 50, while on the normal test without AIO it's
> stays around 20-25.
>
> Right now I'm running the tests on a single machine (no real network
> communication) using an NVMe disk as storage. I use a single mount point.
> The tests I'm running are these:
>
>    - Single dd, 128 GiB, blocks of 1MiB
>    - 16 parallel dd, 8 GiB per dd, blocks of 1MiB
>    - fio in sequential write mode, direct I/O, blocks of 128k, 16
>    threads, 8GiB per file
>    - fio in sequential read mode, direct I/O, blocks of 128k, 16 threads,
>    8GiB per file
>    - fio in random write mode, direct I/O, blocks of 128k, 16 threads,
>    8GiB per file
>    - fio in random read mode, direct I/O, blocks of 128k, 16 threads,
>    8GiB per file
>    - smallfile create, 16 threads, 256 files per thread, 32 MiB per file
>    (with one brick down, for the following test)
>    - self-heal of an entire brick (from the previous smallfile test)
>    - pgbench init phase with scale 100
>
> I run all these tests for a replica 3 volume and a disperse 4+2 volume.
>


Are these performance results available somewhere? I am quite curious to
understand the performance gains on NVMe!

Thanks,
Vijay
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190211/a078fc27/attachment-0001.html>