[Gluster-devel] I/O performance

[Xavi Hernandez]

On Tue, Feb 12, 2019 at 1:30 AM Vijay Bellur <vbellur at redhat.com> wrote:

>
>
> On Tue, Feb 5, 2019 at 10:57 PM Xavi Hernandez <xhernandez at redhat.com>
> wrote:
>
>> On Wed, Feb 6, 2019 at 7:00 AM Poornima Gurusiddaiah <pgurusid at redhat.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Feb 5, 2019, 10:53 PM Xavi Hernandez <xhernandez at redhat.com
>>> wrote:
>>>
>>>> On Fri, Feb 1, 2019 at 1:51 PM Xavi Hernandez <xhernandez at redhat.com>
>>>> wrote:
>>>>
>>>>> On Fri, Feb 1, 2019 at 1:25 PM Poornima Gurusiddaiah <
>>>>> pgurusid at redhat.com> wrote:
>>>>>
>>>>>> Can the threads be categorised to do certain kinds of fops?
>>>>>>
>>>>>
>>>>> Could be, but creating multiple thread groups for different tasks is
>>>>> generally bad because many times you end up with lots of idle threads which
>>>>> waste resources and could increase contention. I think we should only
>>>>> differentiate threads if it's absolutely necessary.
>>>>>
>>>>>
>>>>>> Read/write affinitise to certain set of threads, the other metadata
>>>>>> fops to other set of threads. So we limit the read/write threads and not
>>>>>> the metadata threads? Also if aio is enabled in the backend the threads
>>>>>> will not be blocked on disk IO right?
>>>>>>
>>>>>
>>>>> If we don't block the thread but we don't prevent more requests to go
>>>>> to the disk, then we'll probably have the same problem. Anyway, I'll try to
>>>>> run some tests with AIO to see if anything changes.
>>>>>
>>>>
>>>> I've run some simple tests with AIO enabled and results are not good. A
>>>> simple dd takes >25% more time. Multiple parallel dd take 35% more time to
>>>> complete.
>>>>
>>>
>>>
>>> Thank you. That is strange! Had few questions, what tests are you
>>> running for measuring the io-threads performance(not particularly aoi)? is
>>> it dd from multiple clients?
>>>
>>
>> Yes, it's a bit strange. What I see is that many threads from the thread
>> pool are active but using very little CPU. I also see an AIO thread for
>> each brick, but its CPU usage is not big either. Wait time is always 0 (I
>> think this is a side effect of AIO activity). However system load grows
>> very high. I've seen around 50, while on the normal test without AIO it's
>> stays around 20-25.
>>
>> Right now I'm running the tests on a single machine (no real network
>> communication) using an NVMe disk as storage. I use a single mount point.
>> The tests I'm running are these:
>>
>>    - Single dd, 128 GiB, blocks of 1MiB
>>    - 16 parallel dd, 8 GiB per dd, blocks of 1MiB
>>    - fio in sequential write mode, direct I/O, blocks of 128k, 16
>>    threads, 8GiB per file
>>    - fio in sequential read mode, direct I/O, blocks of 128k, 16
>>    threads, 8GiB per file
>>    - fio in random write mode, direct I/O, blocks of 128k, 16 threads,
>>    8GiB per file
>>    - fio in random read mode, direct I/O, blocks of 128k, 16 threads,
>>    8GiB per file
>>    - smallfile create, 16 threads, 256 files per thread, 32 MiB per file
>>    (with one brick down, for the following test)
>>    - self-heal of an entire brick (from the previous smallfile test)
>>    - pgbench init phase with scale 100
>>
>> I run all these tests for a replica 3 volume and a disperse 4+2 volume.
>>
>
>
> Are these performance results available somewhere? I am quite curious to
> understand the performance gains on NVMe!
>

I'm updating test results with the latest build. I'll report it here once
it's complete.

Xavi

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190213/be58620b/attachment-0001.html>