[Gluster-users] slow write perf for disperse volume

[Ingard Mevåg]

I've done some more testing with tc and introduced latency on one of my
testservers. With 9ms latency artificially introduced using tc ( sudo tc
qdisc add dev bond0 root netem delay 9ms ) to a testserver in the same DC
as the disperse volume servers I get more or less the same throughput as I
do when testing DC1 <-> DC2 (which has ~9ms ping).

I know distribute volumes were more sensitive to latency in the past. At
least I can max out a 1gig link with 9-10ms latency when using distribute.
Disperse seems to max at 12-14MB/s with 8-10ms latency.

ingard

2017-04-24 14:03 GMT+02:00 Ingard Mevåg <ingard at jotta.no>:

> I can confirm mounting the disperse volume locally on one of the three
> servers i got 211 MB/s with dd if=/dev/zero of=./local.dd.test bs=1M
> count=10000.
>
> Its not very good concidering 10gig network, but at least 20x better than
> 10-12MB/s
>
> 2017-04-24 13:53 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com>:
>
>> +Ashish
>>
>> Ashish,
>>        Could you help Ingard? Do let me know what you find.
>>
>> On Mon, Apr 24, 2017 at 4:50 PM, Ingard Mevåg <ingard at jotta.no> wrote:
>>
>>> Hi. I can't see a fuse thread at all. Please see attached screenshot of
>>> top process with threads. Keep in mind this is from inside the container.
>>>
>>> 2017-04-24 12:17 GMT+02:00 Pranith Kumar Karampuri <pkarampu at redhat.com>
>>> :
>>>
>>>> We were able to saturate hardware with EC as well. Could you check
>>>> 'top' in threaded mode to see if fuse thread is saturated when you run dd?
>>>>
>>>> On Mon, Apr 24, 2017 at 3:27 PM, Ingard Mevåg <ingard at jotta.no> wrote:
>>>>
>>>>> Hi
>>>>> I've been playing with disperse volumes the past week, and so far i
>>>>> can not get more than 12MB/s when i do a write test. I've tried a
>>>>> distributed volume on the same bricks and gotten close to gigabit speeds.
>>>>> iperf confirms gigabit speeds to all three servers in the storage pool.
>>>>>
>>>>> The three storage servers have 10gig nics (connected to the same
>>>>> switch). The client is for a now a docker container in a 2nd DC (latency
>>>>> roughly 8-9 ms).
>>>>>
>>>>> dpkg -l|grep -i gluster
>>>>> ii  glusterfs-client               3.10.1-ubuntu1~xenial1
>>>>>  amd64        clustered file-system (client package)
>>>>> ii  glusterfs-common               3.10.1-ubuntu1~xenial1
>>>>>  amd64        GlusterFS common libraries and translator modules
>>>>> ii  glusterfs-server               3.10.1-ubuntu1~xenial1
>>>>>  amd64        clustered file-system (server package)
>>>>>
>>>>> $ gluster volume info
>>>>>
>>>>> Volume Name: DFS-ARCHIVE-001
>>>>> Type: Disperse
>>>>> Volume ID: 1497bc85-cb47-4123-8f91-a07f55c11dcc
>>>>> Status: Started
>>>>> Snapshot Count: 0
>>>>> Number of Bricks: 1 x (4 + 2) = 6
>>>>> Transport-type: tcp
>>>>> Bricks:
>>>>> Brick1: dna-001:/mnt/data01/brick
>>>>> Brick2: dna-001:/mnt/data02/brick
>>>>> Brick3: dna-002:/mnt/data01/brick
>>>>> Brick4: dna-002:/mnt/data02/brick
>>>>> Brick5: dna-003:/mnt/data01/brick
>>>>> Brick6: dna-003:/mnt/data02/brick
>>>>> Options Reconfigured:
>>>>> transport.address-family: inet
>>>>> nfs.disable: on
>>>>>
>>>>> Anyone know the reason for the slow speeds on disperse vs distribute?
>>>>>
>>>>> kind regards
>>>>> ingard
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Pranith
>>>>
>>>
>>>
>>>
>>> --
>>> Ingard Mevåg
>>> Driftssjef
>>> Jottacloud
>>>
>>> Mobil: +47 450 22 834 <+47%20450%2022%20834>
>>> E-post: ingard at jottacloud.com
>>> Webside: www.jottacloud.com
>>>
>>
>>
>>
>> --
>> Pranith
>>
>
>
>
> --
> Ingard Mevåg
> Driftssjef
> Jottacloud
>
> Mobil: +47 450 22 834 <+47%20450%2022%20834>
> E-post: ingard at jottacloud.com
> Webside: www.jottacloud.com
>



-- 
Ingard Mevåg
Driftssjef
Jottacloud

Mobil: +47 450 22 834
E-post: ingard at jottacloud.com
Webside: www.jottacloud.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170424/5d5a44d9/attachment.html>