[Gluster-users] small write speed problem on EBS, distributed replica

[Mohit Anchlia]

It will be good for dev team to look at it in parallel. It will help others too.

First thing that I see is that your network bandwidth sucks. Is it
1GigE? When you run tools like iperf you atleast expect to see close
to 800MB/s. for eg: in my env if I run iperf I get something like:

------------------------------------------------------------
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  6] local 10.1.101.193 port 49503 connected with 10.1.101.149 port 5001
[  4]  0.0-10.0 sec    975 MBytes    815 Mbits/sec
[  5] local 10.1.101.193 port 5001 connected with 10.1.101.149 port 41642

Can you also try another dd test directly on the gluster server where
volume is and post the results?

Regarding other perf related questions I haven't myself tries those
yet so I think you will need to change one at a time and expirement
with it. But if there is a inherent perf problem with the server and
underlying storage then those may not be that helpful.

On Thu, Mar 24, 2011 at 3:55 AM, karol skocik <karol.skocik at gmail.com> wrote:
> Hi Vikas, Mohit,
>  I should disclose our typical use cases:
> We need to read and write files of size several 100s of MBs - the
> ratio of read : write is about 1:1.
>
>> What did you use to calculate latency?
>
> I used http://www.bitmover.com/lmbench they have a tool "lat_tcp".
>
> Numbers below are from lmbench tool "bw_tcp":
>
>> Network bandwidths:
>> dfs01: 54 MB/s
>> dfs02: 62.5 MB/s
>> dfs03: 64 MB/s
>> dfs04: 91.5 MB/s
>
> The setup is Gluster native, no NFS.
>
> About the "Optimizing Gluster" link - I have seen it before, but there
> are several things I don't understand:
>
> 1.) Tuning FUSE to use larger blocksize - when testing PVFS, we
> achieved best performance with bs = 4MB.
> It's hard to understand why it's hardcoded to 128 KB.
> Also I have read somewhere else (referencing FUSE) - that larger
> blocksize doesn't yield more performance.
> I guess when transfering larger amount of data on network with
> significant latency,
> a lot less IO requests should result in higher throughput. (And it's
> cheaper also on EBS).
>
> Are those listed adjustments to FUSE kernel modules still applicable?
>
> 2.) Enabling direct-io mode
> Does this work on current 3.1.2? :
>
> glusterfs --direct-io-mode=write-only -f <spec-file> <mount-point>
>
> also with --direct-io-mode=read-write ?
>
> Of those parameters in "Setting Volume Options", could this one help:
> - performance.write-behind-window-size - increasing 10-20 times?
>
> Now, the raw block device throughput (dd if=/dev/zero
> of=/path/to/ebs/mount bs=128k count=4096 oflag=direct)
> 3 measurements on server machines dfs0[1-4]:
>
> dfs01: 9.0 MB/s, 16.4 MB/s, 18.4 MB/s
> dfs02: 26.0 MB/s, 28.5 MB/s, 13.0 MB/s
> dfs03: 14.4 MB/s, 11.8 MB/s, 32.6 MB/s
> dfs04: 35.5 MB/s, 33.1 MB/s, 31.9 MB/s
>
> This, indeed, varies considerably!
>
> Thanks for help.
> Karol
>
>
> On Wed, Mar 23, 2011 at 7:06 PM, Vikas Gorur <vikas at gluster.com> wrote:
>> Karol,
>>
>> A few general pointers about EBS performance:
>>
>> We've seen throughput to an EBS volume vary considerably. Since EBS is iSCSI underneath, throughput to a volume can fluctuate, and it is also possible that your instance is on degraded hardware that gets very low throughput to the volume.
>>
>> So I would advise you to first gather some data about all your EBS volumes. You can measure throughput to them by doing something like:
>>
>> dd if=/dev/zero of=/path/to/ebs/mount bs=128k count=4096 oflag=direct
>>
>> The "oflag=direct" will give us the raw block device throughput, without the kernel cache in the way.
>>
>> The performance you see on the Gluster mountpoint will be a function of the EBS performance. You might also want to spin up a couple more instances and see their EBS throughput to get an idea of the range of EBS performance.
>>
>> Doing a RAID0 of 4 or 8 EBS volumes using mdadm will also help you increase performance.
>>
>> ------------------------------
>> Vikas Gorur
>> Engineer - Gluster, Inc.
>> ------------------------------
>>
>>
>>
>>
>>
>>
>>
>>
>>
>