[Gluster-users] why i have a good write iops and poor read iops with more stripes?

[Brian Foster]

On 11/08/2012 09:15 PM, 肖力 wrote:
> 
> I need  high random read and write iops in use   small files , it means high 4k random read and write iops,not big files read and write high speed.
> I want to find a way use more pc servers ,more  harddisks , more  nics to improve performance.
> I think there is many way to choose :
> raid or no raid？ 
> Is more stripe  more  performance ?
> How to balance data safe and  performance ? 
> I test 2 weeks,  choose diffenrent servers ,stripe group , raid and no raid ...
> be honest , I have not find a goog way  , i get stuck ,can you offer me some suggestions ? tks
> 

The stripe layout might give you better performance with larger files
(it breaks files into 128kB chunks by default), but smaller files might
all localize to the same brick. For example, if you created a bunch of
4k files in your volume, that data all lands on the first brick.

You could try to reduce the 'block-size' stripe option. It looks like
the minimum value is 16kB. If that doesn't help, your best bet in terms
of iops across many small files might be a basic distribute volume.

If you have the hardware, you might also want to try some testing
against ramdisk backed bricks. My understanding is that our latencies
are such that the backend storage is usually not the bottleneck for iops
intensive workloads (e.g., see my previous comment about driving a
native client with multiple threads), so perhaps that would help you
make a quicker assessment on whether it's in the ballpark of your
requirements.

Brian

>  
> 
> 
> 
> 
> 
> 
> 在 2012-11-09 06:58:17，"Brian Foster" <bfoster at redhat.com> 写道：
>> On 11/07/2012 09:06 PM, 肖力 wrote:
>>> Hi ,I have 4 dell 2970 server , three server  harddisk is 146Gx6 ,one hard disk is 72Gx6:
>>>
>>> I Want to test different stripe iops, and i test result is :
>>>
>>> ----------------------------------------------------------------------
>>>
>>> no stripe 
>>>
>>> gluster volume create test-volume   transport tcp \
>>> 172.16.20.231:/exp2  \
>>> 172.16.20.232:/exp2 \
>>> 172.16.20.233:/exp2  \
>>>
>>> 4k 100 % random write  288
>>>
>>> 4k 100 % random read  264
>>>
>>> ----------------------------------------------------------------------
>>>
>>> 2 stripe
>>>
>>> gluster volume create test-volume   transport tcp \
>>> 172.16.20.231:/exp2  \
>>> 172.16.20.232:/exp2 \
>>> 172.16.20.233:/exp2  \
>>>
>>
>> This looks the same as the "no stripe" case. Given your numbers differ,
>> I presume a copy/paste error?
> i check my record ,it is correct , I can test it again if i have time.
>>
>>> 4k 100 % random write  439
>>>
>>> 4k 100 % random read  241
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> 6 stripe
>>>
>>> gluster volume create test-volume3 stripe 6 transport tcp \
>>> 172.16.20.231:/exp5 172.16.20.231:/exp6 \
>>> 172.16.20.232:/exp5 172.16.20.232:/exp6 \
>>> 172.16.20.233:/exp5 172.16.20.233:/exp6 \
>>>
>>
>> Are you adding bricks on the same spindle(s) here? For example, are
>> 172.16.20.231:/exp{5,6} on the same set of drives? If so, the extra
>> bricks might not be buying you anything.
>>
> That's not clear , i use default cmd ,i don't know how gulster control it.
> 
>>> 4k 100 % random write  683
>>>
>>> 4k 100 % random read  151
>>>
>>> ----------------------------------------------------------------------
>>>
>>> My question is why more stripes more write good iops and more poor read iops?
>>>
>>
>> I don't have a specific answer, but I ran a few similar random read
>> tests to remind myself of the behavior here. One thing our performance
>> guys have pointed out is that you'll want to run multiple threads
>> against a gluster native client to maximize iops. Perhaps you're doing
>> that, but you haven't provided much detail about the actual test you're
>> running.
>>
>> Along with that, there is a known bottleneck in the client that is
>> alleviated by using the 'gid-timeout' mount option (i.e., -o
>> gid-timeout=1').
> Ok, i test gid-timeout=1  thks
> 
> 
>>
>> Brian
>>
>>> thks
>>>
>>> xiao li
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>
>>