[Gluster-devel] [Gluster-users] Regarding the write performance in replica 1 volume in 1Gbps Ethernet, get about 50MB/s while writing single file.

[Roman]

I am interested in this also, so I'll add myself here too :)
3.5.2, same speeds for replica 1 volume: near 50 MB/s or 400 Mbps.
Servers are connected using 10g interfaces, clients using 1gbps.



2014-09-03 8:02 GMT+03:00 Jaden Liang <jaden1q84 at gmail.com>:

> Hi all,
>
> We did some more tests and analysis yesterday. It looks like 50MB/s is the
> top theoretical speed in replica 1 volume over 1Gbps network. GlusterFS
> write 128KB data once a block, then wait for return. The 128KB data would
> cost about 1ms in 1Gbps network. And in the server-side, it took about
> 800us to 1000us to write 128KB to the HDD and return. Plus some other 100us
> to 200us time elapsed. GlusterFS would take about 2ms-2.2ms to finish a
> 128KB block data writing, which is about 50MB/s.
>
> The question is that why don't glusterfs use pipeline writting or reading
> to speed up this chatty process?
>
>
> On Tuesday, September 2, 2014, Jaden Liang <jaden1q84 at gmail.com> wrote:
>
>> Hello, gluster-devel and gluster-users team,
>>
>> We are running a performance test in a replica 1 volume and find out the
>> single file sequence writing performance only get about 50MB/s in a 1Gbps
>> Ethernet. However, if we test multiple files sequence writing, the writing
>> performance can go up to 120MB/s which is the top speed of network.
>>
>> We also tried to use the stat xlator to find out where is the bottleneck
>> of single file write performance. Here is the stat data:
>>
>> Client-side:
>> ......
>> vs_vol_rep1-client-8.latency.WRITE=total:21834371.000000us,
>> mean:2665.328491us, count:8192, max:4063475, min:1849
>> ......
>>
>> Server-side:
>> ......
>> /data/sdb1/brick1.latency.WRITE=total:6156857.000000us,
>> mean:751.569458us, count:8192, max:230864, min:611
>> ......
>>
>> Note that the test is write a 1GB single file sequentially to a replica 1
>> volume through 1Gbps Ethernet network.
>>
>> On the client-side, we can see there are 8192 write requests totally.
>> Every request will write 128KB data. Total eclipsed time is 21834371us,
>> about 21 seconds. The mean time of request is 2665us, about 2.6ms which
>> means it could only serves about 380 requests in 1 seconds. Plus there are
>> other time consuming like statfs, lookup, but those are not major reasons.
>>
>> On the server-side, the mean time of request is 751us include write data
>> to HDD disk. So we think that is not the major reason.
>>
>> And we also modify some codes to do the statistic of system epoll elapsed
>> time. It only took about 20us from enqueue data to finish sent-out.
>>
>> Now we are heading to the rpc mechanism in glusterfs. Still, we think
>> this issue maybe encountered in gluster-devel or gluster-users teams.
>> Therefor, any suggestions would be grateful. Or have anyone know such issue?
>>
>> Best regards,
>> Jaden Liang
>> 9/2/2014
>>
>>
>> --
>> Best regards,
>> Jaden Liang
>>
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>



-- 
Best regards,
Roman.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140903/1ad5705a/attachment.html>