[Gluster-devel] [Gluster-users] Regarding the write performance in replica 1 volume in 1Gbps Ethernet, get about 50MB/s while writing single file.

Jaden Liang jaden1q84 at gmail.com
Thu Sep 4 01:20:36 UTC 2014

Hi Ramon,

I am running on gluster FUSE client.

I maybe not stat clearly my testing environment. Let me explain. The volume
is configured on 2 servers. There is no replication at all, just
distributed volume. So I don't think it is the replicated data issue.
Actually, we can reach 100MB/s when writing mutiple files at the same time.

Here is the volume info:

# gluster volume info vs_vol_rep1

Volume Name: vs_vol_rep1
Type: Distribute
Volume ID: cd137b57-e98a-4755-939a-7fc578f2a8c0
Status: Started
Number of Bricks: 10
Transport-type: tcp
Options Reconfigured:
diagnostics.latency-measurement: on
nfs.disable: on

On Wednesday, September 3, 2014, Ramon Selga <ramon.selga at gmail.com> wrote:

>  Hi Jaden,
> May I ask some more info about your setup?
> Are you using NFS client or gluster FUSE client?
> If you are using NFS Client write data goes to one of nodes of replica
> pair and that node sends write replica data to the other node. If you are
> using one switch for client and server connections and one 1GbE port on
> each device, data received in the first node is re-sended to the other node
> simultaneously and, in theory, you may reach speeds closer to 100MBps.
> In case of gluster FUSE Client, write data goes simultaneously to both
> server nodes using half bandwidth for each of the client's 1GbE port
> because replica is done by client side, that results on a writing speed
> around 50MBps(<60MBps).
> I hope this helps.
> El 03/09/14 a les 07:02, Jaden Liang ha escrit:
> Hi all,
>  We did some more tests and analysis yesterday. It looks like 50MB/s is
> the top theoretical speed in replica 1 volume over 1Gbps network. GlusterFS
> write 128KB data once a block, then wait for return. The 128KB data would
> cost about 1ms in 1Gbps network. And in the server-side, it took about
> 800us to 1000us to write 128KB to the HDD and return. Plus some other 100us
> to 200us time elapsed. GlusterFS would take about 2ms-2.2ms to finish a
> 128KB block data writing, which is about 50MB/s.
>  The question is that why don't glusterfs use pipeline writting or
> reading to speed up this chatty process?
> On Tuesday, September 2, 2014, Jaden Liang <jaden1q84 at gmail.com
> <javascript:_e(%7B%7D,'cvml','jaden1q84 at gmail.com');>> wrote:
>> Hello, gluster-devel and gluster-users team,
>>  We are running a performance test in a replica 1 volume and find out
>> the single file sequence writing performance only get about 50MB/s in a
>> 1Gbps Ethernet. However, if we test multiple files sequence writing, the
>> writing performance can go up to 120MB/s which is the top speed of network.
>>  We also tried to use the stat xlator to find out where is the
>> bottleneck of single file write performance. Here is the stat data:
>>  Client-side:
>> ......
>>  vs_vol_rep1-client-8.latency.WRITE=total:21834371.000000us,
>> mean:2665.328491us, count:8192, max:4063475, min:1849
>>  ......
>>  Server-side:
>> ......
>>  /data/sdb1/brick1.latency.WRITE=total:6156857.000000us,
>> mean:751.569458us, count:8192, max:230864, min:611
>>  ......
>>  Note that the test is write a 1GB single file sequentially to a replica
>> 1 volume through 1Gbps Ethernet network.
>>  On the client-side, we can see there are 8192 write requests totally.
>> Every request will write 128KB data. Total eclipsed time is 21834371us,
>> about 21 seconds. The mean time of request is 2665us, about 2.6ms which
>> means it could only serves about 380 requests in 1 seconds. Plus there are
>> other time consuming like statfs, lookup, but those are not major reasons.
>>  On the server-side, the mean time of request is 751us include write
>> data to HDD disk. So we think that is not the major reason.
>>  And we also modify some codes to do the statistic of system epoll
>> elapsed time. It only took about 20us from enqueue data to finish sent-out.
>>  Now we are heading to the rpc mechanism in glusterfs. Still, we think
>> this issue maybe encountered in gluster-devel or gluster-users teams.
>> Therefor, any suggestions would be grateful. Or have anyone know such issue?
>>  Best regards,
>> Jaden Liang
>> 9/2/2014
>> --
>>  Best regards,
>> Jaden Liang
> _______________________________________________
> Gluster-devel mailing listGluster-devel at gluster.org <javascript:_e(%7B%7D,'cvml','Gluster-devel at gluster.org');>http://supercolony.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20140904/d03433a2/attachment.html>

More information about the Gluster-devel mailing list