[Gluster-users] Writing performance gets slower when writethreadsare many

[Maik Kulbe]

-----Original Mail-----
From: Jung Young Seok [jung.youngseok at gmail.com]
Sent: 22.08.13 - 19:09:32
To: Maik Kulbe [info at linux-web-development.de]
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Writing performance gets slower when writethreadsare many

> 2013/8/23 Maik Kulbe <info at linux-web-development.de>
>
> -----Original Mail-----
> From: Jung Young Seok [jung.youngseok at gmail.com]
> Sent: 22.08.13 - 17:19:32
> To: gluster-users at gluster.org
>
> Subject: [Gluster-users] Writing performance gets slower when write
> threadsare many
>
> Dear gluster users,
>
> We have set up  glusterfs 3.4 on cent 6. 2 Gluster FS servers are
> replicated each (2brick x 2).
> Glusterfs is mounted on WAS as fuse.
>
> We realized when there are more thread to write files on gluster fs,
> it
> gets slower on writing.
>
> We've set 250 thread to write 1M file each repeatedly. In this case we
> get
> under 30 TPS. No NW bandwidth problem.
> Is there any way to avoid performance degradation when there are lots
> of
> write threads?
>
> That depends on a lot of factors - what interconnect are you using? 1GE?
> 10GE? RDMA? What are your bricks built on? SATA? SSD? RAID?
>
> My first guess would be that interconnect latency and a kernel maximum
> for some connection count could be the problem.
>
>  
> Thank you for the comments,
> We use 1GE for storage network. When I checked the net stream using iftop,
> it never reached 1G. (only was under 500Mbps. Since we have 2 glusterfs
> servers, 250Mb for first brick 250Mb for second brick)
> When I scp a file to Gluster servers from WAS directly ,  it reached
> 80MB/s each. (used less encrypted option)
> About HDD, 
> First server is on SAS 10k RAID1.
> Second server is on SATA 10k RAID1.

Have you benchmarked both network and harddisks without gluster? I would especially look at the network timings and there behaviour with multiple connections between clients and gluster servers. From my experience I would say too high latencies could very quickly become a bottleneck for the whole system.

> Those are for test. We'll use SATA RAID1 both finally to reduce cost.
> We are using CentOS 6.4. Since we use vanilla installation, most of kernel
> parameters are defaults.
> If you give me any option that I can test, it would be great.

There is an article on that in the Gluster docs: http://www.gluster.org/community/documentation/index.php/Linux_Kernel_Tuning

You can also look at the performance options that Gluster provides. Those are in the Gluster administration guide: http://www.gluster.org/wp-content/uploads/2012/05/Gluster_File_System-3.3.0-Administration_Guide-en-US.pdf - there are some caching mechanisms, etc. that might increase your write rate.

> Thank you.
>  
>
> Look forward any help.
> Thanks.
>
> Regards,
> Youngseok jung