[Gluster-users] Slow performance - 4 hosts, 10 gigabit ethernet, Gluster 3.2.3

[Thomas Jackson]

Hi Pavan,

Thanks for the reply - my comments inline below

Regards,

Thomas

-----Original Message-----
From: Pavan T C [mailto:tcp at gluster.com] 
Sent: Wednesday, 14 September 2011 9:19 PM
To: Thomas Jackson
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Slow performance - 4 hosts, 10 gigabit
ethernet, Gluster 3.2.3

> On Friday 09 September 2011 10:30 AM, Thomas Jackson wrote:
>> Hi everyone,

> Hello Thomas,

> Try the following:

> 1. In the fuse volume file, try:

> Under write-behind:
> "option cache-size 16MB"

> Under read-ahead:
> "option page-count 16"

> Under io-cache:
> "option cache-size=64MB"

TJ: Results here are not pretty!
	root at my-host:~# dd if=/dev/zero of=/mnt/cluster-volume/test.file
bs=1M count=10000
	10485760000 bytes (10 GB) copied, 107.888 s, 97.2 MB/s


> 2. Did you get 9Gbits/Sec with iperf with a single thread or multiple
threads?

TJ: Single thread

> 3. Can you give me the output of:
> sysctl -a | egrep 'rmem|wmem'

TJ: root at my-host:~# sysctl -a | egrep 'rmem|wmem'
	error: permission denied on key 'vm.compact_memory'
	vm.lowmem_reserve_ratio = 256   256     32
	error: permission denied on key 'net.ipv4.route.flush'
	net.core.wmem_max = 131071
	net.core.rmem_max = 131071
	net.core.wmem_default = 126976
	net.core.rmem_default = 126976
	net.ipv4.tcp_wmem = 4096        16384   4194304
	net.ipv4.tcp_rmem = 4096        87380   4194304
	net.ipv4.udp_rmem_min = 4096
	net.ipv4.udp_wmem_min = 4096
	error: permission denied on key 'net.ipv6.route.flush'

> 4. If it is not a problem for you, can you please create a pure distribute
setup (instead of distributed-replicate) and then report the numbers?

TJ: I've been able to do this with 2 hosts, while I was at it I also tested
a pure replica and pure stripe setup for comparison
	Distribute = 313 MB/sec
	Replica = 166 MB/sec
	Stripe = 529 MB/sec

> 5. What is the inode size with which you formatted you XFS filesystem ?
> This last point might not be related to your throughput problem, but if
you are planning to use this setup for a large number of files, 
> you might be better off using an inode size of 512 instead of the default
256 bytes. To do that, your mkfs command should be:

> mkfs -t xfs -i size=512 /dev/<disk device>

TJ: This is destined for use with VM images, probably a maximum of 200 files
total. That said, I have tried with a bigger inode size and also with ext4
with very similar results each time

In a totally bizarre turn of events, turning on port bonding (each host has
2x 10gig storage ports) in ACTIVE/BACKUP mode has increased the speed a fair
bit
	dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=40000
	41943040000 bytes (42 GB) copied, 176.459 s, 238 MB/s

I have noticed that the inodes are getting very frequently locked/unlocked
by the afs from some brief debugging, not sure if that is related 

> Pavan


>> I am seeing slower-than-expected performance in Gluster 3.2.3 between 
>> 4 hosts with 10 gigabit eth between them all. Each host has 4x 300GB 
>> SAS 15K drives in RAID10, 6-core Xeon E5645 @ 2.40GHz and 24GB RAM 
>> running Ubuntu
>> 10.04 64-bit (I have also tested with Scientific Linux 6.1 and Debian 
>> Squeeze - same results on those as well). All of the hosts mount the 
>> volume using the FUSE module. The base filesystem on all of the nodes 
>> is XFS, however tests with ext4 have yielded similar results.
>>
>> Command used to create the volume:
>> 	gluster volume create cluster-volume replica 2 transport tcp 
>> node01:/mnt/local-store/ node02:/mnt/local-store/ 
>> node03:/mnt/local-store/ node04:/mnt/local-store/
>>
>> Command used to mount the Gluster volume on each node:
>> 	mount -t glusterfs localhost:/cluster-volume /mnt/cluster-volume
>>
>> Creating a 40GB file onto a node's local storage (ie no Gluster
>> involvement):
>> 	dd if=/dev/zero of=/mnt/local-store/test.file bs=1M count=40000
>> 	41943040000 bytes (42 GB) copied, 92.9264 s, 451 MB/s
>>
>> Getting the same file off the node's local storage:
>> 	dd if=/mnt/local-store/test.file of=/dev/null
>> 	41943040000 bytes (42 GB) copied, 81.858 s, 512 MB/s
>>
>> 40GB file onto the Gluster storage:
>> 	dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=40000
>> 	41943040000 bytes (42 GB) copied, 226.934 s, 185 MB/s
>>
>> Getting the same file off the Gluster storage
>> 	dd if=/mnt/cluster-volume/test.file of=/dev/null
>> 	41943040000 bytes (42 GB) copied, 661.561 s, 63.4 MB/s
>>
>> I have also tried using Gluster 3.1, with similar results.
>>
>> According to the Gluster docs, I should be seeing roughly the lesser 
>> of the drive speed and the network speed. The network is able to push 
>> 0.9GB/sec according to iperf so that definitely isn't a limiting 
>> factor here, and each array is able to do 400-500MB/sec as per above 
>> benchmarks. I've tried with/without jumbo frames as well, which doesn't
make any major difference.
>>
>> The glusterfs process is using 120% CPU according to top, and 
>> glusterfsd is sitting at about 90%.
>>
>> Any ideas / tips of where to start for speeding this config up?
>>
>> Thanks,
>>
>> Thomas
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

--
Message  protected by MailGuard: e-mail anti-virus, anti-spam and content
filtering.
http://www.mailguard.com.au

Click here to report this message as spam:
https://login.mailguard.com.au/report/1D7vGPSlkV/1bIPKtdYfJHjPGDWxiqyFd/2.76
6