[Gluster-users] Performance

[Joe Landman]

On 04/19/2011 10:43 PM, Mohit Anchlia wrote:
> It's HW RAID we are using. These are Dell C6100 server with RAID
> controller. We expect around 100MB/sec. When I run "dd" I get 20MB/sec
> and since I have 6 servers I at least expected 3 X 20 MB/sec since it
> replica of 2. We are in a subnet inside a LAB with only us doing the
> testing so network is not a issue for sure.
>
> File distribution are as follows:
>
> bytes %
> 130000 19.762%
> 70000 30.101%
> 100000 20.165%
> 230000 20.016%
> 1100000 0.447%
> 430000 5%
> 2000000 039%
> 630000 4.47%

Hmmm ...  90% of your io sizes seem to be between 70k and 230k bytes. 
This is ok.  What does your dd command look like?

Are you running stripe as well as mirror in gluster (not that it matters 
for performance).

A single dd will be rate limited to the single mirror pair it is writing 
to or reading from, unless you stripe.  Not RAID0 stripe, but stripe in 
gluster.

However ... before we get there ... on the raw volume (e.g. outside of 
gluster) what does

	dd if=/dev/sdX of=/dev/null bs=128k count=80k

e.g.

	[root at jr5-lab ~]# dd if=/dev/sdg of=/dev/null bs=128k count=80k
	81920+0 records in
	81920+0 records out
	10737418240 bytes (11 GB) copied, 14.7611 seconds, 727 MB/s


get you for each of the RAID0?  This will read ~ 11GB of data from the 
RAID0.  No writing.  You can write a file as well (would be very helpful)

	dd if=/dev/zero of=/path/to/gluster/file/system bs=128k \
		count=80k


e.g.

	[root at jr5-lab ~]# dd if=/dev/zero of=/data1/big.file bs=128k \
		count=80k
	81920+0 records in
	81920+0 records out
	10737418240 bytes (11 GB) copied, 29.5108 seconds, 364 MB/s


I am assuming these are PERC RAID controllers.  And assuming you 
configured the RAID0 in hardware rather than software.  What 
chunk/stripe size did you use?



-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615