[Gluster-users] infiniband speed

[Joe Landman]

On 01/10/2011 07:43 PM, Christopher Hawkins wrote:
> I am testing Infiniband for the first time. It seems that I should be
> able to get a lot more speed than I am with some pretty basic tests.
> Maybe someone running Infiniband can confirm that what I am seeing is
> way out of line, and/or help diagnose?

What is the underlying disk speed?

Try this:

	# (on bravo)
	dd if=/dev/zero of=/cluster/shadow/big.file bs=1M count=20k

This will write a 20GB file to the same partition.  We need to see how 
fast that write is (outside of cache)

Do the same test on the other machine.

Infiniband won't make slow disks fast.  So unless you are already 
getting a gigabyte or two per second (we do on our units), adding 
Infiniband won't get you any significant additional performance.

Put another way, you are bound by the slowest aspect of your storage system.

What are your underlying disks?  Are these individual drives?  RAID? 
What sort of RAID if it is a RAID?  How is it organized?

We tune our systems pretty hard, so we start with 2+GB/s for TB sized 
files before we ever touch the next stack up.  Each additional stack you 
traverse takes performance away (you lose it in stack inefficiency).  So 
you should start out with as fast a system as you possibly can, so you 
don't lose too much performance.

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615