[Gluster-users] Gluster and 10GigE - quick survey in the Community

[Robert Minvielle]

>Dear Community Users,
>
>In an effort to harden Gluster's strategy with 
>10GigE technology, I would like to request the following
>information from you -
>
>1) Are you already using 10GigE with either the Gluster
>   servers or clients ( or the platform ) ?

Yes, both servers and clients. 

>2) If not currently, are you considering using 10GigE with
>   Gluster servers or clients ( or platform ) in the future ?

n/a

>3) Which make, driver and on which OS are you using or considering
>   using this 10GigE technology with Gluster ?

Myricom 10G-PCIE-8A-R-E with Finisar 10gig XFP, Myricom driver myri10ge-linux-1.5.0.
Our OS is Gentoo, kernel 2.6.26, dell poweredge 2970 servers, AMD opteron, running
linux in 64 bit. 

>4) If you are already using this technology, would you like to 
>   share your experiences with us ?

Hrm, we have some gluster issues... constantly, but when we can get
it to work, it is quite fast. Currently, a standard NFS can beat it
down, but you guys know that from our past problems. On a test
cluster running 2.0.9 we can get good performance, however, we just
tried a single Dell Equallogic box with a single 1gig connection that
tore the gluster to shreds (2X the throughput). 

Still, we see potential in the gluster, we have not tried the Dell
with nearly as many users as the gluster. The 2X throughput was with
one user on the Dell, and at least 10 HD video editors on the Gluster.
As far as 10gig specifics go, it is much faster than the 1gig ethernet
we were using (actually 2x1gigs bonded). 

For some reason single large file performance is better than many small
files on the gluster. With the gluster in use by my production team, 
when I run tests I average 67-85MB/s from the gluster in read or writing
a single large file. On a dir of 1000s of small files that figure drops to
around 25MB/s. On the dell those figures were 112MB/s and 86MB/s, and the
112 was the max of the 1gig connection. Again, the dell had no users on it,
and the gluster is being hammered.  

Aha, I found my old notes: months ago I ran some tests during a downtime
with no users on the gluster... I see read at 101MB/s and write at 89MB/s,
and I always use the same data set, so they appear to be very close to one another.

Of course, right this second we have a major gluster outage, entire cluster
is down after upgrading from 2.0.2 to 2.0.9, can't  seem to find anyone
on IRC/groups, so if my tone seems to play it down, please ignore... those
numbers I posted are factual from my notes. Again, with no users, they 
seem to be equal. In regards to the 10gig, I know I have figures on it
being faster than the 1gig/2gig ethernet.  

I am also not 100% on the Myricom performance under linux. I have a friend
running Intel 10gig and his figures (not gluster) seem higher than ours. 


>Your feedback is extremely important to us. Please write to me soon.

>Regards,
>Tejas.

>tejas at gluster dot com