[Gluster-users] Gluster on tcp ip compared to infiniband

[Robert Dildy]

John Marshall wrote:
> Jean Spirat wrote:
>   
>> Hi,
>>
>>  I have seen the benchmarks on the gluster website and i wondered if 
>> there is a big difference between infiniband and tcp/ip performances. 
>> Anyone can share some stats on this i do not know the infiniband 
>> technology at all.
>>   
>>     
> Hi,
>
> Some preliminary numbers I have for a 4x SDR IB network, single
> disk (RAID 5, 4+1, SAS, 300G 15K disks), single user:
>
> transport        read    write   notes
> ---------        ----    -----   -----
> local            ~254     ~255   for a 100G file, not compensating for 
> caching
>                                  take them for what their worth :)
> nfs              158     76      not gluster, of course!
> tcp (ipoib)      171     118     no optimizations
> sdp              211     108     no optimizations
> sdp              210     128     iot 4, 32MB
> verbs            218     150     no optimizations
> verbs            220     150     recv/send size 1048675, recv/send-count 16
>
>
> John
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>   
The tuning is just overall so the block sizes can make a huge 
difference.  The numbers just give a good feeling for speeds.

We have 8 servers that are also clients of each other with DDR IB each 
with 4x750 SATA II in RAID 5 with 16GB RAM.

rates are in MBs.  A = aggregat    S = per server

Transport            Read            Write            Read notes
----------           ------            ------           ------
IBverbs               5993 (A)      5111 (A)     cache
IBverbs                 749 (S)        638  (S)     cache
IBverbs               3616 (A)      2598 (A)     exceeding cache, but mixed
IBverbs                 452 (S)        325 (S)      exceeding cache, bu 
tmixed
IBverbs                 958 (A)        747 (A)     no cache
IBverbs                 120 (S)           93 (S)     no cache
NFS (GigE)           874 (A)       739 (A)     cache
NFS (GigE)           109 (S)         92  (S)     cache

Hope it helps.
-robert
         

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20080721/a4b450c8/attachment.html>