[Gluster-users] Gluster on tcp ip compared to infiniband

[John Marshall]

Amar S. Tumballi wrote:
> Hi Robert, John,
>  Thanks for sharing these numbers.
>
> One question for John, are you using performance translators ? we 
> expect to get higher numbers over IB.
Hi,

No. My first goal was purposely to compare without any extras/optimizations.
Of course, with them, I will expect better numbers as you mention :)

John
>
> Regards,
> Amar
>
>
> 2008/7/21 Robert Dildy <Robert at dildy.net <mailto:Robert at dildy.net>>:
>
>
>     John Marshall wrote:
>>     Jean Spirat wrote:
>>       
>>>     Hi,
>>>
>>>      I have seen the benchmarks on the gluster website and i wondered if 
>>>     there is a big difference between infiniband and tcp/ip performances. 
>>>     Anyone can share some stats on this i do not know the infiniband 
>>>     technology at all.
>>>       
>>>         
>>     Hi,
>>
>>     Some preliminary numbers I have for a 4x SDR IB network, single
>>     disk (RAID 5, 4+1, SAS, 300G 15K disks), single user:
>>
>>     transport        read    write   notes
>>     ---------        ----    -----   -----
>>     local            ~254     ~255   for a 100G file, not compensating for 
>>     caching
>>                                      take them for what their worth :)
>>     nfs              158     76      not gluster, of course!
>>     tcp (ipoib)      171     118     no optimizations
>>     sdp              211     108     no optimizations
>>     sdp              210     128     iot 4, 32MB
>>     verbs            218     150     no optimizations
>>     verbs            220     150     recv/send size 1048675, recv/send-count 16
>>
>>
>>     John
>>
>>     _______________________________________________
>>     Gluster-users mailing list
>>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>     http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>       
>     The tuning is just overall so the block sizes can make a huge
>     difference.  The numbers just give a good feeling for speeds.
>
>     We have 8 servers that are also clients of each other with DDR IB
>     each with 4x750 SATA II in RAID 5 with 16GB RAM.
>
>     rates are in MBs.  A = aggregat    S = per server
>
>     Transport            Read            Write            Read notes
>     ----------           ------            ------           ------
>     IBverbs               5993 (A)      5111 (A)     cache
>     IBverbs                 749 (S)        638  (S)     cache
>     IBverbs               3616 (A)      2598 (A)     exceeding cache,
>     but mixed
>     IBverbs                 452 (S)        325 (S)      exceeding
>     cache, bu tmixed
>     IBverbs                 958 (A)        747 (A)     no cache
>     IBverbs                 120 (S)           93 (S)     no cache
>     NFS (GigE)           874 (A)       739 (A)     cache
>     NFS (GigE)           109 (S)         92  (S)     cache
>
>     Hope it helps.
>     -robert
>              
>
>
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
>
>
>
> -- 
> Amar Tumballi
> Gluster/GlusterFS Hacker
> [bulde on #gluster/irc.gnu.org <http://irc.gnu.org>]
> http://www.zresearch.com - Commoditizing Super Storage!