[Gluster-users] 40 gig ethernet

[Stephan von Krawczynski]

On Fri, 14 Jun 2013 14:35:26 -0700
Bryan Whitehead <driver at megahappy.net> wrote:

> GigE is slower. Here is ping from same boxes but using the 1GigE cards:
> 
> [root at node0.cloud ~]# ping -c 10 10.100.0.11
> PING 10.100.0.11 (10.100.0.11) 56(84) bytes of data.
> 64 bytes from 10.100.0.11: icmp_seq=1 ttl=64 time=0.628 ms
> 64 bytes from 10.100.0.11: icmp_seq=2 ttl=64 time=0.283 ms
> 64 bytes from 10.100.0.11: icmp_seq=3 ttl=64 time=0.307 ms
> 64 bytes from 10.100.0.11: icmp_seq=4 ttl=64 time=0.275 ms
> 64 bytes from 10.100.0.11: icmp_seq=5 ttl=64 time=0.313 ms
> 64 bytes from 10.100.0.11: icmp_seq=6 ttl=64 time=0.278 ms
> 64 bytes from 10.100.0.11: icmp_seq=7 ttl=64 time=0.309 ms
> 64 bytes from 10.100.0.11: icmp_seq=8 ttl=64 time=0.197 ms
> 64 bytes from 10.100.0.11: icmp_seq=9 ttl=64 time=0.267 ms
> 64 bytes from 10.100.0.11: icmp_seq=10 ttl=64 time=0.187 ms
> 
> --- 10.100.0.11 ping statistics ---
> 10 packets transmitted, 10 received, 0% packet loss, time 9000ms
> rtt min/avg/max/mdev = 0.187/0.304/0.628/0.116 ms
> 
> Note: The Infiniband interfaces have a constant load of traffic from
> glusterfs. The Nic cards comparatively have very little traffic.

Uh, you should throw away your GigE switch. Example:

# ping 192.168.83.1
PING 192.168.83.1 (192.168.83.1) 56(84) bytes of data.
64 bytes from 192.168.83.1: icmp_seq=1 ttl=64 time=0.310 ms
64 bytes from 192.168.83.1: icmp_seq=2 ttl=64 time=0.199 ms
64 bytes from 192.168.83.1: icmp_seq=3 ttl=64 time=0.119 ms
64 bytes from 192.168.83.1: icmp_seq=4 ttl=64 time=0.115 ms
64 bytes from 192.168.83.1: icmp_seq=5 ttl=64 time=0.099 ms
64 bytes from 192.168.83.1: icmp_seq=6 ttl=64 time=0.082 ms
64 bytes from 192.168.83.1: icmp_seq=7 ttl=64 time=0.091 ms
64 bytes from 192.168.83.1: icmp_seq=8 ttl=64 time=0.096 ms
64 bytes from 192.168.83.1: icmp_seq=9 ttl=64 time=0.097 ms
64 bytes from 192.168.83.1: icmp_seq=10 ttl=64 time=0.095 ms
64 bytes from 192.168.83.1: icmp_seq=11 ttl=64 time=0.097 ms
64 bytes from 192.168.83.1: icmp_seq=12 ttl=64 time=0.102 ms
64 bytes from 192.168.83.1: icmp_seq=13 ttl=64 time=0.103 ms
64 bytes from 192.168.83.1: icmp_seq=14 ttl=64 time=0.108 ms
64 bytes from 192.168.83.1: icmp_seq=15 ttl=64 time=0.098 ms
64 bytes from 192.168.83.1: icmp_seq=16 ttl=64 time=0.093 ms
64 bytes from 192.168.83.1: icmp_seq=17 ttl=64 time=0.099 ms
64 bytes from 192.168.83.1: icmp_seq=18 ttl=64 time=0.102 ms
64 bytes from 192.168.83.1: icmp_seq=19 ttl=64 time=0.092 ms
64 bytes from 192.168.83.1: icmp_seq=20 ttl=64 time=0.111 ms
64 bytes from 192.168.83.1: icmp_seq=21 ttl=64 time=0.112 ms
64 bytes from 192.168.83.1: icmp_seq=22 ttl=64 time=0.099 ms
64 bytes from 192.168.83.1: icmp_seq=23 ttl=64 time=0.092 ms
64 bytes from 192.168.83.1: icmp_seq=24 ttl=64 time=0.102 ms
64 bytes from 192.168.83.1: icmp_seq=25 ttl=64 time=0.108 ms
^C
--- 192.168.83.1 ping statistics ---
25 packets transmitted, 25 received, 0% packet loss, time 23999ms
rtt min/avg/max/mdev = 0.082/0.112/0.310/0.047 ms

That is _loaded_.

-- 
Regards,
Stephan