[Gluster-users] Peer Probe

harry mangalam harry.mangalam at uci.edu
Mon Feb 25 22:40:24 UTC 2013 

That looks OK (but your 2 MTUs are mismatched - should fix that).

>           UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1  <- 1st
>           UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1 <- 2nd
                                                ^^^^^
IBDEV=ibX
modprobe ib_umad 
modprobe ib_ipoib
echo connected > /sys/class/net/${IBDEV}/mode
echo 65520 > /sys/class/net/${IBDEV}/mtu

how did you set up the peering? By name?  by IP#?
(I assume pinging by hostname also works both ways?)

If you can't get the peers to ack, then what do the logs say on failure to:

peer probe <host>
  or create the volume 
gluster volume create <volname>    host1ib:/gl_part   host2ib:/gl_part

hjm

On Monday, February 25, 2013 09:50:02 PM Tony Saenz wrote:
> Trying to first get this working with IPoIB
> 
> [root at fpsgluster ~]# ibhosts
> Ca	: 0x00117500007937b2 ports 1 "fpsgluster2 qib0"
> Ca	: 0x0011750000792af2 ports 1 "fpsgluster qib0"
> 
> I'm able to ping the other box from Infiniband to Infiniband card
> 
> Ifconfig uses the ioctl access method to get the full address information,
> which limits hardware addresses to 8 bytes. Because Infiniband address has
> 20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
> obsolete! For replacement check ip.
> ib0       Link encap:InfiniBand  HWaddr
> 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
> addr:10.0.4.35  Bcast:10.0.4.255  Mask:255.255.255.0 inet6 addr:
> fe80::211:7500:79:2af2/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
>           RX packets:1567 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:587 errors:0 dropped:24 overruns:0 carrier:0
>           collisions:0 txqueuelen:256
>           RX bytes:342622 (334.5 KiB)  TX bytes:96554 (94.2 KiB)
> 
> [root at fpsgluster2 ~]# ifconfig ib0
> Ifconfig uses the ioctl access method to get the full address information,
> which limits hardware addresses to 8 bytes. Because Infiniband address has
> 20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
> obsolete! For replacement check ip.
> ib0       Link encap:InfiniBand  HWaddr
> 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
> addr:10.0.4.34  Bcast:10.0.4.255  Mask:255.255.255.0 inet6 addr:
> fe80::211:7500:79:37b2/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1
>           RX packets:599 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:1558 errors:0 dropped:8 overruns:0 carrier:0
>           collisions:0 txqueuelen:256
>           RX bytes:95180 (92.9 KiB)  TX bytes:346728 (338.6 KiB)
> 
> [root at fpsgluster ~]# ping -I ib0 10.0.4.34
> PING 10.0.4.34 (10.0.4.34) from 10.0.4.35 ib0: 56(84) bytes of data.
> 64 bytes from 10.0.4.34: icmp_seq=1 ttl=64 time=12.6 ms
> 64 bytes from 10.0.4.34: icmp_seq=2 ttl=64 time=0.184 ms
> 
> /etc/hosts looks correct
> 
> [root at fpsgluster2 ~]# cat /etc/hosts | grep ib
> 10.0.4.35      fpsglusterib
> 10.0.4.34      fpsgluster2ib
> 
> [root at fpsgluster ~]# cat /etc/hosts| grep ib
> 10.0.4.35      fpsglusterib
> 10.0.4.34      fpsgluster2ib
> 
> I haven't created the new volume yet as I can't get the peer probe to work
> off the Infiniband card. It's only seeing the NIC cards I currently have it
> hooked in to.
> 
> 
> On Feb 25, 2013, at 11:57 AM, harry mangalam <harry.mangalam at uci.edu>
> 
>  wrote:
> > It /might be/probably is/ DNS-related.
> > 
> > Are you trying to do this with RDMA or IPoIB?
> > 
> > If IPoIB, are ALL your /etc/hosts files in sync (IB names separate and
> > distinct from the ethernet interfaces) and responsive to the appropriate
> > interfaces?
> > 
> > Do the IB interfaces show up as distinct (and connected) on an 'ifconfig
> > -a' and 'ibstat' dump?
> > 
> > Do all the peers show up on an 'ibhosts' query?
> > 
> > What is the output of:
> > gluster volume status <your_volume>
> > and
> > gluster volume status <your_volume> detail
> > 
> > 
> > hjm
> > 
> > On Monday, February 25, 2013 07:46:00 PM Tony Saenz wrote:
> >> It shows this but it's still going through my NIC cards and not the
> >> Infiniband. (Checked the traffic on the cards themselves)
> >> 
> >> [root at fpsgluster ~]# gluster peer status
> >> Number of Peers: 1
> >> 
> >> Hostname: fpsgluster2
> >> Uuid: 9b7e7c2d-f05b-4cc8-b55a-571e383328d0
> >> State: Peer in Cluster (Connected)
> >> 
> >> On Feb 25, 2013, at 10:51 AM, Torbjørn Thorsen <torbjorn at trollweb.no> 
wrote:
> >>> Your error message seems to indicate that the peer is already in the
> >>> storage pool ?
> >>> What is the output of "gluster peer status" ?
> >>> 
> >>> On Mon, Feb 25, 2013 at 7:28 PM, Tony Saenz <tony at filmsolutions.com>
> > 
> > wrote:
> >>>> Any help please? The regular NICs are fine which is what it currently
> >>>> sees but I'd like to move them over to the Infiniband cards.>>
> >>>> On Feb 22, 2013, at 1:50 PM, Anthony Saenz <tony at filmsolutions.com>
> > 
> > wrote:
> >>>>> Hey,
> >>>>> 
> >>>>> I was wondering if I could get a bit of help.. I installed a new
> >>>>> Infiniband card into my servers but I'm unable to get it to come up as
> >>>>> a peer. Is there something I'm missing?
> >>>>> 
> >>>>> [root at fpsgluster testvault]# gluster peer probe fpsgluster2ib
> >>>>> Probe on host fpsgluster2ib port 0 already in peer list
> >>>>> 
> >>>>> [root at fpsgluster testvault]# yum list installed | grep gluster
> >>>>> glusterfs.x86_64                       3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-devel.x86_64                 3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-fuse.x86_64                  3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-geo-replication.x86_64       3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-rdma.x86_64                  3.3.1-1.el6
> >>>>> installed
> >>>>> glusterfs-server.x86_64                3.3.1-1.el6
> >>>>> installed
> >>>>> 
> >>>>> Thanks.
> >>>> 
> >>>> _______________________________________________
> >>>> Gluster-users mailing list
> >>>> Gluster-users at gluster.org
> >>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> >>> 
> >>> --
> >>> Vennlig hilsen
> >>> Torbjørn Thorsen
> >>> Utvikler / driftstekniker
> >>> 
> >>> Trollweb Solutions AS
> >>> - Professional Magento Partner
> >>> www.trollweb.no
> >>> 
> >>> Telefon dagtid: +47 51215300
> >>> Telefon kveld/helg: For kunder med Serviceavtale
> >>> 
> >>> Besøksadresse: Luramyrveien 40, 4313 Sandnes
> >>> Postadresse: Maurholen 57, 4316 Sandnes
> >>> 
> >>> Husk at alle våre standard-vilkår alltid er gjeldende
> >> 
> >> _______________________________________________
> >> Gluster-users mailing list
> >> Gluster-users at gluster.org
> >> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > 
> > ---
> > Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
> > [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
> > 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
> > MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
> > ---
> > "Something must be done. [X] is something. Therefore, we must do it."
> > Bruce Schneier, on American response to just about anything.
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

---
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
---
"Something must be done. [X] is something. Therefore, we must do it."
Bruce Schneier, on American response to just about anything.

More information about the Gluster-users mailing list