[Gluster-users] Peer Probe

Joe Julian joe at julianfamily.org
Thu Feb 28 18:53:12 UTC 2013 

You have a host, fpsgluster, that's a peer. It's assigned a uuid.
You peer probe a different hostname, fpsglusterib, that  reports the 
same uuid - hence, already a peer.
You could remove it and re-add it with the new hostname unless you've 
created volumes using the non-ib hostname. If you created any volumes 
using fpsgluster, that hostname is now locked in. If you try to change 
the hostname for that peer by probing it from another server, it should 
fail because there's a volume using that hostname. It can't /add/ the 
new hostname because that uuid is already in use.

If you want to brute-force it there's two ways of making it happen. Both 
involve downtime.

 1. delete the volume(s)
    delete the server(s) from the peer group (gluster peer detach ...)
    re-add the server(s) (gluster peer probe ...ib) remove the extended
    attributes that mark a brick as having been part of a volume (
    http://joejulian.name/blog/glusterfs-path-or-a-prefix-of-it-is-already-part-of-a-volume/
    )
    re-create the volume(s)
 2. stop your volume(s) (gluster volume stop ...)
    kill glusterd on all servers
    search and replace all instances of the hostname under
    /var/lib/glusterd on each server (find -type f /var/lib/glusterd |
    xargs sed -i 's/fpsgluster/fpsglusterib/g') (This example presumes
    that sed expression is safe. You'll have to ensure that for yourself.)
    start glusterd on all servers
    start your volume(s) (gluster volume start ...)

Method 1 is the "supported" method.


On 02/25/2013 03:07 PM, Tony Saenz wrote:
> [root at fpsgluster ~]# gluster volume create testvault replica 2 transport rdma,tcp fpsglusterib:/mnt/testbrick1 fpsgluster2ib:/mnt/testbrick1 fpsglusterib:/mnt/testbrick2 fpsgluster2ib:/mnt/testbrick2 fpsglusterib:/mnt/testbrick3 fpsgluster2ib:/mnt/testbrick3
> Host fpsgluster2ib not a friend
>
> Not a friend error. I'm at a loss.
>
> On Feb 25, 2013, at 2:40 PM, harry mangalam <harry.mangalam at uci.edu>
>   wrote:
>
>> That looks OK (but your 2 MTUs are mismatched - should fix that).
>>
>>>           UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1  <- 1st
>>>           UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1 <- 2nd
>>                                                 ^^^^^
>> IBDEV=ibX
>> modprobe ib_umad
>> modprobe ib_ipoib
>> echo connected > /sys/class/net/${IBDEV}/mode
>> echo 65520 > /sys/class/net/${IBDEV}/mtu
>>
>> how did you set up the peering? By name?  by IP#?
>> (I assume pinging by hostname also works both ways?)
>>
>> If you can't get the peers to ack, then what do the logs say on failure to:
>>
>> peer probe <host>
>>   or create the volume
>> gluster volume create <volname>    host1ib:/gl_part   host2ib:/gl_part
>>
>> hjm
>>
>> On Monday, February 25, 2013 09:50:02 PM Tony Saenz wrote:
>>> Trying to first get this working with IPoIB
>>>
>>> [root at fpsgluster ~]# ibhosts
>>> Ca	: 0x00117500007937b2 ports 1 "fpsgluster2 qib0"
>>> Ca	: 0x0011750000792af2 ports 1 "fpsgluster qib0"
>>>
>>> I'm able to ping the other box from Infiniband to Infiniband card
>>>
>>> Ifconfig uses the ioctl access method to get the full address information,
>>> which limits hardware addresses to 8 bytes. Because Infiniband address has
>>> 20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
>>> obsolete! For replacement check ip.
>>> ib0       Link encap:InfiniBand  HWaddr
>>> 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
>>> addr:10.0.4.35  Bcast:10.0.4.255  Mask:255.255.255.0 inet6 addr:
>>> fe80::211:7500:79:2af2/64 Scope:Link
>>>           UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
>>>           RX packets:1567 errors:0 dropped:0 overruns:0 frame:0
>>>           TX packets:587 errors:0 dropped:24 overruns:0 carrier:0
>>>           collisions:0 txqueuelen:256
>>>           RX bytes:342622 (334.5 KiB)  TX bytes:96554 (94.2 KiB)
>>>
>>> [root at fpsgluster2 ~]# ifconfig ib0
>>> Ifconfig uses the ioctl access method to get the full address information,
>>> which limits hardware addresses to 8 bytes. Because Infiniband address has
>>> 20 bytes, only the first 8 bytes are displayed correctly. Ifconfig is
>>> obsolete! For replacement check ip.
>>> ib0       Link encap:InfiniBand  HWaddr
>>> 80:00:00:03:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:00 inet
>>> addr:10.0.4.34  Bcast:10.0.4.255  Mask:255.255.255.0 inet6 addr:
>>> fe80::211:7500:79:37b2/64 Scope:Link
>>>           UP BROADCAST RUNNING MULTICAST  MTU:65520  Metric:1
>>>           RX packets:599 errors:0 dropped:0 overruns:0 frame:0
>>>           TX packets:1558 errors:0 dropped:8 overruns:0 carrier:0
>>>           collisions:0 txqueuelen:256
>>>           RX bytes:95180 (92.9 KiB)  TX bytes:346728 (338.6 KiB)
>>>
>>> [root at fpsgluster ~]# ping -I ib0 10.0.4.34
>>> PING 10.0.4.34 (10.0.4.34) from 10.0.4.35 ib0: 56(84) bytes of data.
>>> 64 bytes from 10.0.4.34: icmp_seq=1 ttl=64 time=12.6 ms
>>> 64 bytes from 10.0.4.34: icmp_seq=2 ttl=64 time=0.184 ms
>>>
>>> /etc/hosts looks correct
>>>
>>> [root at fpsgluster2 ~]# cat /etc/hosts | grep ib
>>> 10.0.4.35      fpsglusterib
>>> 10.0.4.34      fpsgluster2ib
>>>
>>> [root at fpsgluster ~]# cat /etc/hosts| grep ib
>>> 10.0.4.35      fpsglusterib
>>> 10.0.4.34      fpsgluster2ib
>>>
>>> I haven't created the new volume yet as I can't get the peer probe to work
>>> off the Infiniband card. It's only seeing the NIC cards I currently have it
>>> hooked in to.
>>>
>>>
>>> On Feb 25, 2013, at 11:57 AM, harry mangalam <harry.mangalam at uci.edu>
>>>
>>> wrote:
>>>> It /might be/probably is/ DNS-related.
>>>>
>>>> Are you trying to do this with RDMA or IPoIB?
>>>>
>>>> If IPoIB, are ALL your /etc/hosts files in sync (IB names separate and
>>>> distinct from the ethernet interfaces) and responsive to the appropriate
>>>> interfaces?
>>>>
>>>> Do the IB interfaces show up as distinct (and connected) on an 'ifconfig
>>>> -a' and 'ibstat' dump?
>>>>
>>>> Do all the peers show up on an 'ibhosts' query?
>>>>
>>>> What is the output of:
>>>> gluster volume status <your_volume>
>>>> and
>>>> gluster volume status <your_volume> detail
>>>>
>>>>
>>>> hjm
>>>>
>>>> On Monday, February 25, 2013 07:46:00 PM Tony Saenz wrote:
>>>>> It shows this but it's still going through my NIC cards and not the
>>>>> Infiniband. (Checked the traffic on the cards themselves)
>>>>>
>>>>> [root at fpsgluster ~]# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: fpsgluster2
>>>>> Uuid: 9b7e7c2d-f05b-4cc8-b55a-571e383328d0
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> On Feb 25, 2013, at 10:51 AM, Torbjørn Thorsen <torbjorn at trollweb.no>
>> wrote:
>>>>>> Your error message seems to indicate that the peer is already in the
>>>>>> storage pool ?
>>>>>> What is the output of "gluster peer status" ?
>>>>>>
>>>>>> On Mon, Feb 25, 2013 at 7:28 PM, Tony Saenz <tony at filmsolutions.com>
>>>> wrote:
>>>>>>> Any help please? The regular NICs are fine which is what it currently
>>>>>>> sees but I'd like to move them over to the Infiniband cards.>>
>>>>>>> On Feb 22, 2013, at 1:50 PM, Anthony Saenz <tony at filmsolutions.com>
>>>> wrote:
>>>>>>>> Hey,
>>>>>>>>
>>>>>>>> I was wondering if I could get a bit of help.. I installed a new
>>>>>>>> Infiniband card into my servers but I'm unable to get it to come up as
>>>>>>>> a peer. Is there something I'm missing?
>>>>>>>>
>>>>>>>> [root at fpsgluster testvault]# gluster peer probe fpsgluster2ib
>>>>>>>> Probe on host fpsgluster2ib port 0 already in peer list
>>>>>>>>
>>>>>>>> [root at fpsgluster testvault]# yum list installed | grep gluster
>>>>>>>> glusterfs.x86_64                       3.3.1-1.el6
>>>>>>>> installed
>>>>>>>> glusterfs-devel.x86_64                 3.3.1-1.el6
>>>>>>>> installed
>>>>>>>> glusterfs-fuse.x86_64                  3.3.1-1.el6
>>>>>>>> installed
>>>>>>>> glusterfs-geo-replication.x86_64       3.3.1-1.el6
>>>>>>>> installed
>>>>>>>> glusterfs-rdma.x86_64                  3.3.1-1.el6
>>>>>>>> installed
>>>>>>>> glusterfs-server.x86_64                3.3.1-1.el6
>>>>>>>> installed
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>>> --
>>>>>> Vennlig hilsen
>>>>>> Torbjørn Thorsen
>>>>>> Utvikler / driftstekniker
>>>>>>
>>>>>> Trollweb Solutions AS
>>>>>> - Professional Magento Partner
>>>>>> www.trollweb.no
>>>>>>
>>>>>> Telefon dagtid: +47 51215300
>>>>>> Telefon kveld/helg: For kunder med Serviceavtale
>>>>>>
>>>>>> Besøksadresse: Luramyrveien 40, 4313 Sandnes
>>>>>> Postadresse: Maurholen 57, 4316 Sandnes
>>>>>>
>>>>>> Husk at alle våre standard-vilkår alltid er gjeldende
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>> ---
>>>> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
>>>> [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
>>>> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
>>>> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
>>>> ---
>>>> "Something must be done. [X] is something. Therefore, we must do it."
>>>> Bruce Schneier, on American response to just about anything.
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>> ---
>> Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
>> [m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
>> 415 South Circle View Dr, Irvine, CA, 92697 [shipping]
>> MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
>> ---
>> "Something must be done. [X] is something. Therefore, we must do it."
>> Bruce Schneier, on American response to just about anything.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130228/1ce3abd4/attachment.html>

More information about the Gluster-users mailing list