[Gluster-users] (3.1.6-1) How should I add extra peers to existing file systems ?

Mohit Anchlia mohitanchlia at gmail.com
Wed Aug 17 04:53:14 UTC 2011 

Not sure. It could be because the new node doesn't have the volume
configs. Can you try gluster volume sync to sync the configs and see
if that helps?

Also, not sure why you are getting "Unable to find hostname: foo-3-private"

On Tue, Aug 16, 2011 at 8:18 PM, Tomoaki Sato <tsato at valinux.co.jp> wrote:
> Mohit,
>
> let me say again.
> 3.1.6-1 fail to 'peer probe' after 'start volume' in my environment.
>  case-A) peer probe foo-3-private --> Peer in Cluster
>
> <delete all configuration files and reboot all foo-X-private nodes>
>
> [root at foo-1-private ~]# gluster peer probe foo-3-private
> Probe successful
> [root at foo-1-private ~]# gluster peer status
> Number of Peers: 1
>
> Hostname: foo-3-private
> Uuid: ef7d3c43-219a-4d13-a918-2639455cfbe7
> State: Peer in Cluster (Connected)
>
> case-B) create volume then peer probe foo-3-private --> Peer in Cluster
>
> <delete all configuration files and reboot all foo-X-private nodes>
>
> [root at foo-1-private ~]# gluster volume create foo foo-1-private:/mnt/brick
> Creation of volume foo has been successful. Please start the volume to
> access data.
> [root at foo-1-private ~]# gluster peer probe foo-3-private
> Probe successful
> [root at foo-1-private ~]# gluster peer status
> Number of Peers: 1
>
> Hostname: foo-3-private
> Uuid: fe44c954-4679-4389-a0e6-4c1fd4569a02
> State: Peer in Cluster (Connected)
>
> case-C) start volume then peer probe foo-3-private --> Peer Rejected
>
> <delete all configuration files and reboot all foo-X-private nodes>
>
> [root at foo-1-private ~]# gluster volume create foo foo-1-private:/mnt/brick
> Creation of volume foo has been successful. Please start the volume to
> access data.
> [root at foo-1-private ~]# gluster volume start foo
> Starting volume foo has been successful
> [root at foo-1-private ~]# gluster peer probe foo-3-private
> Probe successful
> [root at foo-1-private ~]# gluster peer status
> Number of Peers: 1
>
> Hostname: foo-3-private
> Uuid: bb6932e4-5bf0-4d34-872f-4a5fc1d0b6f8
> State: Peer Rejected (Connected)
>
>
>> Can you for now put it in /etc/hosts and test?
>
> All foo-X-private hosts have no entries in /etc/hosts.
> All the nodes obtain IP addresses from a DHCP server and register the IP
> address - host name paires to a DNS server.
>
>>
>> also, make sure you have same version of gluster running on all the nodes.
>
> Since all three foo-X-private hosts are generated from a common VM template,
> same version of gluster run on all the nodes.
>
>>
>> What's the result of gluster peer status on node 3?
>
> [root at foo-1-private ~]# ssh foo-3-private gluster peer status
> Number of Peers: 1
>
> Hostname: 192.168.1.129
> Uuid: 828bcc00-14d3-4505-8b35-d0ac6ca0730a
> State: Peer Rejected (Connected)
> [root at foo-1-private ~]#
>
>
> Best,
>
>
> (2011/08/17 0:53), Mohit Anchlia wrote:
>>
>> I see this in the logs:
>>
>> [2011-08-16 11:57:05.642903] I
>> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to
>> find hostname: foo-3-private
>>
>> Can you for now put it in /etc/hosts and test?
>>
>> also, make sure you have same version of gluster running on all the nodes.
>>
>> What's the result of gluster peer status on node 3?
>>
>> On Mon, Aug 15, 2011 at 8:18 PM, Tomoaki Sato<tsato at valinux.co.jp>  wrote:
>>>
>>> Mohit
>>>
>>> I've tried same test and reproduce the 'Peer Rejected' status.
>>> please find config files and log files in attached taz.
>>>
>>>
>>> [root at vhead-010 ~]# date
>>> Tue Aug 16 11:55:15 JST 2011
>>> [root at vhead-010 ~]# cat a.sh
>>> #!/bin/bash
>>> for i in foo-{1..3}-private
>>> do
>>>        ssh ${i} service glusterd stop
>>>        ssh ${i} 'find /etc/glusterd -type f|xargs rm -f'
>>>        ssh ${i} rm -rf /etc/glusterd/vols/*
>>>        ssh ${i} service iptables stop
>>>        ssh ${i} cp /dev/null /var/log/glusterfs/nfs.log
>>>        ssh ${i} cp /dev/null /var/log/glusterfs/bricks/mnt-brick.log
>>>        ssh ${i} cp /dev/null /var/log/glusterfs/.cmd_log_history
>>>        ssh ${i} cp /dev/null
>>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
>>>        ssh ${i} service glusterd start
>>>        ssh ${i} find /etc/glusterd
>>>        ssh ${i} service glusterd status
>>> done
>>> [root at vhead-010 ~]# bash a.sh
>>> Stopping glusterd:[  OK  ]
>>> Flushing firewall rules: [  OK  ]
>>> Setting chains to policy ACCEPT: filter [  OK  ]
>>> Unloading iptables modules: [  OK  ]
>>> Starting glusterd:[  OK  ]
>>> /etc/glusterd
>>> /etc/glusterd/glusterd.info
>>> /etc/glusterd/nfs
>>> /etc/glusterd/nfs/run
>>> /etc/glusterd/peers
>>> /etc/glusterd/vols
>>> glusterd (pid 15617) is running...
>>> Stopping glusterd:[  OK  ]
>>> Flushing firewall rules: [  OK  ]
>>> Setting chains to policy ACCEPT: filter [  OK  ]
>>> Unloading iptables modules: [  OK  ]
>>> Starting glusterd:[  OK  ]
>>> /etc/glusterd
>>> /etc/glusterd/glusterd.info
>>> /etc/glusterd/nfs
>>> /etc/glusterd/nfs/run
>>> /etc/glusterd/peers
>>> /etc/glusterd/vols
>>> glusterd (pid 15147) is running...
>>> Stopping glusterd:[  OK  ]
>>> Flushing firewall rules: [  OK  ]
>>> Setting chains to policy ACCEPT: filter [  OK  ]
>>> Unloading iptables modules: [  OK  ]
>>> Starting glusterd:[  OK  ]
>>> /etc/glusterd
>>> /etc/glusterd/glusterd.info
>>> /etc/glusterd/nfs
>>> /etc/glusterd/nfs/run
>>> /etc/glusterd/peers
>>> /etc/glusterd/vols
>>> glusterd (pid 15177) is running...
>>> [root at vhead-010 ~]# ssh foo-1-private
>>> Last login: Tue Aug 16 09:51:49 2011 from dlp.local.valinux.co.jp
>>> [root at localhost ~]# gluster peer probe foo-2-private
>>> Probe successful
>>> [root at localhost ~]# gluster peer status
>>> Number of Peers: 1
>>>
>>> Hostname: foo-2-private
>>> Uuid: 20b73d9a-ede0-454f-9fbb-b0eee9ce26a3
>>> State: Peer in Cluster (Connected)
>>> [root at localhost ~]# gluster volume create foo foo-1-private:/mnt/brick
>>> Creation of volume foo has been successful. Please start the volume to
>>> access data.
>>> [root at localhost ~]# gluster volume start foo
>>> Starting volume foo has been successful
>>> [root at localhost ~]# gluster volume add-brick foo foo-2-private:/mnt/brick
>>> Add Brick successful
>>> [root at localhost ~]# gluster peer probe foo-3-private
>>> Probe successful
>>> [root at localhost ~]# gluster peer status
>>> Number of Peers: 2
>>>
>>> Hostname: foo-2-private
>>> Uuid: 20b73d9a-ede0-454f-9fbb-b0eee9ce26a3
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: foo-3-private
>>> Uuid: 7587ae34-9209-484a-9576-3939e061720c
>>> State: Peer Rejected (Connected)
>>> [root at localhost ~]# exit
>>> logout
>>> Connection to foo-1-private closed.
>>> [root at vhead-010 ~]# find foo_log_and_conf
>>> foo_log_and_conf
>>> foo_log_and_conf/foo-2-private
>>> foo_log_and_conf/foo-2-private/glusterd
>>> foo_log_and_conf/foo-2-private/glusterd/glusterd.info
>>> foo_log_and_conf/foo-2-private/glusterd/nfs
>>> foo_log_and_conf/foo-2-private/glusterd/nfs/nfs-server.vol
>>> foo_log_and_conf/foo-2-private/glusterd/nfs/run
>>> foo_log_and_conf/foo-2-private/glusterd/nfs/run/nfs.pid
>>> foo_log_and_conf/foo-2-private/glusterd/peers
>>>
>>> foo_log_and_conf/foo-2-private/glusterd/peers/461f6e21-90c4-4b6c-bda8-7b99bacb2722
>>> foo_log_and_conf/foo-2-private/glusterd/vols
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/info
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks
>>>
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick
>>>
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick
>>>
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/cksum
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/run
>>>
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/run/foo-2-private-mnt-brick.pid
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo-fuse.vol
>>>
>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol
>>> foo_log_and_conf/foo-2-private/glusterfs
>>> foo_log_and_conf/foo-2-private/glusterfs/nfs.log
>>> foo_log_and_conf/foo-2-private/glusterfs/bricks
>>> foo_log_and_conf/foo-2-private/glusterfs/bricks/mnt-brick.log
>>> foo_log_and_conf/foo-2-private/glusterfs/.cmd_log_history
>>> foo_log_and_conf/foo-2-private/glusterfs/etc-glusterfs-glusterd.vol.log
>>> foo_log_and_conf/foo-1-private
>>> foo_log_and_conf/foo-1-private/glusterd
>>> foo_log_and_conf/foo-1-private/glusterd/glusterd.info
>>> foo_log_and_conf/foo-1-private/glusterd/nfs
>>> foo_log_and_conf/foo-1-private/glusterd/nfs/nfs-server.vol
>>> foo_log_and_conf/foo-1-private/glusterd/nfs/run
>>> foo_log_and_conf/foo-1-private/glusterd/nfs/run/nfs.pid
>>> foo_log_and_conf/foo-1-private/glusterd/peers
>>>
>>> foo_log_and_conf/foo-1-private/glusterd/peers/20b73d9a-ede0-454f-9fbb-b0eee9ce26a3
>>>
>>> foo_log_and_conf/foo-1-private/glusterd/peers/7587ae34-9209-484a-9576-3939e061720c
>>> foo_log_and_conf/foo-1-private/glusterd/vols
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/info
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks
>>>
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick
>>>
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick
>>>
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/cksum
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/run
>>>
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/run/foo-1-private-mnt-brick.pid
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo-fuse.vol
>>>
>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol
>>> foo_log_and_conf/foo-1-private/glusterfs
>>> foo_log_and_conf/foo-1-private/glusterfs/nfs.log
>>> foo_log_and_conf/foo-1-private/glusterfs/bricks
>>> foo_log_and_conf/foo-1-private/glusterfs/bricks/mnt-brick.log
>>> foo_log_and_conf/foo-1-private/glusterfs/.cmd_log_history
>>> foo_log_and_conf/foo-1-private/glusterfs/etc-glusterfs-glusterd.vol.log
>>> foo_log_and_conf/foo-3-private
>>> foo_log_and_conf/foo-3-private/glusterd
>>> foo_log_and_conf/foo-3-private/glusterd/glusterd.info
>>> foo_log_and_conf/foo-3-private/glusterd/nfs
>>> foo_log_and_conf/foo-3-private/glusterd/nfs/run
>>> foo_log_and_conf/foo-3-private/glusterd/peers
>>>
>>> foo_log_and_conf/foo-3-private/glusterd/peers/461f6e21-90c4-4b6c-bda8-7b99bacb2722
>>> foo_log_and_conf/foo-3-private/glusterd/vols
>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo
>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/info
>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks
>>>
>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick
>>>
>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick
>>>
>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol
>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/cksum
>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo-fuse.vol
>>>
>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol
>>> foo_log_and_conf/foo-3-private/glusterfs
>>> foo_log_and_conf/foo-3-private/glusterfs/nfs.log
>>> foo_log_and_conf/foo-3-private/glusterfs/bricks
>>> foo_log_and_conf/foo-3-private/glusterfs/bricks/mnt-brick.log
>>> foo_log_and_conf/foo-3-private/glusterfs/.cmd_log_history
>>> foo_log_and_conf/foo-3-private/glusterfs/etc-glusterfs-glusterd.vol.log
>>> [root at vhead-010 ~]# exit
>>>
>>> Best,
>>>
>>> (2011/08/16 9:35), Mohit Anchlia wrote:
>>>>
>>>> I should have also asked you to stop and delete volume before getting
>>>> rid of gluster config files. Can you get rid of directories also
>>>> inside vols and try to restart? It's trying to look for volume files
>>>> that we just removed.
>>>>
>>>> Also, just disable iptables for now explicitly.
>>>>
>>>> On Mon, Aug 15, 2011 at 5:22 PM, Tomoaki Sato<tsato at valinux.co.jp>
>>>>  wrote:
>>>>>
>>>>>> 1) run peer detach for all the servers
>>>>>
>>>>> done.
>>>>>
>>>>>> 2) from server 1 ->3 and 3->1 make sure ports are open and iptables
>>>>>> are turned off.
>>>>>
>>>>> done.
>>>>> by the way, the same test on 3.1.5-1 works fine with same environment.
>>>>>
>>>>>> 3) remove config files under /etc/glusterd
>>>>>
>>>>> please review following logs.
>>>>>
>>>>>> 4) run your tests again.
>>>>>
>>>>> I don't know why but glusterd service failed to start on all 3 hosts.
>>>>>
>>>>> [root at vhead-010 ~]# date
>>>>> Tue Aug 16 09:12:53 JST 2011
>>>>> [root at vhead-010 ~]# cat a.sh
>>>>> #!/bin/bash
>>>>> for i in foo-{1..3}-private
>>>>> do
>>>>>        ssh ${i} service glusterd stop
>>>>>        ssh ${i} 'find /etc/glusterd -type f|xargs rm -f'
>>>>>        ssh ${i} service iptables restart
>>>>>        ssh ${i} iptables -vL
>>>>>        ssh ${i} service glusterd start
>>>>>        ssh ${i} find /etc/glusterd
>>>>>        ssh ${i} service glusterd status
>>>>> done
>>>>> [root at vhead-010 ~]# bash a.sh
>>>>> Stopping glusterd:[  OK  ]
>>>>> Flushing firewall rules: [  OK  ]
>>>>> Setting chains to policy ACCEPT: filter [  OK  ]
>>>>> Unloading iptables modules: [  OK  ]
>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>  pkts bytes target     prot opt in     out     source
>>>>> destination
>>>>>
>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>  pkts bytes target     prot opt in     out     source
>>>>> destination
>>>>>
>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>  pkts bytes target     prot opt in     out     source
>>>>> destination
>>>>> Starting glusterd:[  OK  ]
>>>>> /etc/glusterd
>>>>> /etc/glusterd/glusterd.info
>>>>> /etc/glusterd/nfs
>>>>> /etc/glusterd/nfs/run
>>>>> /etc/glusterd/peers
>>>>> /etc/glusterd/vols
>>>>> /etc/glusterd/vols/foo
>>>>> /etc/glusterd/vols/foo/bricks
>>>>> /etc/glusterd/vols/foo/run
>>>>> glusterd is stopped
>>>>> Stopping glusterd:[  OK  ]
>>>>> Flushing firewall rules: [  OK  ]
>>>>> Setting chains to policy ACCEPT: filter [  OK  ]
>>>>> Unloading iptables modules: [  OK  ]
>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>  pkts bytes target     prot opt in     out     source
>>>>> destination
>>>>>
>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>  pkts bytes target     prot opt in     out     source
>>>>> destination
>>>>>
>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>  pkts bytes target     prot opt in     out     source
>>>>> destination
>>>>> Starting glusterd:[  OK  ]
>>>>> /etc/glusterd
>>>>> /etc/glusterd/glusterd.info
>>>>> /etc/glusterd/nfs
>>>>> /etc/glusterd/nfs/run
>>>>> /etc/glusterd/peers
>>>>> /etc/glusterd/vols
>>>>> /etc/glusterd/vols/foo
>>>>> /etc/glusterd/vols/foo/bricks
>>>>> /etc/glusterd/vols/foo/run
>>>>> glusterd is stopped
>>>>> Stopping glusterd:[  OK  ]
>>>>> Flushing firewall rules: [  OK  ]
>>>>> Setting chains to policy ACCEPT: filter [  OK  ]
>>>>> Unloading iptables modules: [  OK  ]
>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>  pkts bytes target     prot opt in     out     source
>>>>> destination
>>>>>
>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>  pkts bytes target     prot opt in     out     source
>>>>> destination
>>>>>
>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>  pkts bytes target     prot opt in     out     source
>>>>> destination
>>>>> Starting glusterd:[  OK  ]
>>>>> /etc/glusterd
>>>>> /etc/glusterd/glusterd.info
>>>>> /etc/glusterd/nfs
>>>>> /etc/glusterd/nfs/run
>>>>> /etc/glusterd/peers
>>>>> /etc/glusterd/vols
>>>>> /etc/glusterd/vols/foo
>>>>> /etc/glusterd/vols/foo/bricks
>>>>> /etc/glusterd/vols/foo/run
>>>>> glusterd is stopped
>>>>> [root at vhead-010 ~]# date
>>>>> Tue Aug 16 09:13:20 JST 2011
>>>>> [root at vhead-010 ~]# ssh foo-1-private
>>>>> Last login: Tue Aug 16 09:06:57 2011 from dlp.local.valinux.co.jp
>>>>> [root at localhost ~]# tail -20
>>>>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>> ...
>>>>> [2011-08-16 09:13:01.85858] I [glusterd.c:304:init] 0-management: Using
>>>>> /etc/glusterd as working directory
>>>>> [2011-08-16 09:13:01.87294] E [rpc-transport.c:799:rpc_transport_load]
>>>>> 0-rpc-transport:
>>>>> /opt/glusterfs/3.1.6/lib64/glusterfs/3.1.6/rpc-transport/rdma.so:
>>>>> cannot
>>>>> open shared object file: No such file or directory
>>>>> [2011-08-16 09:13:01.87340] E [rpc-transport.c:803:rpc_transport_load]
>>>>> 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not
>>>>> valid or not found on this machine
>>>>> [2011-08-16 09:13:01.87402] E
>>>>> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd:
>>>>> Unable
>>>>> to
>>>>> retrieve store handle for /etc/glusterd/glusterd.info, error: No such
>>>>> file
>>>>> or directory
>>>>> [2011-08-16 09:13:01.87422] E
>>>>> [glusterd-store.c:761:glusterd_retrieve_uuid]
>>>>> 0-: Unable to get store handle!
>>>>> [2011-08-16 09:13:01.87514] I [glusterd.c:95:glusterd_uuid_init]
>>>>> 0-glusterd:
>>>>> generated UUID: c0cef9f9-a79e-4189-8955-d83927db9cee
>>>>> [2011-08-16 09:13:01.87681] E
>>>>> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd:
>>>>> Unable
>>>>> to
>>>>> retrieve store handle for /etc/glusterd/vols/foo/info, error: No such
>>>>> file
>>>>> or directory
>>>>> [2011-08-16 09:13:01.87704] E
>>>>> [glusterd-store.c:1328:glusterd_store_retrieve_volumes] 0-: Unable to
>>>>> restore volume: foo
>>>>
>>>>> [2011-08-16 09:13:01.87732] E [xlator.c:843:xlator_init] 0-management:
>>>>> Initialization of volume 'management' failed, review your volfile again
>>>>> [2011-08-16 09:13:01.87751] E [graph.c:331:glusterfs_graph_init]
>>>>> 0-management: initializing translator failed
>>>>> [2011-08-16 09:13:01.87818] I [glusterfsd.c:712:cleanup_and_exit]
>>>>> 0-glusterfsd: shutting down
>>>>> [root at localhost ~]# exit
>>>>>
>>>>> Best,
>>>>>
>>>>> (2011/08/16 8:52), Mohit Anchlia wrote:
>>>>>>
>>>>>> Logs are generally in /var/log/gluster
>>>>>>
>>>>>> Since you are playing with it. I would suggest this:
>>>>>>
>>>>>> 1) run peer detach for all the servers
>>>>>> 2) from server 1 ->3 and 3->1 make sure ports are open and iptables
>>>>>> are turned off.
>>>>>> 3) remove config files under /etc/glusterd
>>>>>> 4) run your tests again.
>>>>>>
>>>>>> On Mon, Aug 15, 2011 at 4:28 PM, Tomoaki Sato<tsato at valinux.co.jp>
>>>>>>  wrote:
>>>>>>>
>>>>>>> Thanks, Mohit
>>>>>>>
>>>>>>> (2011/08/16 8:05), Mohit Anchlia wrote:
>>>>>>>>
>>>>>>>> What's in your logs?
>>>>>>>
>>>>>>> I can obtain logs needed. could you tell me the instruction to take
>>>>>>> the
>>>>>>> logs?
>>>>>>>
>>>>>>>>
>>>>>>>> Did you have  foo-3-private before in your gluster cluster ever or
>>>>>>>> adding this host for the first time?
>>>>>>>
>>>>>>> It was first time.
>>>>>>> All foo-X-private has no entries in /etc/glusterd/peers/ and
>>>>>>> /etc/glusterd/vols/.
>>>>>>>
>>>>>>>>
>>>>>>>> Try gluster peer detach and then remove any left over configuration
>>>>>>>> in
>>>>>>>> /etc/glusterd config directory. After that try again and see if that
>>>>>>>> works.
>>>>>>>
>>>>>>> [root at vhead-010 ~]# date
>>>>>>> Tue Aug 16 08:17:49 JST 2011
>>>>>>> [root at vhead-010 ~]# cat a.sh
>>>>>>> #!/bin/bash
>>>>>>> for i in foo-{1..3}-private
>>>>>>> do
>>>>>>>        ssh ${i} service glusterd stop
>>>>>>>        ssh ${i} rm -rf /etc/glusterd/peers/*
>>>>>>>        ssh ${i} rm -rf /etc/glusterd/vols/*
>>>>>>>        ssh ${i} service glusterd start
>>>>>>>        ssh ${i} find /etc/glusterd
>>>>>>> done
>>>>>>> [root at vhead-010 ~]# bash a.sh
>>>>>>> Stopping glusterd:[  OK  ]
>>>>>>> Starting glusterd:[  OK  ]
>>>>>>> /etc/glusterd
>>>>>>> /etc/glusterd/glusterd.info
>>>>>>> /etc/glusterd/nfs
>>>>>>> /etc/glusterd/nfs/nfs-server.vol
>>>>>>> /etc/glusterd/nfs/run
>>>>>>> /etc/glusterd/peers
>>>>>>> /etc/glusterd/vols
>>>>>>> Stopping glusterd:[  OK  ]
>>>>>>> Starting glusterd:[  OK  ]
>>>>>>> /etc/glusterd
>>>>>>> /etc/glusterd/glusterd.info
>>>>>>> /etc/glusterd/nfs
>>>>>>> /etc/glusterd/nfs/nfs-server.vol
>>>>>>> /etc/glusterd/nfs/run
>>>>>>> /etc/glusterd/peers
>>>>>>> /etc/glusterd/vols
>>>>>>> Stopping glusterd:[  OK  ]
>>>>>>> Starting glusterd:[  OK  ]
>>>>>>> /etc/glusterd
>>>>>>> /etc/glusterd/glusterd.info
>>>>>>> /etc/glusterd/nfs
>>>>>>> /etc/glusterd/nfs/nfs-server.vol
>>>>>>> /etc/glusterd/nfs/run
>>>>>>> /etc/glusterd/peers
>>>>>>> /etc/glusterd/vols
>>>>>>> [root at vhead-010 ~]# ssh foo-1-private
>>>>>>> [root at localhost ~]# gluster peer probe foo-2-private
>>>>>>> Probe successful
>>>>>>> [root at localhost ~]# gluster peer status
>>>>>>> Number of Peers: 1
>>>>>>>
>>>>>>> Hostname: foo-2-private
>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>> State: Peer in Cluster (Connected)
>>>>>>> [root at localhost ~]# gluster volume create foo
>>>>>>> foo-1-private:/mnt/brick
>>>>>>> Creation of volume foo has been successful. Please start the volume
>>>>>>> to
>>>>>>> access da
>>>>>>> ta.
>>>>>>> [root at localhost ~]# gluster volume start foo
>>>>>>> Starting volume foo has been successful
>>>>>>> [root at localhost ~]# gluster volume add-brick foo
>>>>>>> foo-2-private:/mnt/brick
>>>>>>> Add Brick successful
>>>>>>> [root at localhost ~]# gluster peer probe foo-3-private
>>>>>>> Probe successful
>>>>>>> [root at localhost ~]# gluster peer status
>>>>>>> Number of Peers: 2
>>>>>>>
>>>>>>> Hostname: foo-2-private
>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>
>>>>>>> Hostname: foo-3-private
>>>>>>> Uuid: 7fb98dac-fef7-4b33-837c-6483a767ec3e
>>>>>>> State: Peer Rejected (Connected)
>>>>>>> [root at localhost ~]# cat /var/log/glusterfs/.cmd_log_history
>>>>>>> ...
>>>>>>> [2011-08-16 08:20:28.862619] peer probe :  on host
>>>>>>> foo-2-private:24007
>>>>>>> [2011-08-16 08:20:28.912419] peer probe : on host foo-2-private:24007
>>>>>>> FAILED
>>>>>>> [2011-08-16 08:20:58.382350] Volume create : on volname: foo
>>>>>>> attempted
>>>>>>> [2011-08-16 08:20:58.382461] Volume create : on volname: foo
>>>>>>> type:DEFAULT
>>>>>>> count:
>>>>>>> 1 bricks: foo-1-private:/mnt/brick
>>>>>>> [2011-08-16 08:20:58.384674] Volume create : on volname: foo SUCCESS
>>>>>>> [2011-08-16 08:21:04.831772] volume start : on volname: foo SUCCESS
>>>>>>> [2011-08-16 08:21:22.682292] Volume add-brick : on volname: foo
>>>>>>> attempted
>>>>>>> [2011-08-16 08:21:22.682385] Volume add-brick : volname: foo type
>>>>>>> DEFAULT
>>>>>>> count:
>>>>>>> 1 bricks: foo-2-private:/mnt/brick
>>>>>>> [2011-08-16 08:21:22.682499] Volume add-brick : on volname: foo
>>>>>>> SUCCESS
>>>>>>> [2011-08-16 08:21:39.124574] peer probe :  on host
>>>>>>> foo-3-private:24007
>>>>>>> [2011-08-16 08:21:39.135609] peer probe : on host foo-3-private:24007
>>>>>>> FAILED
>>>>>>>
>>>>>>> Tomo
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Aug 15, 2011 at 3:37 PM, Tomoaki Sato<tsato at valinux.co.jp>
>>>>>>>>  wrote:
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> following instructions work fine with 3.1.5-1 but with 3.1.6-1.
>>>>>>>>>
>>>>>>>>> 1. make a new file system without peers. [OK]
>>>>>>>>>
>>>>>>>>> foo-1-private# gluster volume create foo foo-1-private:/mnt/brick
>>>>>>>>> foo-1-private# gluster volume start foo
>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>> No peers present
>>>>>>>>> foo-1-private#
>>>>>>>>>
>>>>>>>>> 2. add a peer to the file system. [NG]
>>>>>>>>>
>>>>>>>>> foo-1-private# gluster peer probe foo-2-private
>>>>>>>>> Probe successful
>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>> Number of Peers: 1
>>>>>>>>>
>>>>>>>>> Hostname: foo-2-private
>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>> State: Peer Rejected (Connected)
>>>>>>>>> foo-1-private# gluster volume add-brick foo
>>>>>>>>> foo-2-private:/mnt/brick
>>>>>>>>> Host foo-2-private not connected
>>>>>>>>> foo-1-private#
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> following instructions work fine even with 3.1.6-1.
>>>>>>>>>
>>>>>>>>> 1. make a new file system with single peer. [OK]
>>>>>>>>>
>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>> No peer presents
>>>>>>>>> foo-1-private# gluster peer probe foo-2-private
>>>>>>>>> Probe successful
>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>> Number of Peers: 1
>>>>>>>>>
>>>>>>>>> Hostname: foo-2-private
>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>> foo-1-private# gluster volume create foo foo-1-private:/mnt/brick
>>>>>>>>> Creation of volume foo has been successful. Please start the volume
>>>>>>>>> to
>>>>>>>>> access data.
>>>>>>>>> foo-1-private# gluster volume start foo
>>>>>>>>> Starting volume foo has been successful
>>>>>>>>> foo-1-private# gluster volume add-brick foo
>>>>>>>>> foo-2-private:/mnt/brick
>>>>>>>>> Add Brick successful
>>>>>>>>> foo-1-private#
>>>>>>>>>
>>>>>>>>> But ...
>>>>>>>>>
>>>>>>>>> 2. add a peer to the file system. [NG]
>>>>>>>>>
>>>>>>>>> foo-1-private# gluster peer probe foo-3-private
>>>>>>>>> Probe successful
>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>> Number of Peers: 2
>>>>>>>>>
>>>>>>>>> Hostname: foo-2-private
>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>
>>>>>>>>> Hostname: foo-3-private
>>>>>>>>> Uuid: 7fb98dac-fef704b33-837c-6483a767ec3e
>>>>>>>>> State: Peer Rejected (Connected)
>>>>>>>>> foo-1-private# gluster volume add-brick foo
>>>>>>>>> foo-3-private:/mnt/brick
>>>>>>>>> Host foo-3-private not connected
>>>>>>>>> foo-1-private#
>>>>>>>>>
>>>>>>>>> How should I add extra peers to existing file systems ?
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>> _______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>>
>
>

More information about the Gluster-users mailing list