[Gluster-users] (3.1.6-1) How should I add extra peers to existing file systems ?
krish
kp at gluster.com
Wed Aug 17 15:04:02 UTC 2011
Tomoaki,
You seem to have hit a bug in peer probe operation. This will be tracked in
bug 3432: see http://bugs.gluster.com/show_bug.cgi?id=3432
thanks,
kp
On 08/17/2011 12:28 PM, Tomoaki Sato wrote:
> kp,
>
> please find attached tar files.
>
> [root at vhead-010 ~]# ssh foo-1-private
> [root at foo-1-private ~]# gluster volume create foo
> foo-1-private:/mnt/brick
> Creation of volume foo has been successful. Please start the volume to
> access data.
> [root at foo-1-private ~]# gluster volume start foo
> Starting volume foo has been successful
> [root at foo-1-private ~]# gluster peer probe foo-3-private
> Probe successful
> [root at foo-1-private ~]# gluster peer status
> Number of Peers: 1
>
> Hostname: foo-3-private
> Uuid: 0c76c6ec-a7ea-405e-9d1d-8e100e155ec3
> State: Peer Rejected (Connected)
> [root at foo-1-private ~]# cd /etc/glusterd/vols/
> [root at foo-1-private vols]# tar cf /tmp/foo-1-private.tar foo
> [root at foo-1-private vols]# ssh foo-3-private
> [root at foo-3-private ~]# cd /etc/glusterd/vols
> [root at foo-3-private vols]# tar cf /tmp/foo-3-private.tar foo
> [root at foo-3-private vols]#
>
> Thanks,
> Tomo
>
> (2011/08/17 15:46), krish wrote:
>> Tomoaki,
>>
>> Can you send the info file under /etc/glusterd/vols/<volname>/ from
>> machines were the peer probe
>> command was issued and the 'peer' who is being rejected?
>>
>> thanks,
>> kp
>>
>> On 08/17/2011 12:10 PM, Tomoaki Sato wrote:
>>> Mohit
>>>
>>> (2011/08/17 13:53), Mohit Anchlia wrote:
>>>> Not sure. It could be because the new node doesn't have the volume
>>>> configs. Can you try gluster volume sync to sync the configs and see
>>>> if that helps?
>>>
>>> - at foo-1-private -
>>> gluster> volume sync foo-3-private
>>> please delete all the volumes before full sync
>>> gluster>
>>> --
>>>
>>> - at foo-3-private -
>>> gluster> volume sync foo foo-1-private
>>> sync from localhost not allowed
>>> gluster>
>>> --
>>>
>>> Question is "How should I add extra peers to existing file systems ?".
>>> extra peers = new nodes.
>>>
>>> Could you tell me right instructions to gluster probe new nodes
>>> after the volume starting ?
>>>
>>>>
>>>> Also, not sure why you are getting "Unable to find hostname:
>>>> foo-3-private"
>>>>
>>>
>>> "Unable to find hostname: foo-3-private" was printed out on both OK
>>> and NG cases.
>>> "Cksums of volume foo differ. local cksum = 1403573944, remote cksum
>>> = -1413994823" was printed out on NG case only.
>>>
>>> 'Peer in Cluster', OK case:
>>> [2011-08-17 15:08:47.462646] I
>>> [glusterd-handler.c:602:glusterd_handle_cli_probe] 0-glusterd:
>>> Received CLI probe req foo-3-private 24007
>>> [2011-08-17 15:08:47.466194] I
>>> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to
>>> find hostname: foo-3-private
>>> [2011-08-17 15:08:47.466224] I
>>> [glusterd-handler.c:3120:glusterd_probe_begin] 0-glusterd: Unable to
>>> find peerinfo for host: foo-3-private (24007)
>>> [2011-08-17 15:08:47.469365] W
>>> [rpc-transport.c:728:rpc_transport_load] 0-rpc-transport: missing
>>> 'option transport-type'. defaulting to "socket"
>>> [2011-08-17 15:08:47.473671] I
>>> [glusterd-handler.c:3102:glusterd_friend_add] 0-glusterd: connect
>>> returned 0
>>> [2011-08-17 15:08:47.474167] I
>>> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using
>>> Program glusterd clnt mgmt, Num (1238433), Version (1)
>>> [2011-08-17 15:08:47.474214] I
>>> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd:
>>> Friend foo-3-private found.. state: 0
>>> [2011-08-17 15:08:47.483485] I
>>> [glusterd-rpc-ops.c:364:glusterd3_1_probe_cbk] 0-glusterd: Received
>>> probe resp from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host:
>>> foo-3-private
>>> [2011-08-17 15:08:47.483516] I
>>> [glusterd-handler.c:379:glusterd_friend_find] 0-glusterd: Unable to
>>> find peer by uuid
>>> [2011-08-17 15:08:47.483562] I
>>> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd:
>>> Friend foo-3-private found.. state: 0
>>> [2011-08-17 15:08:47.483764] I
>>> [glusterd-rpc-ops.c:409:glusterd3_1_probe_cbk] 0-glusterd: Received
>>> resp to probe req
>>> [2011-08-17 15:08:47.484040] I
>>> [glusterd-rpc-ops.c:454:glusterd3_1_friend_add_cbk] 0-glusterd:
>>> Received ACC from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host:
>>> foo-3-private, port: 0
>>> [2011-08-17 15:08:47.484088] I
>>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
>>> Friend found... state: Probe Sent to Peer
>>> [2011-08-17 15:08:47.484153] I
>>> [glusterd-handler.c:3293:glusterd_xfer_cli_probe_resp] 0-glusterd:
>>> Responded to CLI, ret: 0
>>> [2011-08-17 15:08:47.484554] I
>>> [glusterd-handler.c:2882:glusterd_handle_probe_query] 0-glusterd:
>>> Received probe from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f
>>> [2011-08-17 15:08:47.484585] I
>>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
>>> Friend found... state: Accepted peer request
>>> [2011-08-17 15:08:47.484647] I
>>> [glusterd-handler.c:2917:glusterd_handle_probe_query] 0-glusterd:
>>> Responded to 192.168.1.129, op_ret: 0, op_errno: 0, ret: 0
>>> [2011-08-17 15:08:47.485499] I
>>> [glusterd-handler.c:2614:glusterd_handle_incoming_friend_req]
>>> 0-glusterd: Received probe from uuid:
>>> 464b3ea0-1b2b-4683-8209-72220dcb295f
>>> [2011-08-17 15:08:47.485536] I
>>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
>>> Friend found... state: Accepted peer request
>>> [2011-08-17 15:08:47.485590] I
>>> [glusterd-handler.c:3270:glusterd_xfer_friend_add_resp] 0-glusterd:
>>> Responded to 192.168.1.129 (0), ret: 0
>>> [2011-08-17 15:08:47.485713] I
>>> [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid:
>>> 464b3ea0-1b2b-4683-8209-72220dcb295f, host: foo-3-private
>>> [2011-08-17 15:08:47.486203] I
>>> [glusterd-rpc-ops.c:636:glusterd3_1_friend_update_cbk] 0-glusterd:
>>> Received ACC from uuid:
>>> [2011-08-17 15:08:47.486259] I
>>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
>>> Friend found... state: Peer in Cluster
>>> [2011-08-17 15:08:47.486284] I
>>> [glusterd-handler.c:2761:glusterd_handle_friend_update] 0-glusterd:
>>> Received friend update from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f
>>> [2011-08-17 15:08:47.486316] I
>>> [glusterd-handler.c:2806:glusterd_handle_friend_update] 0-: Received
>>> uuid: 4b5b0ecb-7d18-4ec4-90d9-0df2d392b63f, hostname:192.168.1.129
>>> [2011-08-17 15:08:47.486335] I
>>> [glusterd-handler.c:2809:glusterd_handle_friend_update] 0-: Received
>>> my uuid as Friend
>>>
>>>
>>> 'Peer Rejected', NG case:
>>> [2011-08-17 15:10:19.21262] I
>>> [glusterd-handler.c:602:glusterd_handle_cli_probe] 0-glusterd:
>>> Received CLI probe req foo-3-private 24007
>>> [2011-08-17 15:10:19.24605] I
>>> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to
>>> find hostname: foo-3-private
>>> [2011-08-17 15:10:19.24648] I
>>> [glusterd-handler.c:3120:glusterd_probe_begin] 0-glusterd: Unable to
>>> find peerinfo for host: foo-3-private (24007)
>>> [2011-08-17 15:10:19.27736] W
>>> [rpc-transport.c:728:rpc_transport_load] 0-rpc-transport: missing
>>> 'option transport-type'. defaulting to "socket"
>>> [2011-08-17 15:10:19.32034] I
>>> [glusterd-handler.c:3102:glusterd_friend_add] 0-glusterd: connect
>>> returned 0
>>> [2011-08-17 15:10:19.32389] I
>>> [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using
>>> Program glusterd clnt mgmt, Num (1238433), Version (1)
>>> [2011-08-17 15:10:19.32426] I
>>> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd:
>>> Friend foo-3-private found.. state: 0
>>> [2011-08-17 15:10:19.40671] I
>>> [glusterd-rpc-ops.c:364:glusterd3_1_probe_cbk] 0-glusterd: Received
>>> probe resp from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host:
>>> foo-3-private
>>> [2011-08-17 15:10:19.40720] I
>>> [glusterd-handler.c:379:glusterd_friend_find] 0-glusterd: Unable to
>>> find peer by uuid
>>> [2011-08-17 15:10:19.40748] I
>>> [glusterd-utils.c:2127:glusterd_friend_find_by_hostname] 0-glusterd:
>>> Friend foo-3-private found.. state: 0
>>> [2011-08-17 15:10:19.40983] I
>>> [glusterd-rpc-ops.c:409:glusterd3_1_probe_cbk] 0-glusterd: Received
>>> resp to probe req
>>> [2011-08-17 15:10:19.42854] I [rpc-clnt.c:696:rpc_clnt_handle_cbk]
>>> 0-rpc-clnt: recieved rpc message (XID: 0x2a, Ver: 2, Program:
>>> 52743234, ProgVers: 1, Proc: 1) from rpc-transport (management)
>>> [2011-08-17 15:10:19.50762] I
>>> [glusterd-rpc-ops.c:454:glusterd3_1_friend_add_cbk] 0-glusterd:
>>> Received ACC from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f, host:
>>> foo-3-private, port: 0
>>> [2011-08-17 15:10:19.50794] I
>>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
>>> Friend found... state: Probe Sent to Peer
>>> [2011-08-17 15:10:19.50851] I
>>> [glusterd-handler.c:3293:glusterd_xfer_cli_probe_resp] 0-glusterd:
>>> Responded to CLI, ret: 0
>>> [2011-08-17 15:10:19.51413] I
>>> [glusterd-handler.c:2882:glusterd_handle_probe_query] 0-glusterd:
>>> Received probe from uuid: 464b3ea0-1b2b-4683-8209-72220dcb295f
>>> [2011-08-17 15:10:19.51444] I
>>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
>>> Friend found... state: Accepted peer request
>>> [2011-08-17 15:10:19.51487] I
>>> [glusterd-handler.c:2917:glusterd_handle_probe_query] 0-glusterd:
>>> Responded to 192.168.1.129, op_ret: 0, op_errno: 0, ret: 0
>>> [2011-08-17 15:10:19.51853] I
>>> [glusterd-handler.c:2614:glusterd_handle_incoming_friend_req]
>>> 0-glusterd: Received probe from uuid:
>>> 464b3ea0-1b2b-4683-8209-72220dcb295f
>>> [2011-08-17 15:10:19.51885] I
>>> [glusterd-utils.c:2088:glusterd_friend_find_by_uuid] 0-glusterd:
>>> Friend found... state: Accepted peer request
>>> [2011-08-17 15:10:19.51930] E
>>> [glusterd-utils.c:1407:glusterd_compare_friend_volume] 0-: Cksums of
>>> volume foo differ. local cksum = 1403573944, remote cksum = -1413994823
>>> [2011-08-17 15:10:19.51975] I
>>> [glusterd-handler.c:3270:glusterd_xfer_friend_add_resp] 0-glusterd:
>>> Responded to 192.168.1.129 (0), ret: 0
>>>
>>>
>>>> On Tue, Aug 16, 2011 at 8:18 PM, Tomoaki Sato<tsato at valinux.co.jp>
>>>> wrote:
>>>>> Mohit,
>>>>>
>>>>> let me say again.
>>>>> 3.1.6-1 fail to 'peer probe' after 'start volume' in my environment.
>>>>> case-A) peer probe foo-3-private --> Peer in Cluster
>>>>>
>>>>> <delete all configuration files and reboot all foo-X-private nodes>
>>>>>
>>>>> [root at foo-1-private ~]# gluster peer probe foo-3-private
>>>>> Probe successful
>>>>> [root at foo-1-private ~]# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: foo-3-private
>>>>> Uuid: ef7d3c43-219a-4d13-a918-2639455cfbe7
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> case-B) create volume then peer probe foo-3-private --> Peer in
>>>>> Cluster
>>>>>
>>>>> <delete all configuration files and reboot all foo-X-private nodes>
>>>>>
>>>>> [root at foo-1-private ~]# gluster volume create foo
>>>>> foo-1-private:/mnt/brick
>>>>> Creation of volume foo has been successful. Please start the
>>>>> volume to
>>>>> access data.
>>>>> [root at foo-1-private ~]# gluster peer probe foo-3-private
>>>>> Probe successful
>>>>> [root at foo-1-private ~]# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: foo-3-private
>>>>> Uuid: fe44c954-4679-4389-a0e6-4c1fd4569a02
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> case-C) start volume then peer probe foo-3-private --> Peer Rejected
>>>>>
>>>>> <delete all configuration files and reboot all foo-X-private nodes>
>>>>>
>>>>> [root at foo-1-private ~]# gluster volume create foo
>>>>> foo-1-private:/mnt/brick
>>>>> Creation of volume foo has been successful. Please start the
>>>>> volume to
>>>>> access data.
>>>>> [root at foo-1-private ~]# gluster volume start foo
>>>>> Starting volume foo has been successful
>>>>> [root at foo-1-private ~]# gluster peer probe foo-3-private
>>>>> Probe successful
>>>>> [root at foo-1-private ~]# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: foo-3-private
>>>>> Uuid: bb6932e4-5bf0-4d34-872f-4a5fc1d0b6f8
>>>>> State: Peer Rejected (Connected)
>>>>>
>>>>>
>>>>>> Can you for now put it in /etc/hosts and test?
>>>>>
>>>>> All foo-X-private hosts have no entries in /etc/hosts.
>>>>> All the nodes obtain IP addresses from a DHCP server and register
>>>>> the IP
>>>>> address - host name paires to a DNS server.
>>>>>
>>>>>>
>>>>>> also, make sure you have same version of gluster running on all
>>>>>> the nodes.
>>>>>
>>>>> Since all three foo-X-private hosts are generated from a common VM
>>>>> template,
>>>>> same version of gluster run on all the nodes.
>>>>>
>>>>>>
>>>>>> What's the result of gluster peer status on node 3?
>>>>>
>>>>> [root at foo-1-private ~]# ssh foo-3-private gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: 192.168.1.129
>>>>> Uuid: 828bcc00-14d3-4505-8b35-d0ac6ca0730a
>>>>> State: Peer Rejected (Connected)
>>>>> [root at foo-1-private ~]#
>>>>>
>>>>>
>>>>> Best,
>>>>>
>>>>>
>>>>> (2011/08/17 0:53), Mohit Anchlia wrote:
>>>>>>
>>>>>> I see this in the logs:
>>>>>>
>>>>>> [2011-08-16 11:57:05.642903] I
>>>>>> [glusterd-handler.c:391:glusterd_friend_find] 0-glusterd: Unable to
>>>>>> find hostname: foo-3-private
>>>>>>
>>>>>> Can you for now put it in /etc/hosts and test?
>>>>>>
>>>>>> also, make sure you have same version of gluster running on all
>>>>>> the nodes.
>>>>>>
>>>>>> What's the result of gluster peer status on node 3?
>>>>>>
>>>>>> On Mon, Aug 15, 2011 at 8:18 PM, Tomoaki
>>>>>> Sato<tsato at valinux.co.jp> wrote:
>>>>>>>
>>>>>>> Mohit
>>>>>>>
>>>>>>> I've tried same test and reproduce the 'Peer Rejected' status.
>>>>>>> please find config files and log files in attached taz.
>>>>>>>
>>>>>>>
>>>>>>> [root at vhead-010 ~]# date
>>>>>>> Tue Aug 16 11:55:15 JST 2011
>>>>>>> [root at vhead-010 ~]# cat a.sh
>>>>>>> #!/bin/bash
>>>>>>> for i in foo-{1..3}-private
>>>>>>> do
>>>>>>> ssh ${i} service glusterd stop
>>>>>>> ssh ${i} 'find /etc/glusterd -type f|xargs rm -f'
>>>>>>> ssh ${i} rm -rf /etc/glusterd/vols/*
>>>>>>> ssh ${i} service iptables stop
>>>>>>> ssh ${i} cp /dev/null /var/log/glusterfs/nfs.log
>>>>>>> ssh ${i} cp /dev/null /var/log/glusterfs/bricks/mnt-brick.log
>>>>>>> ssh ${i} cp /dev/null /var/log/glusterfs/.cmd_log_history
>>>>>>> ssh ${i} cp /dev/null
>>>>>>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>>>> ssh ${i} service glusterd start
>>>>>>> ssh ${i} find /etc/glusterd
>>>>>>> ssh ${i} service glusterd status
>>>>>>> done
>>>>>>> [root at vhead-010 ~]# bash a.sh
>>>>>>> Stopping glusterd:[ OK ]
>>>>>>> Flushing firewall rules: [ OK ]
>>>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>>>> Unloading iptables modules: [ OK ]
>>>>>>> Starting glusterd:[ OK ]
>>>>>>> /etc/glusterd
>>>>>>> /etc/glusterd/glusterd.info
>>>>>>> /etc/glusterd/nfs
>>>>>>> /etc/glusterd/nfs/run
>>>>>>> /etc/glusterd/peers
>>>>>>> /etc/glusterd/vols
>>>>>>> glusterd (pid 15617) is running...
>>>>>>> Stopping glusterd:[ OK ]
>>>>>>> Flushing firewall rules: [ OK ]
>>>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>>>> Unloading iptables modules: [ OK ]
>>>>>>> Starting glusterd:[ OK ]
>>>>>>> /etc/glusterd
>>>>>>> /etc/glusterd/glusterd.info
>>>>>>> /etc/glusterd/nfs
>>>>>>> /etc/glusterd/nfs/run
>>>>>>> /etc/glusterd/peers
>>>>>>> /etc/glusterd/vols
>>>>>>> glusterd (pid 15147) is running...
>>>>>>> Stopping glusterd:[ OK ]
>>>>>>> Flushing firewall rules: [ OK ]
>>>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>>>> Unloading iptables modules: [ OK ]
>>>>>>> Starting glusterd:[ OK ]
>>>>>>> /etc/glusterd
>>>>>>> /etc/glusterd/glusterd.info
>>>>>>> /etc/glusterd/nfs
>>>>>>> /etc/glusterd/nfs/run
>>>>>>> /etc/glusterd/peers
>>>>>>> /etc/glusterd/vols
>>>>>>> glusterd (pid 15177) is running...
>>>>>>> [root at vhead-010 ~]# ssh foo-1-private
>>>>>>> Last login: Tue Aug 16 09:51:49 2011 from dlp.local.valinux.co.jp
>>>>>>> [root at localhost ~]# gluster peer probe foo-2-private
>>>>>>> Probe successful
>>>>>>> [root at localhost ~]# gluster peer status
>>>>>>> Number of Peers: 1
>>>>>>>
>>>>>>> Hostname: foo-2-private
>>>>>>> Uuid: 20b73d9a-ede0-454f-9fbb-b0eee9ce26a3
>>>>>>> State: Peer in Cluster (Connected)
>>>>>>> [root at localhost ~]# gluster volume create foo
>>>>>>> foo-1-private:/mnt/brick
>>>>>>> Creation of volume foo has been successful. Please start the
>>>>>>> volume to
>>>>>>> access data.
>>>>>>> [root at localhost ~]# gluster volume start foo
>>>>>>> Starting volume foo has been successful
>>>>>>> [root at localhost ~]# gluster volume add-brick foo
>>>>>>> foo-2-private:/mnt/brick
>>>>>>> Add Brick successful
>>>>>>> [root at localhost ~]# gluster peer probe foo-3-private
>>>>>>> Probe successful
>>>>>>> [root at localhost ~]# gluster peer status
>>>>>>> Number of Peers: 2
>>>>>>>
>>>>>>> Hostname: foo-2-private
>>>>>>> Uuid: 20b73d9a-ede0-454f-9fbb-b0eee9ce26a3
>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>
>>>>>>> Hostname: foo-3-private
>>>>>>> Uuid: 7587ae34-9209-484a-9576-3939e061720c
>>>>>>> State: Peer Rejected (Connected)
>>>>>>> [root at localhost ~]# exit
>>>>>>> logout
>>>>>>> Connection to foo-1-private closed.
>>>>>>> [root at vhead-010 ~]# find foo_log_and_conf
>>>>>>> foo_log_and_conf
>>>>>>> foo_log_and_conf/foo-2-private
>>>>>>> foo_log_and_conf/foo-2-private/glusterd
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/glusterd.info
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs/nfs-server.vol
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs/run
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/nfs/run/nfs.pid
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/peers
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/peers/461f6e21-90c4-4b6c-bda8-7b99bacb2722
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/info
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick
>>>>>>>
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick
>>>>>>>
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/cksum
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/run
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/run/foo-2-private-mnt-brick.pid
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo-fuse.vol
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol
>>>>>>>
>>>>>>> foo_log_and_conf/foo-2-private/glusterfs
>>>>>>> foo_log_and_conf/foo-2-private/glusterfs/nfs.log
>>>>>>> foo_log_and_conf/foo-2-private/glusterfs/bricks
>>>>>>> foo_log_and_conf/foo-2-private/glusterfs/bricks/mnt-brick.log
>>>>>>> foo_log_and_conf/foo-2-private/glusterfs/.cmd_log_history
>>>>>>> foo_log_and_conf/foo-2-private/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private
>>>>>>> foo_log_and_conf/foo-1-private/glusterd
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/glusterd.info
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs/nfs-server.vol
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs/run
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/nfs/run/nfs.pid
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/peers
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/peers/20b73d9a-ede0-454f-9fbb-b0eee9ce26a3
>>>>>>>
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/peers/7587ae34-9209-484a-9576-3939e061720c
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/info
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick
>>>>>>>
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick
>>>>>>>
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/cksum
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/run
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/run/foo-1-private-mnt-brick.pid
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo-fuse.vol
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol
>>>>>>>
>>>>>>> foo_log_and_conf/foo-1-private/glusterfs
>>>>>>> foo_log_and_conf/foo-1-private/glusterfs/nfs.log
>>>>>>> foo_log_and_conf/foo-1-private/glusterfs/bricks
>>>>>>> foo_log_and_conf/foo-1-private/glusterfs/bricks/mnt-brick.log
>>>>>>> foo_log_and_conf/foo-1-private/glusterfs/.cmd_log_history
>>>>>>> foo_log_and_conf/foo-1-private/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>>>>
>>>>>>> foo_log_and_conf/foo-3-private
>>>>>>> foo_log_and_conf/foo-3-private/glusterd
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/glusterd.info
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/nfs
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/nfs/run
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/peers
>>>>>>>
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/peers/461f6e21-90c4-4b6c-bda8-7b99bacb2722
>>>>>>>
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/info
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks
>>>>>>>
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks/foo-2-private:-mnt-brick
>>>>>>>
>>>>>>>
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/bricks/foo-1-private:-mnt-brick
>>>>>>>
>>>>>>>
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo.foo-2-private.mnt-brick.vol
>>>>>>>
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/cksum
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo-fuse.vol
>>>>>>>
>>>>>>> foo_log_and_conf/foo-3-private/glusterd/vols/foo/foo.foo-1-private.mnt-brick.vol
>>>>>>>
>>>>>>> foo_log_and_conf/foo-3-private/glusterfs
>>>>>>> foo_log_and_conf/foo-3-private/glusterfs/nfs.log
>>>>>>> foo_log_and_conf/foo-3-private/glusterfs/bricks
>>>>>>> foo_log_and_conf/foo-3-private/glusterfs/bricks/mnt-brick.log
>>>>>>> foo_log_and_conf/foo-3-private/glusterfs/.cmd_log_history
>>>>>>> foo_log_and_conf/foo-3-private/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>>>>
>>>>>>> [root at vhead-010 ~]# exit
>>>>>>>
>>>>>>> Best,
>>>>>>>
>>>>>>> (2011/08/16 9:35), Mohit Anchlia wrote:
>>>>>>>>
>>>>>>>> I should have also asked you to stop and delete volume before
>>>>>>>> getting
>>>>>>>> rid of gluster config files. Can you get rid of directories also
>>>>>>>> inside vols and try to restart? It's trying to look for volume
>>>>>>>> files
>>>>>>>> that we just removed.
>>>>>>>>
>>>>>>>> Also, just disable iptables for now explicitly.
>>>>>>>>
>>>>>>>> On Mon, Aug 15, 2011 at 5:22 PM, Tomoaki Sato<tsato at valinux.co.jp>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> 1) run peer detach for all the servers
>>>>>>>>>
>>>>>>>>> done.
>>>>>>>>>
>>>>>>>>>> 2) from server 1 ->3 and 3->1 make sure ports are open and
>>>>>>>>>> iptables
>>>>>>>>>> are turned off.
>>>>>>>>>
>>>>>>>>> done.
>>>>>>>>> by the way, the same test on 3.1.5-1 works fine with same
>>>>>>>>> environment.
>>>>>>>>>
>>>>>>>>>> 3) remove config files under /etc/glusterd
>>>>>>>>>
>>>>>>>>> please review following logs.
>>>>>>>>>
>>>>>>>>>> 4) run your tests again.
>>>>>>>>>
>>>>>>>>> I don't know why but glusterd service failed to start on all 3
>>>>>>>>> hosts.
>>>>>>>>>
>>>>>>>>> [root at vhead-010 ~]# date
>>>>>>>>> Tue Aug 16 09:12:53 JST 2011
>>>>>>>>> [root at vhead-010 ~]# cat a.sh
>>>>>>>>> #!/bin/bash
>>>>>>>>> for i in foo-{1..3}-private
>>>>>>>>> do
>>>>>>>>> ssh ${i} service glusterd stop
>>>>>>>>> ssh ${i} 'find /etc/glusterd -type f|xargs rm -f'
>>>>>>>>> ssh ${i} service iptables restart
>>>>>>>>> ssh ${i} iptables -vL
>>>>>>>>> ssh ${i} service glusterd start
>>>>>>>>> ssh ${i} find /etc/glusterd
>>>>>>>>> ssh ${i} service glusterd status
>>>>>>>>> done
>>>>>>>>> [root at vhead-010 ~]# bash a.sh
>>>>>>>>> Stopping glusterd:[ OK ]
>>>>>>>>> Flushing firewall rules: [ OK ]
>>>>>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>>>>>> Unloading iptables modules: [ OK ]
>>>>>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>>>> pkts bytes target prot opt in out source
>>>>>>>>> destination
>>>>>>>>>
>>>>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>>>>> pkts bytes target prot opt in out source
>>>>>>>>> destination
>>>>>>>>>
>>>>>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>>>> pkts bytes target prot opt in out source
>>>>>>>>> destination
>>>>>>>>> Starting glusterd:[ OK ]
>>>>>>>>> /etc/glusterd
>>>>>>>>> /etc/glusterd/glusterd.info
>>>>>>>>> /etc/glusterd/nfs
>>>>>>>>> /etc/glusterd/nfs/run
>>>>>>>>> /etc/glusterd/peers
>>>>>>>>> /etc/glusterd/vols
>>>>>>>>> /etc/glusterd/vols/foo
>>>>>>>>> /etc/glusterd/vols/foo/bricks
>>>>>>>>> /etc/glusterd/vols/foo/run
>>>>>>>>> glusterd is stopped
>>>>>>>>> Stopping glusterd:[ OK ]
>>>>>>>>> Flushing firewall rules: [ OK ]
>>>>>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>>>>>> Unloading iptables modules: [ OK ]
>>>>>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>>>> pkts bytes target prot opt in out source
>>>>>>>>> destination
>>>>>>>>>
>>>>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>>>>> pkts bytes target prot opt in out source
>>>>>>>>> destination
>>>>>>>>>
>>>>>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>>>> pkts bytes target prot opt in out source
>>>>>>>>> destination
>>>>>>>>> Starting glusterd:[ OK ]
>>>>>>>>> /etc/glusterd
>>>>>>>>> /etc/glusterd/glusterd.info
>>>>>>>>> /etc/glusterd/nfs
>>>>>>>>> /etc/glusterd/nfs/run
>>>>>>>>> /etc/glusterd/peers
>>>>>>>>> /etc/glusterd/vols
>>>>>>>>> /etc/glusterd/vols/foo
>>>>>>>>> /etc/glusterd/vols/foo/bricks
>>>>>>>>> /etc/glusterd/vols/foo/run
>>>>>>>>> glusterd is stopped
>>>>>>>>> Stopping glusterd:[ OK ]
>>>>>>>>> Flushing firewall rules: [ OK ]
>>>>>>>>> Setting chains to policy ACCEPT: filter [ OK ]
>>>>>>>>> Unloading iptables modules: [ OK ]
>>>>>>>>> Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>>>> pkts bytes target prot opt in out source
>>>>>>>>> destination
>>>>>>>>>
>>>>>>>>> Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
>>>>>>>>> pkts bytes target prot opt in out source
>>>>>>>>> destination
>>>>>>>>>
>>>>>>>>> Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
>>>>>>>>> pkts bytes target prot opt in out source
>>>>>>>>> destination
>>>>>>>>> Starting glusterd:[ OK ]
>>>>>>>>> /etc/glusterd
>>>>>>>>> /etc/glusterd/glusterd.info
>>>>>>>>> /etc/glusterd/nfs
>>>>>>>>> /etc/glusterd/nfs/run
>>>>>>>>> /etc/glusterd/peers
>>>>>>>>> /etc/glusterd/vols
>>>>>>>>> /etc/glusterd/vols/foo
>>>>>>>>> /etc/glusterd/vols/foo/bricks
>>>>>>>>> /etc/glusterd/vols/foo/run
>>>>>>>>> glusterd is stopped
>>>>>>>>> [root at vhead-010 ~]# date
>>>>>>>>> Tue Aug 16 09:13:20 JST 2011
>>>>>>>>> [root at vhead-010 ~]# ssh foo-1-private
>>>>>>>>> Last login: Tue Aug 16 09:06:57 2011 from dlp.local.valinux.co.jp
>>>>>>>>> [root at localhost ~]# tail -20
>>>>>>>>> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>>>>>> ...
>>>>>>>>> [2011-08-16 09:13:01.85858] I [glusterd.c:304:init]
>>>>>>>>> 0-management: Using
>>>>>>>>> /etc/glusterd as working directory
>>>>>>>>> [2011-08-16 09:13:01.87294] E
>>>>>>>>> [rpc-transport.c:799:rpc_transport_load]
>>>>>>>>> 0-rpc-transport:
>>>>>>>>> /opt/glusterfs/3.1.6/lib64/glusterfs/3.1.6/rpc-transport/rdma.so:
>>>>>>>>> cannot
>>>>>>>>> open shared object file: No such file or directory
>>>>>>>>> [2011-08-16 09:13:01.87340] E
>>>>>>>>> [rpc-transport.c:803:rpc_transport_load]
>>>>>>>>> 0-rpc-transport: volume 'rdma.management': transport-type
>>>>>>>>> 'rdma' is not
>>>>>>>>> valid or not found on this machine
>>>>>>>>> [2011-08-16 09:13:01.87402] E
>>>>>>>>> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd:
>>>>>>>>> Unable
>>>>>>>>> to
>>>>>>>>> retrieve store handle for /etc/glusterd/glusterd.info, error:
>>>>>>>>> No such
>>>>>>>>> file
>>>>>>>>> or directory
>>>>>>>>> [2011-08-16 09:13:01.87422] E
>>>>>>>>> [glusterd-store.c:761:glusterd_retrieve_uuid]
>>>>>>>>> 0-: Unable to get store handle!
>>>>>>>>> [2011-08-16 09:13:01.87514] I [glusterd.c:95:glusterd_uuid_init]
>>>>>>>>> 0-glusterd:
>>>>>>>>> generated UUID: c0cef9f9-a79e-4189-8955-d83927db9cee
>>>>>>>>> [2011-08-16 09:13:01.87681] E
>>>>>>>>> [glusterd-store.c:654:glusterd_store_handle_retrieve] 0-glusterd:
>>>>>>>>> Unable
>>>>>>>>> to
>>>>>>>>> retrieve store handle for /etc/glusterd/vols/foo/info, error:
>>>>>>>>> No such
>>>>>>>>> file
>>>>>>>>> or directory
>>>>>>>>> [2011-08-16 09:13:01.87704] E
>>>>>>>>> [glusterd-store.c:1328:glusterd_store_retrieve_volumes] 0-:
>>>>>>>>> Unable to
>>>>>>>>> restore volume: foo
>>>>>>>>
>>>>>>>>> [2011-08-16 09:13:01.87732] E [xlator.c:843:xlator_init]
>>>>>>>>> 0-management:
>>>>>>>>> Initialization of volume 'management' failed, review your
>>>>>>>>> volfile again
>>>>>>>>> [2011-08-16 09:13:01.87751] E [graph.c:331:glusterfs_graph_init]
>>>>>>>>> 0-management: initializing translator failed
>>>>>>>>> [2011-08-16 09:13:01.87818] I [glusterfsd.c:712:cleanup_and_exit]
>>>>>>>>> 0-glusterfsd: shutting down
>>>>>>>>> [root at localhost ~]# exit
>>>>>>>>>
>>>>>>>>> Best,
>>>>>>>>>
>>>>>>>>> (2011/08/16 8:52), Mohit Anchlia wrote:
>>>>>>>>>>
>>>>>>>>>> Logs are generally in /var/log/gluster
>>>>>>>>>>
>>>>>>>>>> Since you are playing with it. I would suggest this:
>>>>>>>>>>
>>>>>>>>>> 1) run peer detach for all the servers
>>>>>>>>>> 2) from server 1 ->3 and 3->1 make sure ports are open and
>>>>>>>>>> iptables
>>>>>>>>>> are turned off.
>>>>>>>>>> 3) remove config files under /etc/glusterd
>>>>>>>>>> 4) run your tests again.
>>>>>>>>>>
>>>>>>>>>> On Mon, Aug 15, 2011 at 4:28 PM, Tomoaki
>>>>>>>>>> Sato<tsato at valinux.co.jp>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Thanks, Mohit
>>>>>>>>>>>
>>>>>>>>>>> (2011/08/16 8:05), Mohit Anchlia wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> What's in your logs?
>>>>>>>>>>>
>>>>>>>>>>> I can obtain logs needed. could you tell me the instruction
>>>>>>>>>>> to take
>>>>>>>>>>> the
>>>>>>>>>>> logs?
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Did you have foo-3-private before in your gluster cluster
>>>>>>>>>>>> ever or
>>>>>>>>>>>> adding this host for the first time?
>>>>>>>>>>>
>>>>>>>>>>> It was first time.
>>>>>>>>>>> All foo-X-private has no entries in /etc/glusterd/peers/ and
>>>>>>>>>>> /etc/glusterd/vols/.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Try gluster peer detach and then remove any left over
>>>>>>>>>>>> configuration
>>>>>>>>>>>> in
>>>>>>>>>>>> /etc/glusterd config directory. After that try again and
>>>>>>>>>>>> see if that
>>>>>>>>>>>> works.
>>>>>>>>>>>
>>>>>>>>>>> [root at vhead-010 ~]# date
>>>>>>>>>>> Tue Aug 16 08:17:49 JST 2011
>>>>>>>>>>> [root at vhead-010 ~]# cat a.sh
>>>>>>>>>>> #!/bin/bash
>>>>>>>>>>> for i in foo-{1..3}-private
>>>>>>>>>>> do
>>>>>>>>>>> ssh ${i} service glusterd stop
>>>>>>>>>>> ssh ${i} rm -rf /etc/glusterd/peers/*
>>>>>>>>>>> ssh ${i} rm -rf /etc/glusterd/vols/*
>>>>>>>>>>> ssh ${i} service glusterd start
>>>>>>>>>>> ssh ${i} find /etc/glusterd
>>>>>>>>>>> done
>>>>>>>>>>> [root at vhead-010 ~]# bash a.sh
>>>>>>>>>>> Stopping glusterd:[ OK ]
>>>>>>>>>>> Starting glusterd:[ OK ]
>>>>>>>>>>> /etc/glusterd
>>>>>>>>>>> /etc/glusterd/glusterd.info
>>>>>>>>>>> /etc/glusterd/nfs
>>>>>>>>>>> /etc/glusterd/nfs/nfs-server.vol
>>>>>>>>>>> /etc/glusterd/nfs/run
>>>>>>>>>>> /etc/glusterd/peers
>>>>>>>>>>> /etc/glusterd/vols
>>>>>>>>>>> Stopping glusterd:[ OK ]
>>>>>>>>>>> Starting glusterd:[ OK ]
>>>>>>>>>>> /etc/glusterd
>>>>>>>>>>> /etc/glusterd/glusterd.info
>>>>>>>>>>> /etc/glusterd/nfs
>>>>>>>>>>> /etc/glusterd/nfs/nfs-server.vol
>>>>>>>>>>> /etc/glusterd/nfs/run
>>>>>>>>>>> /etc/glusterd/peers
>>>>>>>>>>> /etc/glusterd/vols
>>>>>>>>>>> Stopping glusterd:[ OK ]
>>>>>>>>>>> Starting glusterd:[ OK ]
>>>>>>>>>>> /etc/glusterd
>>>>>>>>>>> /etc/glusterd/glusterd.info
>>>>>>>>>>> /etc/glusterd/nfs
>>>>>>>>>>> /etc/glusterd/nfs/nfs-server.vol
>>>>>>>>>>> /etc/glusterd/nfs/run
>>>>>>>>>>> /etc/glusterd/peers
>>>>>>>>>>> /etc/glusterd/vols
>>>>>>>>>>> [root at vhead-010 ~]# ssh foo-1-private
>>>>>>>>>>> [root at localhost ~]# gluster peer probe foo-2-private
>>>>>>>>>>> Probe successful
>>>>>>>>>>> [root at localhost ~]# gluster peer status
>>>>>>>>>>> Number of Peers: 1
>>>>>>>>>>>
>>>>>>>>>>> Hostname: foo-2-private
>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>> [root at localhost ~]# gluster volume create foo
>>>>>>>>>>> foo-1-private:/mnt/brick
>>>>>>>>>>> Creation of volume foo has been successful. Please start the
>>>>>>>>>>> volume
>>>>>>>>>>> to
>>>>>>>>>>> access da
>>>>>>>>>>> ta.
>>>>>>>>>>> [root at localhost ~]# gluster volume start foo
>>>>>>>>>>> Starting volume foo has been successful
>>>>>>>>>>> [root at localhost ~]# gluster volume add-brick foo
>>>>>>>>>>> foo-2-private:/mnt/brick
>>>>>>>>>>> Add Brick successful
>>>>>>>>>>> [root at localhost ~]# gluster peer probe foo-3-private
>>>>>>>>>>> Probe successful
>>>>>>>>>>> [root at localhost ~]# gluster peer status
>>>>>>>>>>> Number of Peers: 2
>>>>>>>>>>>
>>>>>>>>>>> Hostname: foo-2-private
>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>>
>>>>>>>>>>> Hostname: foo-3-private
>>>>>>>>>>> Uuid: 7fb98dac-fef7-4b33-837c-6483a767ec3e
>>>>>>>>>>> State: Peer Rejected (Connected)
>>>>>>>>>>> [root at localhost ~]# cat /var/log/glusterfs/.cmd_log_history
>>>>>>>>>>> ...
>>>>>>>>>>> [2011-08-16 08:20:28.862619] peer probe : on host
>>>>>>>>>>> foo-2-private:24007
>>>>>>>>>>> [2011-08-16 08:20:28.912419] peer probe : on host
>>>>>>>>>>> foo-2-private:24007
>>>>>>>>>>> FAILED
>>>>>>>>>>> [2011-08-16 08:20:58.382350] Volume create : on volname: foo
>>>>>>>>>>> attempted
>>>>>>>>>>> [2011-08-16 08:20:58.382461] Volume create : on volname: foo
>>>>>>>>>>> type:DEFAULT
>>>>>>>>>>> count:
>>>>>>>>>>> 1 bricks: foo-1-private:/mnt/brick
>>>>>>>>>>> [2011-08-16 08:20:58.384674] Volume create : on volname: foo
>>>>>>>>>>> SUCCESS
>>>>>>>>>>> [2011-08-16 08:21:04.831772] volume start : on volname: foo
>>>>>>>>>>> SUCCESS
>>>>>>>>>>> [2011-08-16 08:21:22.682292] Volume add-brick : on volname: foo
>>>>>>>>>>> attempted
>>>>>>>>>>> [2011-08-16 08:21:22.682385] Volume add-brick : volname: foo
>>>>>>>>>>> type
>>>>>>>>>>> DEFAULT
>>>>>>>>>>> count:
>>>>>>>>>>> 1 bricks: foo-2-private:/mnt/brick
>>>>>>>>>>> [2011-08-16 08:21:22.682499] Volume add-brick : on volname: foo
>>>>>>>>>>> SUCCESS
>>>>>>>>>>> [2011-08-16 08:21:39.124574] peer probe : on host
>>>>>>>>>>> foo-3-private:24007
>>>>>>>>>>> [2011-08-16 08:21:39.135609] peer probe : on host
>>>>>>>>>>> foo-3-private:24007
>>>>>>>>>>> FAILED
>>>>>>>>>>>
>>>>>>>>>>> Tomo
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Aug 15, 2011 at 3:37 PM, Tomoaki
>>>>>>>>>>>> Sato<tsato at valinux.co.jp>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> following instructions work fine with 3.1.5-1 but with
>>>>>>>>>>>>> 3.1.6-1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. make a new file system without peers. [OK]
>>>>>>>>>>>>>
>>>>>>>>>>>>> foo-1-private# gluster volume create foo
>>>>>>>>>>>>> foo-1-private:/mnt/brick
>>>>>>>>>>>>> foo-1-private# gluster volume start foo
>>>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>>>> No peers present
>>>>>>>>>>>>> foo-1-private#
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. add a peer to the file system. [NG]
>>>>>>>>>>>>>
>>>>>>>>>>>>> foo-1-private# gluster peer probe foo-2-private
>>>>>>>>>>>>> Probe successful
>>>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>>>> Number of Peers: 1
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hostname: foo-2-private
>>>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>>>>>> State: Peer Rejected (Connected)
>>>>>>>>>>>>> foo-1-private# gluster volume add-brick foo
>>>>>>>>>>>>> foo-2-private:/mnt/brick
>>>>>>>>>>>>> Host foo-2-private not connected
>>>>>>>>>>>>> foo-1-private#
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> following instructions work fine even with 3.1.6-1.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. make a new file system with single peer. [OK]
>>>>>>>>>>>>>
>>>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>>>> No peer presents
>>>>>>>>>>>>> foo-1-private# gluster peer probe foo-2-private
>>>>>>>>>>>>> Probe successful
>>>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>>>> Number of Peers: 1
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hostname: foo-2-private
>>>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>>>> foo-1-private# gluster volume create foo
>>>>>>>>>>>>> foo-1-private:/mnt/brick
>>>>>>>>>>>>> Creation of volume foo has been successful. Please start
>>>>>>>>>>>>> the volume
>>>>>>>>>>>>> to
>>>>>>>>>>>>> access data.
>>>>>>>>>>>>> foo-1-private# gluster volume start foo
>>>>>>>>>>>>> Starting volume foo has been successful
>>>>>>>>>>>>> foo-1-private# gluster volume add-brick foo
>>>>>>>>>>>>> foo-2-private:/mnt/brick
>>>>>>>>>>>>> Add Brick successful
>>>>>>>>>>>>> foo-1-private#
>>>>>>>>>>>>>
>>>>>>>>>>>>> But ...
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2. add a peer to the file system. [NG]
>>>>>>>>>>>>>
>>>>>>>>>>>>> foo-1-private# gluster peer probe foo-3-private
>>>>>>>>>>>>> Probe successful
>>>>>>>>>>>>> foo-1-private# gluster peer status
>>>>>>>>>>>>> Number of Peers: 2
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hostname: foo-2-private
>>>>>>>>>>>>> Uuid: c2b314ac-6ed1-455a-84d4-ec22041ee2b2
>>>>>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hostname: foo-3-private
>>>>>>>>>>>>> Uuid: 7fb98dac-fef704b33-837c-6483a767ec3e
>>>>>>>>>>>>> State: Peer Rejected (Connected)
>>>>>>>>>>>>> foo-1-private# gluster volume add-brick foo
>>>>>>>>>>>>> foo-3-private:/mnt/brick
>>>>>>>>>>>>> Host foo-3-private not connected
>>>>>>>>>>>>> foo-1-private#
>>>>>>>>>>>>>
>>>>>>>>>>>>> How should I add extra peers to existing file systems ?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best,
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110817/7e62a6d9/attachment.html>
More information about the Gluster-users
mailing list