[Gluster-users] and again peer probe node1 hangs

Fri Apr 24 09:21:47 UTC 2015

Atin,
As I'm trying to collect all the logs, I noticed that actually gfs3 does 
not continue handshaking.
On the gfs3 I can see that gfs1 actually responded to request:
> [2015-04-24 09:14:38.957968] D 
> [glusterd-utils.c:6111:glusterd_sm_tr_log_transition_add] 
> 0-management: Transitioning from 'Establishing Connection' to 'Probe 
> Sent to Peer' due to event 'GD_FRIEND_EVENT_INIT_FRIEND_REQ'
> [2015-04-24 09:14:38.957984] D 
> [glusterd-utils.c:6113:glusterd_sm_tr_log_transition_add] 
> 0-management: returning 0
> [2015-04-24 09:14:38.958000] D 
> [glusterd-store.c:2334:glusterd_store_create_peer_dir] 0-: Returning 
> with 0
> [2015-04-24 09:14:38.958010] D [store.c:434:gf_store_handle_destroy] 
> 0-: Returning 0
> [2015-04-24 09:14:38.958066] D [store.c:394:gf_store_handle_new] 0-: 
> Returning 0
> [2015-04-24 09:14:38.958713] D [store.c:348:gf_store_save_value] 0-: 
> returning: 0
> [2015-04-24 09:14:38.959153] D [store.c:348:gf_store_save_value] 0-: 
> returning: 0
> [2015-04-24 09:14:38.959548] D [store.c:348:gf_store_save_value] 0-: 
> returning: 0
> [2015-04-24 09:14:38.959572] D 
> [glusterd-store.c:2451:glusterd_store_peer_write] 0-: Returning with 0
> [2015-04-24 09:14:38.959641] D 
> [glusterd-store.c:2479:glusterd_store_perform_peer_store] 0-: Returning 0
> [2015-04-24 09:14:38.959658] D 
> [glusterd-store.c:2500:glusterd_store_peerinfo] 0-: Returning with 0
> [2015-04-24 09:14:38.959671] I 
> [glusterd-rpc-ops.c:306:__glusterd_probe_cbk] 0-glusterd: Received 
> resp to probe req

and after that gfs3 does not do anything until command timeouts.

24.04.2015 7:25, Atin Mukherjee пишет:
>
> On 04/23/2015 09:03 PM, free.aaa wrote:
>> Atin,
>>
>> Could you please clarify when a node becomes a node in the cluster?
>> After it receives the whole cluster config and before it starts to
>> launching any volumes?
> A node becomes part of the cluster only when the peer handshaking is
> successful. It goes through multiple phases of RPC exchanges and if the
> last RPC is successful then that peer will be marked as connected.
> Somehow, I am now confused about the problem as the logs are quite
> scattered. Could you raise a bug collecting sosreport on all the nodes.
> Please note that you should use sosreport -a to collect all log rotated
> files as well.
>
> ~Atin
>> Here is the beginning at gfs1 (deleted /var/lib/gluster and started again):
>>> I [glusterd-handler.c:2346:__glusterd_handle_probe_query] 0-glusterd:
>>> Received probe from uuid: 787c6a3c-f823-49d1-ab5a-537ed600654d
>>> [2015-04-23 14:11:40.319799] D
>>> [glusterd-utils.c:5446:glusterd_friend_find_by_uuid] 0-management:
>>> Friend with uuid: 787c6a3c-f823-49d1-ab5a-537ed600654d, not found
>>> [2015-04-23 14:11:40.319817] D
>>> [glusterd-handler.c:567:glusterd_friend_find] 0-management: Unable to
>>> find peer by uuid: 787c6a3c-f823-49d1-ab5a-537ed600654d
>>> [2015-04-23 14:11:40.321734] D
>>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>>> Unable to find friend: 192.168.9.53
>>> [2015-04-23 14:11:40.321753] D
>>> [glusterd-handler.c:579:glusterd_friend_find] 0-management: Unable to
>>> find hostname: 192.168.9.53
>>> [2015-04-23 14:11:40.321763] I
>>> [glusterd-handler.c:2374:__glusterd_handle_probe_query] 0-glusterd:
>>> Unable to find peerinfo for host: 192.168.9.53 (24007)
>>> [2015-04-23 14:11:40.321773] D
>>> [glusterd-utils.c:6062:glusterd_sm_tr_log_init] 0-: returning 0
>>> [2015-04-23 14:11:40.321780] D
>>> [glusterd-utils.c:6157:glusterd_peerinfo_new] 0-: returning 0
>>> [2015-04-23 14:11:40.321791] D
>>> [glusterd-store.c:2334:glusterd_store_create_peer_dir] 0-: Returning
>>> with 0
>>> [2015-04-23 14:11:40.321825] D [store.c:394:gf_store_handle_new] 0-:
>>> Returning 0
>>> [2015-04-23 14:11:40.322268] D [store.c:348:gf_store_save_value] 0-:
>>> returning: 0
>>> [2015-04-23 14:11:40.322693] D [store.c:348:gf_store_save_value] 0-:
>>> returning: 0
>>> [2015-04-23 14:11:40.323050] D [store.c:348:gf_store_save_value] 0-:
>>> returning: 0
>>> [2015-04-23 14:11:40.323063] D
>>> [glusterd-store.c:2451:glusterd_store_peer_write] 0-: Returning with 0
>>> [2015-04-23 14:11:40.323101] D
>>> [glusterd-store.c:2479:glusterd_store_perform_peer_store] 0-: Returning 0
>>> [2015-04-23 14:11:40.323110] D
>>> [glusterd-store.c:2500:glusterd_store_peerinfo] 0-: Returning with 0
>>> [2015-04-23 14:11:40.323130] D
>>> [glusterd-handler.c:2803:glusterd_transport_inet_options_build]
>>> 0-glusterd: Returning 0
>>> [2015-04-23 14:11:40.323153] I
>>> [rpc-clnt.c:972:rpc_clnt_connection_init] 0-management: setting
>>> frame-timeout to 600
>>> [2015-04-23 14:11:40.323167] D
>>> [rpc-transport.c:262:rpc_transport_load] 0-rpc-transport: attempt to
>>> load file
>>> /usr/lib/x86_64-linux-gnu/glusterfs/3.5.2/rpc-transport/socket.so
>>> [2015-04-23 14:11:40.323207] I [socket.c:3561:socket_init]
>>> 0-management: SSL support is NOT enabled
>>> [2015-04-23 14:11:40.323215] I [socket.c:3576:socket_init]
>>> 0-management: using system polling thread
>>> [2015-04-23 14:11:40.323543] D [common-utils.c:248:gf_resolve_ip6]
>>> 0-resolver: returning ip-192.168.9.53 (port-24007) for hostname:
>>> 192.168.9.53 and port: 24007
>>> [2015-04-23 14:11:40.323651] D
>>> [glusterd-handler.c:2742:glusterd_rpc_create] 0-management: returning 0
>>> [2015-04-23 14:11:40.323666] I
>>> [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect
>>> returned 0
>>> [2015-04-23 14:11:40.323696] I
>>> [glusterd-handler.c:2398:__glusterd_handle_probe_query] 0-glusterd:
>>> Responded to 192.168.9.53, op_ret: 0, op_errno: 0, ret: 0
>>> [2015-04-23 14:11:40.323816] D
>>> [glusterd-handler.c:3854:__glusterd_peer_rpc_notify] 0-management: got
>>> RPC_CLNT_CONNECT
>>> [2015-04-23 14:11:40.325448] I
>>> [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req]
>>> 0-glusterd: Received probe from uuid:
>>> 787c6a3c-f823-49d1-ab5a-537ed600654d
>>> [2015-04-23 14:11:40.325471] D
>>> [glusterd-utils.c:5446:glusterd_friend_find_by_uuid] 0-management:
>>> Friend with uuid: 787c6a3c-f823-49d1-ab5a-537ed600654d, not found
>>> [2015-04-23 14:11:40.325480] D
>>> [glusterd-handler.c:567:glusterd_friend_find] 0-management: Unable to
>>> find peer by uuid: 787c6a3c-f823-49d1-ab5a-537ed600654d
>>> [2015-04-23 14:11:40.325488] D
>>> [glusterd-utils.c:5483:glusterd_friend_find_by_hostname] 0-management:
>>> Friend 192.168.9.53 found.. state: 8
>>> [2015-04-23 14:11:40.325652] D
>>> [glusterd-sm.c:937:glusterd_friend_sm_inject_event] 0-glusterd:
>>> Enqueue event: 'GD_FRIEND_EVENT_RCVD_FRIEND_REQ'
>>> [2015-04-23 14:11:40.325664] D [glusterd-sm.c:1022:glusterd_friend_sm]
>>> 0-: Dequeued event of type: 'GD_FRIEND_EVENT_RCVD_FRIEND_REQ'
>>> [2015-04-23 14:11:40.325681] D
>>> [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning -1
>>> [2015-04-23 14:11:40.325688] D
>>> [glusterd-utils.c:2431:glusterd_compare_friend_volume] 0-management:
>>> Returning with ret: 0, status: 2
>>> [2015-04-23 14:11:40.325695] D
>>> [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning -1
>>> [2015-04-23 14:11:40.325701] D
>>> [glusterd-utils.c:2431:glusterd_compare_friend_volume] 0-management:
>>> Returning with ret: 0, status: 2
>>> [2015-04-23 14:11:40.325708] D
>>> [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning -1
>>> [2015-04-23 14:11:40.325714] D
>>> [glusterd-utils.c:2431:glusterd_compare_friend_volume] 0-management:
>>> Returning with ret: 0, status: 2
>>> [2015-04-23 14:11:40.325722] D
>>> [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning -1
>>> [2015-04-23 14:11:40.325727] D
>>> [glusterd-utils.c:2431:glusterd_compare_friend_volume] 0-management:
>>> Returning with ret: 0, status: 2
>>> [2015-04-23 14:11:40.325735] D
>>> [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning -1
>>> [2015-04-23 14:11:40.325741] D
>>> [glusterd-utils.c:2431:glusterd_compare_friend_volume] 0-management:
>>> Returning with ret: 0, status: 2
>>> [2015-04-23 14:11:40.325749] D
>>> [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning -1
>>> [2015-04-23 14:11:40.325755] D
>>> [glusterd-utils.c:2431:glusterd_compare_friend_volume] 0-management:
>>> Returning with ret: 0, status: 2
>>> [2015-04-23 14:11:40.325763] D
>>> [glusterd-utils.c:1163:glusterd_volinfo_find] 0-management: Returning -1
>>> [2015-04-23 14:11:40.325769] D
>>> [glusterd-utils.c:2431:glusterd_compare_friend_volume] 0-management:
>>> Returning with ret: 0, status: 2
>>> [2015-04-23 14:11:40.325814] D
>>> [glusterd-utils.c:2484:import_prdict_dict] 0-: Returning with 0
>>> [2015-04-23 14:11:40.325834] D
>>> [glusterd-utils.c:518:glusterd_volinfo_new] 0-management: Returning 0
>>> [2015-04-23 14:11:40.325855] D
>>> [glusterd-utils.c:3028:gd_import_friend_volume_rebal_dict]
>>> 0-management: Returning with 0
>>> [2015-04-23 14:11:40.325865] D
>>> [glusterd-utils.c:3028:gd_import_friend_volume_rebal_dict]
>>> 0-management: Returning with 0
>>> [2015-04-23 14:11:40.325879] D
>>> [glusterd-utils.c:2484:import_prdict_dict] 0-: Returning with 0
>>> [2015-04-23 14:11:40.325887] D
>>> [glusterd-utils.c:2484:import_prdict_dict] 0-: Returning with 0
>>> [2015-04-23 14:11:40.325892] D
>>> [glusterd-utils.c:2796:glusterd_import_friend_volume_opts] 0-:
>>> Returning with 0
>>> [2015-04-23 14:11:40.325908] D
>>> [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
>>> [2015-04-23 14:11:40.327010] D
>>> [glusterd-utils.c:5523:glusterd_friend_find_by_hostname] 0-management:
>>> Friend gfs3 found.. state: 8
>>> [2015-04-23 14:11:40.327036] D
>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>> returning 0
>>> [2015-04-23 14:11:40.327047] D
>>> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning 0
>>> [2015-04-23 14:11:40.327058] D
>>> [glusterd-utils.c:2858:glusterd_import_new_brick] 0-: Returning with 0
>>> [2015-04-23 14:11:40.327085] D
>>> [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
>>> [2015-04-23 14:11:40.329360] D
>>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>>> Unable to find friend: gfs6
>>> [2015-04-23 14:11:40.329891] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.56
>>> [2015-04-23 14:11:40.330116] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.56
>>> [2015-04-23 14:11:40.330260] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.56
>>> [2015-04-23 14:11:40.330497] D [common-utils.c:2946:gf_is_local_addr]
>>> 0-management: gfs6 is not local
>>> [2015-04-23 14:11:40.330509] D
>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>> returning -1
>>> [2015-04-23 14:11:40.330515] D
>>> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1
>>> [2015-04-23 14:11:40.330521] D
>>> [glusterd-utils.c:2858:glusterd_import_new_brick] 0-: Returning with 0
>>> [2015-04-23 14:11:40.330533] D
>>> [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
>>> [2015-04-23 14:11:40.332935] D
>>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>>> Unable to find friend: gfs4
>>> [2015-04-23 14:11:40.333563] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.54
>>> [2015-04-23 14:11:40.333752] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.54
>>> [2015-04-23 14:11:40.333896] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.54
>>> [2015-04-23 14:11:40.334037] D [common-utils.c:2946:gf_is_local_addr]
>>> 0-management: gfs4 is not local
>>> [2015-04-23 14:11:40.334047] D
>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>> returning -1
>>> [2015-04-23 14:11:40.334053] D
>>> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1
>>> [2015-04-23 14:11:40.334059] D
>>> [glusterd-utils.c:2858:glusterd_import_new_brick] 0-: Returning with 0
>>> [2015-04-23 14:11:40.334071] D
>>> [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
>>>
>> After that there are quite a lot of repeating log lines about "Unable to
>> find friend: xxx" :
>>> [2015-04-23 14:11:40.329360] D
>>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>>> Unable to find friend: gfs6
>>> [2015-04-23 14:11:40.329891] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.56
>>> [2015-04-23 14:11:40.330116] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.56
>>> [2015-04-23 14:11:40.330260] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.56
>>> [2015-04-23 14:11:40.330497] D [common-utils.c:2946:gf_is_local_addr]
>>> 0-management: gfs6 is not local
>>> [2015-04-23 14:11:40.330509] D
>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>> returning -1
>>> [2015-04-23 14:11:40.330515] D
>>> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1
>>> [2015-04-23 14:11:40.330521] D
>>> [glusterd-utils.c:2858:glusterd_import_new_brick] 0-: Returning with 0
>>> [2015-04-23 14:11:40.330533] D
>>> [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
>>> [2015-04-23 14:11:40.332935] D
>>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>>> Unable to find friend: gfs4
>>> [2015-04-23 14:11:40.333563] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.54
>>> [2015-04-23 14:11:40.333752] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.54
>>> [2015-04-23 14:11:40.333896] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.54
>>> [2015-04-23 14:11:40.334037] D [common-utils.c:2946:gf_is_local_addr]
>>> 0-management: gfs4 is not local
>>> [2015-04-23 14:11:40.334047] D
>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>> returning -1
>>> [2015-04-23 14:11:40.334053] D
>>> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1
>>> [2015-04-23 14:11:40.334059] D
>>> [glusterd-utils.c:2858:glusterd_import_new_brick] 0-: Returning with 0
>>> [2015-04-23 14:11:40.334071] D
>>> [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
>>> [2015-04-23 14:11:40.336009] D
>>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>>> Unable to find friend: gfs6
>>> [2015-04-23 14:11:40.336628] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.56
>>> [2015-04-23 14:11:40.336820] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.56
>>> [2015-04-23 14:11:40.336964] D [common-utils.c:2930:gf_is_local_addr]
>>> 0-management: 192.168.9.56
>>> [2015-04-23 14:11:40.337104] D [common-utils.c:2946:gf_is_local_addr]
>>> 0-management: gfs6 is not local
>>> [2015-04-23 14:11:40.337115] D
>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>> returning -1
>>> [2015-04-23 14:11:40.337121] D
>>> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1
>>> [2015-04-23 14:11:40.337127] D
>>> [glusterd-utils.c:2858:glusterd_import_new_brick] 0-: Returning with 0
>>> [2015-04-23 14:11:40.337139] D
>>> [glusterd-utils.c:669:glusterd_brickinfo_new] 0-management: Returning 0
>>> [2015-04-23 14:11:40.338120] D
>>> [glusterd-utils.c:5523:glusterd_friend_find_by_hostname] 0-management:
>>> Friend gfs3 found.. state: 8
>> In the middle of the log there are:
>>> [2015-04-23 14:11:40.998958] D [run.c:190:runner_log] 0-: Starting the
>>> nfs/glustershd services: /usr/sbin/glusterfs -s localhost --volfile-id
>>> gluster/nfs -p /var/lib/glusterd/nfs/run/nfs.pid -l
>>> /var/log/glusterfs/nfs.log -S
>>> /var/run/747c9e2447b63d0105d4a5a5a79fe22a.socket
>> and after
>>> [2015-04-23 14:11:41.078337] D [run.c:190:runner_log] 0-: Starting the
>>> nfs/glustershd services: /usr/sbin/glusterfs -s localhost --volfile-id
>>> gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l
>>> /var/log/glusterfs/quotad.log -S
>>> /var/run/ed7eeee033aa97faedeae7bcc0761203.socket --xlator-option
>>> *replicate*.data-self-heal=off --xlator-option
>>> *replicate*.metadata-self-heal=off --xlator-option
>>> *replicate*.entry-self-heal=off
>> it hangs because no new logs after this line.
>> After that if I kill all the glusterfs-server services and try to start
>> it again if will fail. So I need to rm -rf /var/lib/glusterd and start
>> again.
>>
>>
>>
>>
>> 23.04.2015 18:09, Atin Mukherjee пишет:
>>> On 04/23/2015 08:05 PM, free.aaa wrote:
>>>> Hi everybody!
>>>>
>>>> I have "gluster peer probe gfs1" command hung with the result of "Probe
>>>> Sent to Peer (connected)"
>>>>> gfs3#gluster peer status
>>>>> Number of Peers: 3
>>>>>
>>>>> Hostname: gfs6
>>>>> Uuid: 6bd6ee25-e257-4703-b500-330741b90471
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> Hostname: gfs4
>>>>> Uuid: bb1bed20-25bf-43b0-8faa-49f1b5b9ae59
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> Hostname: gfs1
>>>>> Uuid: bb67c1da-2698-4c35-b29d-f80f8eb814a6
>>>>> State: Probe Sent to Peer (Connected)
>>>> I double checked the dns resolution. Forward and reverse resolution
>>>> works fine.
>>>> I tried to debug on gfs1 and gfs3 at the moment of probe command and it
>>>> seems to me that gfs1 does not sync all the gluster configuration from
>>>> gfs3. Especially peers folder contains only 1 peer with the following
>>>> content:
>>>>> gfs1#cat /var/lib/glusterd/peers/192.168.9.53
>>>>> uuid=00000000-0000-0000-0000-000000000000
>>>>> state=8
>>>>> hostname1=192.168.9.53
>>>> so there are no uuids both in the content of the file and in filename
>>>> itself. Besides there no info about other peers like gfs4, gfs6 in the
>>>> folder.
>>> gfs1 would not have all the peers configuration until and unless it
>>> becomes a node in the cluster. Here the peer handshaking between gfs1 &
>>> gfs3 did not happen completely, some how gfs3 didn't receive a response
>>> which it was expecting from gfs1. Can you check whether glusterd
>>> instance of gfs1 some how went down at the point when peer probe was
>>> executed?
>>>> Last lines of the debug log on gfs1 looks like this:
>>>>> [2015-04-23 10:12:28.334239] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.54
>>>>> [2015-04-23 10:12:28.334413] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.54
>>>>> [2015-04-23 10:12:28.334557] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.54
>>>>> [2015-04-23 10:12:28.334697] D [common-utils.c:2946:gf_is_local_addr]
>>>>> 0-management: gfs4 is not local
>>>>> [2015-04-23 10:12:28.334707] D
>>>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>>>> returning -1
>>>>> [2015-04-23 10:12:28.334714] D
>>>>> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
>>>>> Returning -1
>>>>> [2015-04-23 10:12:28.336881] D
>>>>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>>>>> Unable to find friend: gfs6
>>>>> [2015-04-23 10:12:28.337490] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.56
>>>>> [2015-04-23 10:12:28.337697] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.56
>>>>> [2015-04-23 10:12:28.337841] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.56
>>>>> [2015-04-23 10:12:28.337981] D [common-utils.c:2946:gf_is_local_addr]
>>>>> 0-management: gfs6 is not local
>>>>> [2015-04-23 10:12:28.337991] D
>>>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>>>> returning -1
>>>>> [2015-04-23 10:12:28.337998] D
>>>>> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
>>>>> Returning -1
>>>>> [2015-04-23 10:12:28.338858] D
>>>>> [glusterd-utils.c:5523:glusterd_friend_find_by_hostname] 0-management:
>>>>> Friend gfs3 found.. state: 8
>>>>> [2015-04-23 10:12:28.338873] D
>>>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>>>> returning 0
>>>>> [2015-04-23 10:12:28.340989] D
>>>>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>>>>> Unable to find friend: gfs6
>>>>> [2015-04-23 10:12:28.341545] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.56
>>>>> [2015-04-23 10:12:28.341697] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.56
>>>>> [2015-04-23 10:12:28.341840] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.56
>>>>> [2015-04-23 10:12:28.341980] D [common-utils.c:2946:gf_is_local_addr]
>>>>> 0-management: gfs6 is not local
>>>>> [2015-04-23 10:12:28.341991] D
>>>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>>>> returning -1
>>>>> [2015-04-23 10:12:28.341997] D
>>>>> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management:
>>>>> Returning -1
>>>>> [2015-04-23 10:12:28.342003] D
>>>>> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
>>>>> Returning -1
>>>>> [2015-04-23 10:12:28.344065] D
>>>>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>>>>> Unable to find friend: gfs4
>>>>> [2015-04-23 10:12:28.344620] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.54
>>>>> [2015-04-23 10:12:28.344772] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.54
>>>>> [2015-04-23 10:12:28.344914] D [common-utils.c:2930:gf_is_local_addr]
>>>>> 0-management: 192.168.9.54
>>>>> [2015-04-23 10:12:28.345054] D [common-utils.c:2946:gf_is_local_addr]
>>>>> 0-management: gfs4 is not local
>>>>> [2015-04-23 10:12:28.345064] D
>>>>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>>>>> returning -1
>>>>> [2015-04-23 10:12:28.345071] D
>>>>> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
>>>>> Returning -1
>>>>> [2015-04-23 10:12:28.345543] D [run.c:190:runner_log] 0-: Starting the
>>>>> nfs/glustershd services: /usr/sbin/glusterfs -s localhost --volfile-id
>>>>> gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l
>>>>> /var/log/glusterfs/quotad.log -S
>>>>> /var/run/3e619fbfe69c96b1dbc7486a7d38a7be.socket --xlator-option
>>>>> *replicate*.data-self-heal=off --xlator-option
>>>>> *replicate*.metadata-self-heal=off --xlator-option
>>>>> *replicate*.entry-self-heal=off
>>>>> ^C[2015-04-23 10:20:50.237403] W [glusterfsd.c:1095:cleanup_and_exit]
>>>>> (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_end_reuse+0x26)
>>>>> [0x7fa81cd3dfb6]
>>>>> (-->/lib/x86_64-linux-gnu/libpthread.so.0(waitpid+0x5b)
>>>>> [0x7fa81c499c8b] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x321e0)
>>>>> [0x7fa81bd3a1e0]))) 0-: received signum (2), shutting down
>>>>> [2015-04-23 10:20:50.237436] D
>>>>> [glusterfsd-mgmt.c:2025:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt:
>>>>> portmapper signout arguments not given
>>>> So I think server hangs when trying to start volume with references to
>>>> gfs4 and gfs6 which it does not know about, but why gfs1 peers
>>>> configuration is empty?
>>>>
>>>> Any help?
>>>>
>>>> Thanks in advance!
>>>> Best regards,
>>>> Alex.
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>