[Gluster-users] and again peer probe node1 hangs

[free.aaa]

Hi everybody!

I have "gluster peer probe gfs1" command hung with the result of "Probe 
Sent to Peer (connected)"
> gfs3#gluster peer status
> Number of Peers: 3
>
> Hostname: gfs6
> Uuid: 6bd6ee25-e257-4703-b500-330741b90471
> State: Peer in Cluster (Connected)
>
> Hostname: gfs4
> Uuid: bb1bed20-25bf-43b0-8faa-49f1b5b9ae59
> State: Peer in Cluster (Connected)
>
> Hostname: gfs1
> Uuid: bb67c1da-2698-4c35-b29d-f80f8eb814a6
> State: Probe Sent to Peer (Connected)

I double checked the dns resolution. Forward and reverse resolution 
works fine.
I tried to debug on gfs1 and gfs3 at the moment of probe command and it 
seems to me that gfs1 does not sync all the gluster configuration from 
gfs3. Especially peers folder contains only 1 peer with the following 
content:
> gfs1#cat /var/lib/glusterd/peers/192.168.9.53
> uuid=00000000-0000-0000-0000-000000000000
> state=8
> hostname1=192.168.9.53
so there are no uuids both in the content of the file and in filename 
itself. Besides there no info about other peers like gfs4, gfs6 in the 
folder.
Last lines of the debug log on gfs1 looks like this:
> [2015-04-23 10:12:28.334239] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.334413] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.334557] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.334697] D [common-utils.c:2946:gf_is_local_addr] 
> 0-management: gfs4 is not local
> [2015-04-23 10:12:28.334707] D 
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: 
> returning -1
> [2015-04-23 10:12:28.334714] D 
> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: 
> Returning -1
> [2015-04-23 10:12:28.336881] D 
> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management: 
> Unable to find friend: gfs6
> [2015-04-23 10:12:28.337490] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.337697] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.337841] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.337981] D [common-utils.c:2946:gf_is_local_addr] 
> 0-management: gfs6 is not local
> [2015-04-23 10:12:28.337991] D 
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: 
> returning -1
> [2015-04-23 10:12:28.337998] D 
> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: 
> Returning -1
> [2015-04-23 10:12:28.338858] D 
> [glusterd-utils.c:5523:glusterd_friend_find_by_hostname] 0-management: 
> Friend gfs3 found.. state: 8
> [2015-04-23 10:12:28.338873] D 
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: 
> returning 0
> [2015-04-23 10:12:28.340989] D 
> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management: 
> Unable to find friend: gfs6
> [2015-04-23 10:12:28.341545] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.341697] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.341840] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.56
> [2015-04-23 10:12:28.341980] D [common-utils.c:2946:gf_is_local_addr] 
> 0-management: gfs6 is not local
> [2015-04-23 10:12:28.341991] D 
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: 
> returning -1
> [2015-04-23 10:12:28.341997] D 
> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1
> [2015-04-23 10:12:28.342003] D 
> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: 
> Returning -1
> [2015-04-23 10:12:28.344065] D 
> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management: 
> Unable to find friend: gfs4
> [2015-04-23 10:12:28.344620] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.344772] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.344914] D [common-utils.c:2930:gf_is_local_addr] 
> 0-management: 192.168.9.54
> [2015-04-23 10:12:28.345054] D [common-utils.c:2946:gf_is_local_addr] 
> 0-management: gfs4 is not local
> [2015-04-23 10:12:28.345064] D 
> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management: 
> returning -1
> [2015-04-23 10:12:28.345071] D 
> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management: 
> Returning -1
> [2015-04-23 10:12:28.345543] D [run.c:190:runner_log] 0-: Starting the 
> nfs/glustershd services: /usr/sbin/glusterfs -s localhost --volfile-id 
> gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l 
> /var/log/glusterfs/quotad.log -S 
> /var/run/3e619fbfe69c96b1dbc7486a7d38a7be.socket --xlator-option 
> *replicate*.data-self-heal=off --xlator-option 
> *replicate*.metadata-self-heal=off --xlator-option 
> *replicate*.entry-self-heal=off
> ^C[2015-04-23 10:20:50.237403] W [glusterfsd.c:1095:cleanup_and_exit] 
> (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_end_reuse+0x26) 
> [0x7fa81cd3dfb6] 
> (-->/lib/x86_64-linux-gnu/libpthread.so.0(waitpid+0x5b) 
> [0x7fa81c499c8b] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x321e0) 
> [0x7fa81bd3a1e0]))) 0-: received signum (2), shutting down
> [2015-04-23 10:20:50.237436] D 
> [glusterfsd-mgmt.c:2025:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt: 
> portmapper signout arguments not given
So I think server hangs when trying to start volume with references to 
gfs4 and gfs6 which it does not know about, but why gfs1 peers 
configuration is empty?

Any help?

Thanks in advance!
Best regards,
Alex.