[Gluster-users] and again peer probe node1 hangs

Thu Apr 23 15:09:57 UTC 2015


On 04/23/2015 08:05 PM, free.aaa wrote:
> Hi everybody!
> 
> I have "gluster peer probe gfs1" command hung with the result of "Probe
> Sent to Peer (connected)"
>> gfs3#gluster peer status
>> Number of Peers: 3
>>
>> Hostname: gfs6
>> Uuid: 6bd6ee25-e257-4703-b500-330741b90471
>> State: Peer in Cluster (Connected)
>>
>> Hostname: gfs4
>> Uuid: bb1bed20-25bf-43b0-8faa-49f1b5b9ae59
>> State: Peer in Cluster (Connected)
>>
>> Hostname: gfs1
>> Uuid: bb67c1da-2698-4c35-b29d-f80f8eb814a6
>> State: Probe Sent to Peer (Connected)
> 
> I double checked the dns resolution. Forward and reverse resolution
> works fine.
> I tried to debug on gfs1 and gfs3 at the moment of probe command and it
> seems to me that gfs1 does not sync all the gluster configuration from
> gfs3. Especially peers folder contains only 1 peer with the following
> content:
>> gfs1#cat /var/lib/glusterd/peers/192.168.9.53
>> uuid=00000000-0000-0000-0000-000000000000
>> state=8
>> hostname1=192.168.9.53
> so there are no uuids both in the content of the file and in filename
> itself. Besides there no info about other peers like gfs4, gfs6 in the
> folder.
gfs1 would not have all the peers configuration until and unless it
becomes a node in the cluster. Here the peer handshaking between gfs1 &
gfs3 did not happen completely, some how gfs3 didn't receive a response
which it was expecting from gfs1. Can you check whether glusterd
instance of gfs1 some how went down at the point when peer probe was
executed?
> Last lines of the debug log on gfs1 looks like this:
>> [2015-04-23 10:12:28.334239] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.54
>> [2015-04-23 10:12:28.334413] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.54
>> [2015-04-23 10:12:28.334557] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.54
>> [2015-04-23 10:12:28.334697] D [common-utils.c:2946:gf_is_local_addr]
>> 0-management: gfs4 is not local
>> [2015-04-23 10:12:28.334707] D
>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>> returning -1
>> [2015-04-23 10:12:28.334714] D
>> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
>> Returning -1
>> [2015-04-23 10:12:28.336881] D
>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>> Unable to find friend: gfs6
>> [2015-04-23 10:12:28.337490] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.56
>> [2015-04-23 10:12:28.337697] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.56
>> [2015-04-23 10:12:28.337841] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.56
>> [2015-04-23 10:12:28.337981] D [common-utils.c:2946:gf_is_local_addr]
>> 0-management: gfs6 is not local
>> [2015-04-23 10:12:28.337991] D
>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>> returning -1
>> [2015-04-23 10:12:28.337998] D
>> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
>> Returning -1
>> [2015-04-23 10:12:28.338858] D
>> [glusterd-utils.c:5523:glusterd_friend_find_by_hostname] 0-management:
>> Friend gfs3 found.. state: 8
>> [2015-04-23 10:12:28.338873] D
>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>> returning 0
>> [2015-04-23 10:12:28.340989] D
>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>> Unable to find friend: gfs6
>> [2015-04-23 10:12:28.341545] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.56
>> [2015-04-23 10:12:28.341697] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.56
>> [2015-04-23 10:12:28.341840] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.56
>> [2015-04-23 10:12:28.341980] D [common-utils.c:2946:gf_is_local_addr]
>> 0-management: gfs6 is not local
>> [2015-04-23 10:12:28.341991] D
>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>> returning -1
>> [2015-04-23 10:12:28.341997] D
>> [glusterd-utils.c:685:glusterd_resolve_brick] 0-management: Returning -1
>> [2015-04-23 10:12:28.342003] D
>> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
>> Returning -1
>> [2015-04-23 10:12:28.344065] D
>> [glusterd-utils.c:5532:glusterd_friend_find_by_hostname] 0-management:
>> Unable to find friend: gfs4
>> [2015-04-23 10:12:28.344620] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.54
>> [2015-04-23 10:12:28.344772] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.54
>> [2015-04-23 10:12:28.344914] D [common-utils.c:2930:gf_is_local_addr]
>> 0-management: 192.168.9.54
>> [2015-04-23 10:12:28.345054] D [common-utils.c:2946:gf_is_local_addr]
>> 0-management: gfs4 is not local
>> [2015-04-23 10:12:28.345064] D
>> [glusterd-utils.c:5567:glusterd_hostname_to_uuid] 0-management:
>> returning -1
>> [2015-04-23 10:12:28.345071] D
>> [glusterd-utils.c:1035:glusterd_volume_brickinfo_get] 0-management:
>> Returning -1
>> [2015-04-23 10:12:28.345543] D [run.c:190:runner_log] 0-: Starting the
>> nfs/glustershd services: /usr/sbin/glusterfs -s localhost --volfile-id
>> gluster/quotad -p /var/lib/glusterd/quotad/run/quotad.pid -l
>> /var/log/glusterfs/quotad.log -S
>> /var/run/3e619fbfe69c96b1dbc7486a7d38a7be.socket --xlator-option
>> *replicate*.data-self-heal=off --xlator-option
>> *replicate*.metadata-self-heal=off --xlator-option
>> *replicate*.entry-self-heal=off
>> ^C[2015-04-23 10:20:50.237403] W [glusterfsd.c:1095:cleanup_and_exit]
>> (-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_end_reuse+0x26)
>> [0x7fa81cd3dfb6]
>> (-->/lib/x86_64-linux-gnu/libpthread.so.0(waitpid+0x5b)
>> [0x7fa81c499c8b] (-->/lib/x86_64-linux-gnu/libc.so.6(+0x321e0)
>> [0x7fa81bd3a1e0]))) 0-: received signum (2), shutting down
>> [2015-04-23 10:20:50.237436] D
>> [glusterfsd-mgmt.c:2025:glusterfs_mgmt_pmap_signout] 0-fsd-mgmt:
>> portmapper signout arguments not given
> So I think server hangs when trying to start volume with references to
> gfs4 and gfs6 which it does not know about, but why gfs1 peers
> configuration is empty?
> 
> Any help?
> 
> Thanks in advance!
> Best regards,
> Alex.
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 
> 

-- 
~Atin