[Gluster-users] What does this error mean?

Mon Mar 21 19:20:40 UTC 2011

Is node 5 still showing "Disconnected" for node 6?

On Mon, Mar 21, 2011 at 12:08 PM, Burnash, James <jburnash at knight.com> wrote:
> After 'service glusterd restart' on node 6:
>
> root at jc1letgfs5:/etc/glusterd/vols# gluster peer status
> Number of Peers: 3
>
> Hostname: jc1letgfs6
> Uuid: cd590fad-022c-4b9a-97f5-3262080d772d
> State: Peer in Cluster (Disconnected)
>
> Hostname: jc1letgfs7
> Uuid: c5f40de4-9bb1-47ad-93b6-d52c6689ee29
> State: Peer in Cluster (Connected)
>
> Hostname: jc1letgfs8
> Uuid: 13f4ce3f-042e-4144-a76c-d2b1b91676bd
> State: Peer in Cluster (Connected)
>
> BUT ... after 'service glusterd restart' on node 5:
>
> root at jc1letgfs5:/etc/glusterd/vols# gluster peer status
> Number of Peers: 3
>
> Hostname: jc1letgfs7
> Uuid: c5f40de4-9bb1-47ad-93b6-d52c6689ee29
> State: Peer in Cluster (Connected)
>
> Hostname: jc1letgfs8
> Uuid: 13f4ce3f-042e-4144-a76c-d2b1b91676bd
> State: Peer in Cluster (Connected)
>
> Hostname: jc1letgfs6
> Uuid: cd590fad-022c-4b9a-97f5-3262080d772d
> State: Peer in Cluster (Connected)
>
> Works now. Thanks so much. I suspect a race condition of some sort, though what I'll leave up to the devs.
>
> -----Original Message-----
> From: Mohit Anchlia [mailto:mohitanchlia at gmail.com]
> Sent: Monday, March 21, 2011 2:57 PM
> To: Burnash, James; gluster-users at gluster.org
> Subject: Re: [Gluster-users] What does this error mean?
>
> At this point can you do /etc/init.d/gluster stop and then start and
> see if this changes anything? Or do you see same behaviour? I am
> thinking gluster might have tried to start too soon on reboot.
>
> On Mon, Mar 21, 2011 at 11:43 AM, Burnash, James <jburnash at knight.com> wrote:
>> Short answers - yes all on the same subnet.
>> Every host can ping the others
>> Iptables shows empty entries for all filters
>>
>> Details are here - http://pastebin.com/eKtRMbGE
>>
>> I did explicitly turn the iptables off again, and then checked again:
>>
>> jc1letgfs5
>> Firewall is stopped.
>>
>> jc1letgfs6
>> Firewall is stopped.
>>
>> jc1letgfs7
>> Firewall is stopped.
>>
>> jc1letgfs8
>> Firewall is stopped.
>>
>> Thanks,
>>
>> James
>>
>> -----Original Message-----
>> From: Mohit Anchlia [mailto:mohitanchlia at gmail.com]
>> Sent: Monday, March 21, 2011 2:25 PM
>> To: Burnash, James
>> Cc: gluster-users at gluster.org
>> Subject: Re: [Gluster-users] What does this error mean?
>>
>> Are they in same subnet? What happens if you ping these hosts
>> individually? Do they ping?
>>
>> I closely looked at the error you posted and "connection to
>> 10.20.72.157:24007 failed (No route to host" points to either firewall
>> issue or could be a switch issue on the network. Ping test on each
>> host to each other will be helpful.
>>
>> Can you post results of ping and also "service iptables status" from each node?
>>
>> On Mon, Mar 21, 2011 at 11:16 AM, Burnash, James <jburnash at knight.com> wrote:
>>> A little more information:
>>>
>>> From the original (first peer node):
>>> root at jc1letgfs5:/etc/glusterd/vols# gluster peer status
>>> Number of Peers: 3
>>>
>>> Hostname: jc1letgfs6
>>> Uuid: cd590fad-022c-4b9a-97f5-3262080d772d
>>> State: Peer in Cluster (Disconnected)
>>>
>>> Hostname: jc1letgfs7
>>> Uuid: c5f40de4-9bb1-47ad-93b6-d52c6689ee29
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: jc1letgfs8
>>> Uuid: 13f4ce3f-042e-4144-a76c-d2b1b91676bd
>>> State: Peer in Cluster (Connected)
>>>
>>>
>>> From the problem node:
>>> *** NOTE - only one Peer seen
>>> root at jc1letgfs6:~# gluster peer status
>>> Number of Peers: 1
>>>
>>> Hostname: 10.20.72.156
>>> Uuid: 95e1d79a-632a-4774-9d7e-a7234cb084ca
>>> State: Peer in Cluster (Connected)
>>>
>>>
>>> From a different peer node:
>>> root at jc1letgfs8:~# gluster peer status
>>> Number of Peers: 3
>>>
>>> Hostname: jc1letgfs6
>>> Uuid: cd590fad-022c-4b9a-97f5-3262080d772d
>>> State: Peer Rejected (Connected)
>>>
>>> Hostname: jc1letgfs7
>>> Uuid: c5f40de4-9bb1-47ad-93b6-d52c6689ee29
>>> State: Peer in Cluster (Connected)
>>>
>>> Hostname: 10.20.72.156
>>> Uuid: 95e1d79a-632a-4774-9d7e-a7234cb084ca
>>> State: Peer in Cluster (Connected)
>>>
>>> -----Original Message-----
>>> From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James
>>> Sent: Monday, March 21, 2011 2:05 PM
>>> To: Mohit Anchlia
>>> Cc: gluster-users at gluster.org
>>> Subject: Re: [Gluster-users] What does this error mean?
>>>
>>> I did do this, and noting in particular stands out.
>>>
>>> I'll exercise it some more, and see if we can get something that will at least point in the proper direction.
>>>
>>> I suspect that another reboot of the affected machine will fix this condition - but it won't help me understand the root problem the next time this happens.
>>>
>>> Thanks,
>>>
>>> James
>>>
>>> -----Original Message-----
>>> From: Mohit Anchlia [mailto:mohitanchlia at gmail.com]
>>> Sent: Monday, March 21, 2011 12:40 PM
>>> To: Burnash, James
>>> Cc: gluster-users at gluster.org
>>> Subject: Re: [Gluster-users] What does this error mean?
>>>
>>> Can you turn on DEBUG and see if there is something that stands out?
>>>
>>> On Mon, Mar 21, 2011 at 9:34 AM, Burnash, James <jburnash at knight.com> wrote:
>>>> Does anybody have any clue as to why this is happening? The problem has persisted for several days now, but I can't find anything at all in the logs to possibly explain why this is so.
>>>>
>>>> -----Original Message-----
>>>> From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Burnash, James
>>>> Sent: Wednesday, March 16, 2011 9:10 AM
>>>> To: gluster-users at gluster.org
>>>> Subject: [SPAM?] [Gluster-users] What does this error mean?
>>>> Importance: Low
>>>>
>>>> Hello.
>>>>
>>>> After purposely crashing (via ' echo b>/proc/sysrq-trigger ) node jc1letgfs6 to test mirroring, even after the node has rebooted and is back online I am still seeing the statement "Disconnected" for that node when I execute the following command on the first storage node:
>>>>
>>>> root at jc1letgfs5:/etc/glusterd/vols# gluster peer status
>>>> Number of Peers: 3
>>>>
>>>> Hostname: jc1letgfs6
>>>> Uuid: cd590fad-022c-4b9a-97f5-3262080d772d
>>>> State: Peer in Cluster (Disconnected)
>>>>
>>>> Hostname: jc1letgfs7
>>>> Uuid: c5f40de4-9bb1-47ad-93b6-d52c6689ee29
>>>> State: Peer in Cluster (Disconnected)
>>>>
>>>> Hostname: jc1letgfs8
>>>> Uuid: 13f4ce3f-042e-4144-a76c-d2b1b91676bd
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> This is running on 4 servers with CentOS 5.5 (x86_64), GlusterFS 3.1.1
>>>>
>>>> Here is the volume info:
>>>>
>>>> # gluster volume info
>>>>
>>>> Volume Name: test-pfs-ro1
>>>> Type: Distributed-Replicate
>>>> Status: Started
>>>> Number of Bricks: 4 x 2 = 8
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: jc1letgfs5:/export/read-only/g01
>>>> Brick2: jc1letgfs6:/export/read-only/g01
>>>> Brick3: jc1letgfs5:/export/read-only/g02
>>>> Brick4: jc1letgfs6:/export/read-only/g02
>>>> Brick5: jc1letgfs7:/export/read-only/g01
>>>> Brick6: jc1letgfs8:/export/read-only/g01
>>>> Brick7: jc1letgfs7:/export/read-only/g02
>>>> Brick8: jc1letgfs8:/export/read-only/g02
>>>> Options Reconfigured:
>>>> performance.stat-prefetch: on
>>>> performance.cache-size: 2GB
>>>> network.ping-timeout: 10
>>>>
>>>> Even with this error, mirroring functions as expected, and the node is recognized and utilized, as can be seen in this log fragment from jc1letgfs5: /var/log/glusterfs/etc-glusterfs-glusterd.vol.log
>>>>
>>>> [2011-03-13 23:51:31.458329] E [socket.c:1656:socket_connect_finish] management: connection to 10.20.72.157:24007 failed (No route to ho
>>>> st)
>>>> [2011-03-13 23:53:49.42170] I [glusterd3_1-mops.c:172:glusterd3_1_friend_add_cbk] glusterd: Received ACC from uuid: cd590fad-022c-4b9a-9
>>>> 7f5-3262080d772d, host: jc1letgfs6, port: 0
>>>> [2011-03-13 23:53:49.42204] I [glusterd-utils.c:2062:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster
>>>> [2011-03-13 23:53:49.42320] I [glusterd-utils.c:2062:glusterd_friend_find_by_uuid] glusterd: Friend found.. state: Peer in Cluster
>>>> [2011-03-13 23:53:49.42336] I [glusterd-handler.c:2267:glusterd_handle_friend_update] glusterd: Received friend update from uuid: cd590f
>>>> ad-022c-4b9a-97f5-3262080d772d
>>>> [2011-03-13 23:53:49.42359] I [glusterd-handler.c:2312:glusterd_handle_friend_update] : Received uuid: 95e1d79a-632a-4774-9d7e-a7234cb08
>>>> 4ca, hostname:10.20.72.156
>>>> [2011-03-13 23:53:49.42412] I [glusterd-handler.c:2315:glusterd_handle_friend_update] : Received my uuid as Friend
>>>>
>>>>
>>>> Any pointers or help would be appreciated.
>>>>
>>>> James Burnash, Unix Engineering
>>>>
>>>>
>>>> DISCLAIMER:
>>>> This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission.
>>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>
>