[Gluster-users] NFS/RPC/Glusterd error on one brick
Atin Mukherjee
amukherj at redhat.com
Fri Nov 27 07:11:45 UTC 2015
I think your setup is messed up. peer status from these two nodes are
completely out of sync. Another thumb-rule related to peer probe is you
should probe a node from your existing cluster not the vice versa. I'd
request you to clean up /var/lib/glusterd/* from the faulty node,
restart glusterd service and then attempt to probe this node from either
main/glusternode2/glusternode4. That should work.
Thanks,
Atin
On 11/27/2015 12:01 PM, Frank Rothenstein wrote:
> Hi Atin,
>
> my problems also started by not connecting correctly, so I followed
> advices to reconnect by deleting contents of /var/lib/gluster, then
> peer probe an sync. It syncs ok, the "main" hosts peers correctly with
> another one, which has no bricks atm:
>
> main:
> gluster peer status
> Number of Peers: 2
>
> Hostname: glusternode2
> Uuid: 5fd7c455-4eef-4e1f-9ef4-91b47748bebc
> State: Sent and Received peer request (Connected)
> Other names:
> 192.168.222.101
> 139.64.206.101
>
> Hostname: glusternode4
> Uuid: 4a7bd9b7-40d2-49ba-8ec6-eefcf46b6e5b
> State: Peer in Cluster (Connected)
> Other names:
> 139.64.206.103
> 192.168.222.103
>
> faulty:
>
> gluster peer probe glusternode4
> peer probe: failed: Probe returned with Transport endpoint is not
> connected
>
> gluster peer status
> Number of Peers: 1
>
> Hostname: glusternode1
> Uuid: 56a2baab-6e33-42e7-9d4d-f5fe3c9fcdd4
> State: Probe Sent to Peer (Connected)
> Other names:
> 139.64.206.100
> 192.168.206.100
>
> What I don't understand is, why no .socket files are created in
> /var/run/gluster and rpcinfo shows no registered services...
>
> When I started the NFS-Kernel-Server for testing purposes these
> services got registered...
>
> Enough space left, matching permissions, same versions as on the other
> host - I'm still stuck.
>
> Thanks, Frank
>
> Am Donnerstag, den 26.11.2015, 22:12 +0530 schrieb Atin Mukherjee:
>> The logs seems to be normal, do you see any difference in peer status
>> command output from different nodes?
>> -Atin
>> Sent from one plus one
>> On Nov 26, 2015 9:04 PM, "Frank Rothenstein" <f.rothenstein at bodden-kl
>> iniken.de> wrote:
>>> Today's cli.log: http://pastebin.com/MjbxRkTj
>>> Last start of glusterd.service, etc-glusterfs-
>>> glusterd.vol.log: http://
>>> pastebin.com/21gDJZwb
>>> A lot of output from glustershd.log: http://pastebin.com/rTXMqTPC
>>>
>>>
>>> Am Donnerstag, den 26.11.2015, 20:41 +0530 schrieb Atin Mukherjee:
>>>> Please attach the glusterd log of the faulty node.
>>>> -Atin
>>>> Sent from one plus one
>>>> On Nov 26, 2015 8:31 PM, "Frank Rothenstein" <f.rothenstein at bodde
>>> n-kl
>>>> iniken.de> wrote:
>>>>> Hey @all,
>>>>>
>>>>> I'm having trouble on one of my gluster hosts/peers:
>>>>> glusterd is starting but won't connect properly to other
>>> peer(s).
>>>>>
>>>>> I'm stuck, after reading Bugzilla/Google Groups/Gluster Users
>>> etc I
>>>>> still have no clue why Gluster won't start only on this
>>> particular
>>>>> machine. It's Centos 7, i have another machine (same hardware,
>>> same
>>>>> config) running with no problems.
>>>>> The weirdest part is that the cluster ran before and since the
>>>>> upgrade
>>>>> with only minor problems. Then i had to restart the machines
>>> and
>>>>> this
>>>>> one never came back properly.
>>>>> I'm can't find the cause and appreciate any help.
>>>>>
>>>>> rpcinfo -p
>>>>> program vers proto port service
>>>>> 100000 4 tcp 111 portmapper
>>>>> 100000 3 tcp 111 portmapper
>>>>> 100000 2 tcp 111 portmapper
>>>>> 100000 4 udp 111 portmapper
>>>>> 100000 3 udp 111 portmapper
>>>>> 100000 2 udp 111 portmapper
>>>>> 100024 1 udp 48184 status
>>>>> 100024 1 tcp 34779 status
>>>>>
>>>>> Then of course:
>>>>>
>>>>> showmount -e
>>>>> clnt_create: RPC: Program not registered
>>>>>
>>>>>
>>>>> No .socket-files
>>>>>
>>>>> ls /var/run/gluster
>>>>> snaps
>>>>>
>>>>> If I run glusterd in DEBUG I'm constantly getting:
>>>>>
>>>>> [2015-11-26 14:30:05.290866] D [socket.c:558:__socket_rwv] 0-
>>>>> socket.management: EOF on socket
>>>>> [2015-11-26 14:30:05.290886] D
>>> [socket.c:2355:socket_event_handler]
>>>>> 0-
>>>>> transport: disconnecting now
>>>>> [2015-11-26 14:30:05.290903] D [MSGID: 0] [event-
>>>>> epoll.c:591:event_dispatch_epoll_handler] 0-epoll: generation
>>>>> bumped on
>>>>> idx=4 from gen=382 to slot->gen=383, fd=10, slot->fd=10
>>>>>
>>>>>
>>>>> ls -al /var/run/gluster
>>>>> total 0
>>>>> drwxr-xr-x 3 root root 60 Nov 26 12:51 .
>>>>> drwxr-xr-x 37 root root 1180 Nov 26 15:30 ..
>>>>> drwxr-xr-x 2 root root 40 Nov 26 12:51 snaps
>>>>>
>>>>> Thanks in advance
>>>>>
>>>>> Frank
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>> ___________________________________________________________________
>>>>> ___________
>>>>> BODDEN-KLINIKEN Ribnitz-Damgarten GmbH
>>>>> Sandhufe 2
>>>>> 18311 Ribnitz-Damgarten
>>>>>
>>>>> Telefon: 03821-700-0
>>>>> Fax: 03821-700-240
>>>>>
>>>>> E-Mail: info at bodden-kliniken.de Internet: http://www.bodden-k
>>> lini
>>>>> ken.de
>>>>>
>>>>> Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919,
>>> Steuer-
>>>>> Nr.: 079/133/40188
>>>>> Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr.
>>>>> Falko Milski
>>>>>
>>>>> Der Inhalt dieser E-Mail ist ausschließlich für den
>>> bezeichneten
>>>>> Adressaten bestimmt. Wenn Sie nicht der vorge-
>>>>> sehene Adressat dieser E-Mail oder dessen Vertreter sein
>>> sollten,
>>>>> beachten Sie bitte, dass jede Form der Veröf-
>>>>> fentlichung, Vervielfältigung oder Weitergabe des Inhalts
>>> dieser E-
>>>>> Mail unzulässig ist. Wir bitten Sie, sofort den
>>>>> Absender zu informieren und die E-Mail zu löschen.
>>>>>
>>>>>
>>>>> Bodden-Kliniken Ribnitz-Damgarten GmbH 2015
>>>>> *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>>
>>>
>>> ___________________________________________________________________
>>> ___________
>>> BODDEN-KLINIKEN Ribnitz-Damgarten GmbH
>>> Sandhufe 2
>>> 18311 Ribnitz-Damgarten
>>>
>>> Telefon: 03821-700-0
>>> Fax: 03821-700-240
>>>
>>> E-Mail: info at bodden-kliniken.de Internet: http://www.bodden-klini
>>> ken.de
>>>
>>> Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-
>>> Nr.: 079/133/40188
>>> Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr.
>>> Falko Milski
>>>
>>> Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten
>>> Adressaten bestimmt. Wenn Sie nicht der vorge-
>>> sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten,
>>> beachten Sie bitte, dass jede Form der Veröf-
>>> fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-
>>> Mail unzulässig ist. Wir bitten Sie, sofort den
>>> Absender zu informieren und die E-Mail zu löschen.
>>>
>>>
>>> Bodden-Kliniken Ribnitz-Damgarten GmbH 2015
>>> *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***
>>>
>>>
>
>
>
>
>
> ______________________________________________________________________________
> BODDEN-KLINIKEN Ribnitz-Damgarten GmbH
> Sandhufe 2
> 18311 Ribnitz-Damgarten
>
> Telefon: 03821-700-0
> Fax: 03821-700-240
>
> E-Mail: info at bodden-kliniken.de Internet: http://www.bodden-kliniken.de
>
> Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-Nr.: 079/133/40188
> Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. Falko Milski
>
> Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten Adressaten bestimmt. Wenn Sie nicht der vorge-
> sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten, beachten Sie bitte, dass jede Form der Veröf-
> fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-Mail unzulässig ist. Wir bitten Sie, sofort den
> Absender zu informieren und die E-Mail zu löschen.
>
>
> Bodden-Kliniken Ribnitz-Damgarten GmbH 2015
> *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list