[Gluster-users] Trouble to add brick
Mohit Anchlia
mohitanchlia at gmail.com
Tue May 24 17:09:11 UTC 2011
Other people have seen these too. Seems like some kind of deadlock
issue. What commands did you run prior to running rebalance command?
Can you restart /etc/init.d/glusterd on all the servers and try again?
On Tue, May 24, 2011 at 10:02 AM, Karim Latouche
<klatouche at digitalchocolate.com> wrote:
> Hi
>
> Sorry i forgot to include the log message I got when running a
> gluster volume rebalance
>
>> I [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log]
>> 0-glusterd: Received rebalance volume 1 on dccbackup
>> [2011-05-24 03:59:23.645067] W
>> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading
>> from socket failed. Error (Transport endpoint is not connected), peer
>> (127.0.0.1:1019)
>> [2011-05-24 03:59:28.323938] I
>> [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] 0-glusterd:
>> Received rebalance volume 1 on dccbackup
>> [2011-05-24 03:59:28.352874] W
>> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading
>> from socket failed. Error (Transport endpoint is not connected), peer
>> (127.0.0.1:1018)
>
> And this is the output of the command
>
> gluster volume rebalance dccbackup start
> starting rebalance on volume dccbackup has been unsuccessful
>
> Thx
>
> KL
>
>> Look in the gluster logs and see what it says. /var/log/gluster*
>>
>> If you intention is to migrate data then alternatively you can also do:
>>
>> gluster volume rebalance VOLNAME start
>>
>>
>> On Tue, May 24, 2011 at 2:59 AM, Karim Latouche
>> <klatouche at digitalchocolate.com> wrote:
>>>
>>> Hi
>>>
>>> Thx for your help.
>>>
>>> I finaly could add the last brick .
>>>
>>> I now have a Distributed-replicated volume
>>>
>>>> gluster volume info
>>>>
>>>> Volume Name: dccbackup
>>>> Type: Distributed-Replicate
>>>> Status: Started
>>>> Number of Bricks: 2 x 2 = 4
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: mysqldccbackup:/export
>>>> Brick2: mysqldccbackupmirror:/export
>>>> Brick3: gfsbrick1:/brick1
>>>> Brick4: gfsbrick2:/brick2
>>>
>>> On the client side though I don't see any increasing in storage size.
>>> I tried to rebalance
>>>
>>> gluster volume rebalance dccbackup fix-layout start
>>>
>>> and got .
>>>
>>>> starting rebalance on volume dccbackup has been unsuccessful
>>>
>>> I'm kind of stuck here.
>>>
>>> Thank you
>>>
>>> KL
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> What is the output of:
>>>>
>>>> # ls /etc/glusterd/peers
>>>> 39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>> 4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>> a70858c6-cac3-43fa-b743-4d2226f8a433
>>>>
>>>> Above output is mine.
>>>>
>>>> If you cat each of these files, there should be one for each peer OTHER
>>>> than the host that your are logged into), and hostnames each files,
>>>> like
>>>> this:
>>>> # cat /etc/glusterd/peers/*
>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>> state=3
>>>> hostname1=jc1letgfs15-pfs1
>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>> state=3
>>>> hostname1=jc1letgfs18-pfs1
>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>>> state=3
>>>> hostname1=jc1letgfs14-pfs1
>>>>
>>>> Here is a listing of all the peer files on all four nodes of my
>>>> GlusterFS
>>>> servers - I created this configuration in the same way you did - two
>>>> servers, then add two servers:
>>>>
>>>> jc1letgfs14
>>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>> state=3
>>>> hostname1=jc1letgfs15-pfs1
>>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>> state=3
>>>> hostname1=jc1letgfs18-pfs1
>>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>> state=3
>>>> hostname1=10.20.72.191
>>>>
>>>> jc1letgfs15
>>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>> state=3
>>>> hostname1=jc1letgfs18-pfs1
>>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>> state=3
>>>> hostname1=10.20.72.191
>>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433
>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>>> state=3
>>>> hostname1=jc1letgfs14-pfs1
>>>>
>>>> jc1letgfs17
>>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>> state=3
>>>> hostname1=jc1letgfs15-pfs1
>>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>> state=3
>>>> hostname1=jc1letgfs18-pfs1
>>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433
>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>>> state=3
>>>> hostname1=jc1letgfs14-pfs1
>>>>
>>>> jc1letgfs18
>>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>> state=3
>>>> hostname1=jc1letgfs15-pfs1
>>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>> state=3
>>>> hostname1=10.20.72.191
>>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433
>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>>> state=3
>>>> hostname1=jc1letgfs14-pfs1
>>>>
>>>> I hope this is of some help.
>>>>
>>>> James Burnash
>>>> Unix Engineer
>>>>
>>>>
>>>> -----Original Message-----
>>>> From: gluster-users-bounces at gluster.org
>>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Karim Latouche
>>>> Sent: Monday, May 23, 2011 1:02 PM
>>>> To: Mohit Anchlia
>>>> Cc: gluster-users at gluster.org
>>>> Subject: Re: [Gluster-users] Trouble to add brick
>>>>
>>>> hi
>>>>
>>>> Here is the output of gluster peer status command of the four nodes
>>>>
>>>> mysqldccbackup mysqldccbackupmirror are the two first nodes
>>>>
>>>> I got issues when I tried to add gfsbrick1 and two to the already
>>>> working
>>>> cluster
>>>>
>>>>
>>>>
>>>>> [root at gfsbrick1 ~]# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: 10.0.4.225
>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>> State: Establishing Connection (Connected)
>>>>> [root at gfsbrick2 ~]# gluster peer status
>>>>> Number of Peers: 2
>>>>>
>>>>> Hostname: 10.0.4.225
>>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> Hostname: mysqldccbackupmirror
>>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939
>>>>> State: Peer Rejected (Connected)
>>>>> [root at mysqldccbackup ~]# gluster peer status
>>>>> Number of Peers: 3
>>>>>
>>>>> Hostname: mysqldccbackupmirror
>>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> Hostname: gfsbrick2
>>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f
>>>>> State: Peer in Cluster (Connected)
>>>>>
>>>>> Hostname: gfsbrick1
>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>> State: Establishing Connection (Connected)
>>>>
>>>>> root at mysqldccbackupmirror ~]# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: 10.0.4.225
>>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6
>>>>> State: Peer in Cluster (Connected)
>>>>> [root at mysqldccbackupmirror ~]# gluster peer status
>>>>> Number of Peers: 1
>>>>>
>>>>> Hostname: 10.0.4.225
>>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6
>>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Thx a lot
>>>>
>>>> KL
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> Please send the complete output of gluster peer status from both the
>>>>> nodes.
>>>>>
>>>>> On Mon, May 23, 2011 at 9:44 AM, Karim Latouche
>>>>> <klatouche at digitalchocolate.com> wrote:
>>>>>>
>>>>>> hi
>>>>>>
>>>>>> Thx for your answer.
>>>>>>
>>>>>> I should have mention that I already set up a proper hosts file to
>>>>>> avoid
>>>>>> DNS
>>>>>> resolution issue but I still have same issue.
>>>>>>
>>>>>> The last brick is allways the one having issues to be added . I tried
>>>>>> adding
>>>>>> gfsbrick2 first then gfsbrick1 and the result is the same
>>>>>>
>>>>>> I get
>>>>>>
>>>>>>> Hostname: gfsbrick2
>>>>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f
>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>
>>>>>>> Hostname: gfsbrick1
>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>>> State: Establishing Connection (Connected)
>>>>>>
>>>>>> instead of
>>>>>>
>>>>>>
>>>>>>> Hostname: gfsbrick2
>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>>> State: Establishing Connection (Connected)
>>>>>>
>>>>>> It 's really strange.
>>>>>>
>>>>>>
>>>>>> Thx
>>>>>>
>>>>>> KL
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Karim,
>>>>>>> gfsbrick1 is not able to contact gfsbrick2 yet. Generally
>>>>>>> happens
>>>>>>> if
>>>>>>> the DNS resolutions dont happen as expected. You can map them
>>>>>>> manually
>>>>>>> in
>>>>>>> /etc/hosts for both the machines?.
>>>>>>>
>>>>>>> something like:
>>>>>>> on gfsbrick1
>>>>>>> <ip-of-brick2> gfsbrick2
>>>>>>> on gfsbrick2
>>>>>>> <ip-of-brick1> gfsbrick1
>>>>>>>
>>>>>>> Pranith.
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> From: "Karim Latouche"<klatouche at digitalchocolate.com>
>>>>>>> To: gluster-users at gluster.org
>>>>>>> Sent: Monday, May 23, 2011 7:56:22 PM
>>>>>>> Subject: [Gluster-users] Trouble to add brick
>>>>>>>
>>>>>>> Hi
>>>>>>>
>>>>>>> I got a strange problems
>>>>>>>
>>>>>>> I run a two bricks replicated volume and I tried to add 2 more bricks
>>>>>>>
>>>>>>> The first one went ok
>>>>>>>
>>>>>>>> gluster peer probe gfsbrick1
>>>>>>>> Probe successful
>>>>>>>
>>>>>>> It seems I can't add the second one
>>>>>>>
>>>>>>> I get a
>>>>>>>
>>>>>>>> Hostname: gfsbrick2
>>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>>>> State: Establishing Connection (Connected)
>>>>>>>
>>>>>>> Did anyone encounter this issue ?
>>>>>>>
>>>>>>> Thx
>>>>>>>
>>>>>>> KL
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>
>>>>
>>>> DISCLAIMER:
>>>> This e-mail, and any attachments thereto, is intended only for use by
>>>> the
>>>> addressee(s) named herein and may contain legally privileged and/or
>>>> confidential information. If you are not the intended recipient of this
>>>> e-mail, you are hereby notified that any dissemination, distribution or
>>>> copying of this e-mail, and any attachments thereto, is strictly
>>>> prohibited.
>>>> If you have received this in error, please immediately notify me and
>>>> permanently delete the original and any copy of any e-mail and any
>>>> printout
>>>> thereof. E-mail transmission cannot be guaranteed to be secure or
>>>> error-free. The sender therefore does not accept liability for any
>>>> errors or
>>>> omissions in the contents of this message which arise as a result of
>>>> e-mail
>>>> transmission.
>>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may,
>>>> at
>>>> its discretion, monitor and review the content of all e-mail
>>>> communications.
>>>> http://www.knight.com
>>>
>
>
More information about the Gluster-users
mailing list