[Gluster-users] Trouble to add brick

Karim Latouche klatouche at digitalchocolate.com
Tue May 24 17:02:29 UTC 2011 

Hi

Sorry i forgot to include the log message I got when running a
gluster volume rebalance

> I [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] 
> 0-glusterd: Received rebalance volume 1 on dccbackup
> [2011-05-24 03:59:23.645067] W 
> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: 
> reading from socket failed. Error (Transport endpoint is not 
> connected), peer (127.0.0.1:1019)
> [2011-05-24 03:59:28.323938] I 
> [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] 
> 0-glusterd: Received rebalance volume 1 on dccbackup
> [2011-05-24 03:59:28.352874] W 
> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: 
> reading from socket failed. Error (Transport endpoint is not 
> connected), peer (127.0.0.1:1018)
And this is the output of the command

gluster volume rebalance dccbackup start
starting rebalance on volume dccbackup has been unsuccessful

Thx

KL

> Look in the gluster logs and see what it says. /var/log/gluster*
>
> If you intention is to migrate data then alternatively you can also do:
>
> gluster volume rebalance VOLNAME start
>
>
> On Tue, May 24, 2011 at 2:59 AM, Karim Latouche
> <klatouche at digitalchocolate.com>  wrote:
>> Hi
>>
>> Thx for your help.
>>
>> I finaly could add the last brick .
>>
>> I now have a Distributed-replicated volume
>>
>>> gluster volume info
>>>
>>> Volume Name: dccbackup
>>> Type: Distributed-Replicate
>>> Status: Started
>>> Number of Bricks: 2 x 2 = 4
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: mysqldccbackup:/export
>>> Brick2: mysqldccbackupmirror:/export
>>> Brick3: gfsbrick1:/brick1
>>> Brick4: gfsbrick2:/brick2
>> On the client side though  I don't see any increasing in storage size.
>> I tried to rebalance
>>
>> gluster volume rebalance  dccbackup fix-layout start
>>
>> and got .
>>
>>> starting rebalance on volume dccbackup has been unsuccessful
>> I'm kind of stuck here.
>>
>> Thank you
>>
>> KL
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>> What is the output of:
>>>
>>> # ls /etc/glusterd/peers
>>> 39bd607b-b355-4e0d-80ae-ef507e0dbabe  4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>   a70858c6-cac3-43fa-b743-4d2226f8a433
>>>
>>> Above output is mine.
>>>
>>> If you cat each of these files, there should be one for each peer OTHER
>>> than the host that your are logged into), and hostnames each  files, like
>>> this:
>>> # cat /etc/glusterd/peers/*
>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>> state=3
>>> hostname1=jc1letgfs15-pfs1
>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>> state=3
>>> hostname1=jc1letgfs18-pfs1
>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>> state=3
>>> hostname1=jc1letgfs14-pfs1
>>>
>>> Here is a listing of all the peer files on all four nodes of my GlusterFS
>>> servers - I created this configuration in the same way you did - two
>>> servers, then add two servers:
>>>
>>> jc1letgfs14
>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>> state=3
>>> hostname1=jc1letgfs15-pfs1
>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f
>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>> state=3
>>> hostname1=jc1letgfs18-pfs1
>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>> state=3
>>> hostname1=10.20.72.191
>>>
>>> jc1letgfs15
>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f
>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>> state=3
>>> hostname1=jc1letgfs18-pfs1
>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>> state=3
>>> hostname1=10.20.72.191
>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433
>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>> state=3
>>> hostname1=jc1letgfs14-pfs1
>>>
>>> jc1letgfs17
>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>> state=3
>>> hostname1=jc1letgfs15-pfs1
>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f
>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>> state=3
>>> hostname1=jc1letgfs18-pfs1
>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433
>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>> state=3
>>> hostname1=jc1letgfs14-pfs1
>>>
>>> jc1letgfs18
>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>> state=3
>>> hostname1=jc1letgfs15-pfs1
>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>> state=3
>>> hostname1=10.20.72.191
>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433
>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>> state=3
>>> hostname1=jc1letgfs14-pfs1
>>>
>>> I hope this is of some help.
>>>
>>> James Burnash
>>> Unix Engineer
>>>
>>>
>>> -----Original Message-----
>>> From: gluster-users-bounces at gluster.org
>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Karim Latouche
>>> Sent: Monday, May 23, 2011 1:02 PM
>>> To: Mohit Anchlia
>>> Cc: gluster-users at gluster.org
>>> Subject: Re: [Gluster-users] Trouble to add brick
>>>
>>> hi
>>>
>>> Here is the output of gluster peer status  command of the four nodes
>>>
>>> mysqldccbackup mysqldccbackupmirror are the two first nodes
>>>
>>> I got issues when I tried to add gfsbrick1 and two to the already working
>>> cluster
>>>
>>>
>>>
>>>> [root at gfsbrick1 ~]# gluster peer status
>>>> Number of Peers: 1
>>>>
>>>> Hostname: 10.0.4.225
>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>> State: Establishing Connection (Connected)
>>>> [root at gfsbrick2 ~]# gluster peer status
>>>> Number of Peers: 2
>>>>
>>>> Hostname: 10.0.4.225
>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: mysqldccbackupmirror
>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939
>>>> State: Peer Rejected (Connected)
>>>> [root at mysqldccbackup ~]# gluster peer status
>>>> Number of Peers: 3
>>>>
>>>> Hostname: mysqldccbackupmirror
>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: gfsbrick2
>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f
>>>> State: Peer in Cluster (Connected)
>>>>
>>>> Hostname: gfsbrick1
>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>> State: Establishing Connection (Connected)
>>>
>>>> root at mysqldccbackupmirror ~]# gluster peer status
>>>> Number of Peers: 1
>>>>
>>>> Hostname: 10.0.4.225
>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6
>>>> State: Peer in Cluster (Connected)
>>>> [root at mysqldccbackupmirror ~]# gluster peer status
>>>> Number of Peers: 1
>>>>
>>>> Hostname: 10.0.4.225
>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6
>>>> State: Peer in Cluster (Connected)
>>> Thx a lot
>>>
>>> KL
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>> Please send the complete output of gluster peer status from both the
>>>> nodes.
>>>>
>>>> On Mon, May 23, 2011 at 9:44 AM, Karim Latouche
>>>> <klatouche at digitalchocolate.com>     wrote:
>>>>> hi
>>>>>
>>>>> Thx for your answer.
>>>>>
>>>>> I should have mention that I already set up a proper hosts file to avoid
>>>>> DNS
>>>>> resolution issue but I still have same issue.
>>>>>
>>>>> The last brick is allways the one having issues to be added . I tried
>>>>> adding
>>>>> gfsbrick2 first then gfsbrick1 and the result is the same
>>>>>
>>>>> I get
>>>>>
>>>>>> Hostname: gfsbrick2
>>>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f
>>>>>> State: Peer in Cluster (Connected)
>>>>>>
>>>>>> Hostname: gfsbrick1
>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>> State: Establishing Connection (Connected)
>>>>> instead of
>>>>>
>>>>>
>>>>>> Hostname: gfsbrick2
>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>> State: Establishing Connection (Connected)
>>>>> It 's really strange.
>>>>>
>>>>>
>>>>> Thx
>>>>>
>>>>> KL
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Karim,
>>>>>>       gfsbrick1 is not able to contact gfsbrick2 yet. Generally happens
>>>>>> if
>>>>>> the DNS resolutions dont happen as expected. You can map them manually
>>>>>> in
>>>>>> /etc/hosts for both the machines?.
>>>>>>
>>>>>> something like:
>>>>>> on gfsbrick1
>>>>>> <ip-of-brick2>       gfsbrick2
>>>>>> on gfsbrick2
>>>>>> <ip-of-brick1>       gfsbrick1
>>>>>>
>>>>>> Pranith.
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: "Karim Latouche"<klatouche at digitalchocolate.com>
>>>>>> To: gluster-users at gluster.org
>>>>>> Sent: Monday, May 23, 2011 7:56:22 PM
>>>>>> Subject: [Gluster-users] Trouble to add brick
>>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I got a strange problems
>>>>>>
>>>>>> I run a two bricks replicated volume and I tried to add 2 more bricks
>>>>>>
>>>>>> The first one went ok
>>>>>>
>>>>>>> gluster peer probe gfsbrick1
>>>>>>> Probe successful
>>>>>> It seems I can't add the second one
>>>>>>
>>>>>> I get a
>>>>>>
>>>>>>> Hostname: gfsbrick2
>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>>> State: Establishing Connection (Connected)
>>>>>> Did anyone encounter this issue ?
>>>>>>
>>>>>> Thx
>>>>>>
>>>>>> KL
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>
>>>
>>> DISCLAIMER:
>>> This e-mail, and any attachments thereto, is intended only for use by the
>>> addressee(s) named herein and may contain legally privileged and/or
>>> confidential information. If you are not the intended recipient of this
>>> e-mail, you are hereby notified that any dissemination, distribution or
>>> copying of this e-mail, and any attachments thereto, is strictly prohibited.
>>> If you have received this in error, please immediately notify me and
>>> permanently delete the original and any copy of any e-mail and any printout
>>> thereof. E-mail transmission cannot be guaranteed to be secure or
>>> error-free. The sender therefore does not accept liability for any errors or
>>> omissions in the contents of this message which arise as a result of e-mail
>>> transmission.
>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at
>>> its discretion, monitor and review the content of all e-mail communications.
>>> http://www.knight.com
>>

More information about the Gluster-users mailing list