[Gluster-users] Trouble to add brick

Karim Latouche klatouche at digitalchocolate.com
Tue May 24 17:41:55 UTC 2011 

On 5/24/11 8:09 PM, Mohit Anchlia wrote:
> Other people have seen these too. Seems like some kind of deadlock
> issue. What commands did you run prior to running rebalance command?

I tried gluster volume rebalance  dccbackup fix-layout start but was 
unsuccessfull

> Can you restart /etc/init.d/glusterd on all the servers and try again?

I did and still get the same error

I [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] 
0-glusterd: Received rebalance volume 1 on dccbackup
[2011-05-24 04:42:18.521586] W 
[socket.c:1494:__socket_proto_state_machine] 0-socket.management: 
reading from socket failed. Error (Transport endpoint is not connected), 
peer (127.0.0.1:1022)


KL
> On Tue, May 24, 2011 at 10:02 AM, Karim Latouche
> <klatouche at digitalchocolate.com>  wrote:
>> Hi
>>
>> Sorry i forgot to include the log message I got when running a
>> gluster volume rebalance
>>
>>> I [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log]
>>> 0-glusterd: Received rebalance volume 1 on dccbackup
>>> [2011-05-24 03:59:23.645067] W
>>> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading
>>> from socket failed. Error (Transport endpoint is not connected), peer
>>> (127.0.0.1:1019)
>>> [2011-05-24 03:59:28.323938] I
>>> [glusterd-rebalance.c:483:glusterd_rebalance_cmd_attempted_log] 0-glusterd:
>>> Received rebalance volume 1 on dccbackup
>>> [2011-05-24 03:59:28.352874] W
>>> [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading
>>> from socket failed. Error (Transport endpoint is not connected), peer
>>> (127.0.0.1:1018)
>> And this is the output of the command
>>
>> gluster volume rebalance dccbackup start
>> starting rebalance on volume dccbackup has been unsuccessful
>>
>> Thx
>>
>> KL
>>
>>> Look in the gluster logs and see what it says. /var/log/gluster*
>>>
>>> If you intention is to migrate data then alternatively you can also do:
>>>
>>> gluster volume rebalance VOLNAME start
>>>
>>>
>>> On Tue, May 24, 2011 at 2:59 AM, Karim Latouche
>>> <klatouche at digitalchocolate.com>    wrote:
>>>> Hi
>>>>
>>>> Thx for your help.
>>>>
>>>> I finaly could add the last brick .
>>>>
>>>> I now have a Distributed-replicated volume
>>>>
>>>>> gluster volume info
>>>>>
>>>>> Volume Name: dccbackup
>>>>> Type: Distributed-Replicate
>>>>> Status: Started
>>>>> Number of Bricks: 2 x 2 = 4
>>>>> Transport-type: tcp
>>>>> Bricks:
>>>>> Brick1: mysqldccbackup:/export
>>>>> Brick2: mysqldccbackupmirror:/export
>>>>> Brick3: gfsbrick1:/brick1
>>>>> Brick4: gfsbrick2:/brick2
>>>> On the client side though  I don't see any increasing in storage size.
>>>> I tried to rebalance
>>>>
>>>> gluster volume rebalance  dccbackup fix-layout start
>>>>
>>>> and got .
>>>>
>>>>> starting rebalance on volume dccbackup has been unsuccessful
>>>> I'm kind of stuck here.
>>>>
>>>> Thank you
>>>>
>>>> KL
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> What is the output of:
>>>>>
>>>>> # ls /etc/glusterd/peers
>>>>> 39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>>>   4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>>>   a70858c6-cac3-43fa-b743-4d2226f8a433
>>>>>
>>>>> Above output is mine.
>>>>>
>>>>> If you cat each of these files, there should be one for each peer OTHER
>>>>> than the host that your are logged into), and hostnames each  files,
>>>>> like
>>>>> this:
>>>>> # cat /etc/glusterd/peers/*
>>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>>> state=3
>>>>> hostname1=jc1letgfs15-pfs1
>>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>>> state=3
>>>>> hostname1=jc1letgfs18-pfs1
>>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>>>> state=3
>>>>> hostname1=jc1letgfs14-pfs1
>>>>>
>>>>> Here is a listing of all the peer files on all four nodes of my
>>>>> GlusterFS
>>>>> servers - I created this configuration in the same way you did - two
>>>>> servers, then add two servers:
>>>>>
>>>>> jc1letgfs14
>>>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>>> state=3
>>>>> hostname1=jc1letgfs15-pfs1
>>>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>>> state=3
>>>>> hostname1=jc1letgfs18-pfs1
>>>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>>> state=3
>>>>> hostname1=10.20.72.191
>>>>>
>>>>> jc1letgfs15
>>>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>>> state=3
>>>>> hostname1=jc1letgfs18-pfs1
>>>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>>> state=3
>>>>> hostname1=10.20.72.191
>>>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433
>>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>>>> state=3
>>>>> hostname1=jc1letgfs14-pfs1
>>>>>
>>>>> jc1letgfs17
>>>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>>> state=3
>>>>> hostname1=jc1letgfs15-pfs1
>>>>> /etc/glusterd/peers/4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>>> uuid=4583d8b7-f724-4017-b549-9aa9e27dd50f
>>>>> state=3
>>>>> hostname1=jc1letgfs18-pfs1
>>>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433
>>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>>>> state=3
>>>>> hostname1=jc1letgfs14-pfs1
>>>>>
>>>>> jc1letgfs18
>>>>> /etc/glusterd/peers/39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>>> uuid=39bd607b-b355-4e0d-80ae-ef507e0dbabe
>>>>> state=3
>>>>> hostname1=jc1letgfs15-pfs1
>>>>> /etc/glusterd/peers/94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>>> uuid=94f28f8c-8958-435b-80e4-3bd6d4fc5156
>>>>> state=3
>>>>> hostname1=10.20.72.191
>>>>> /etc/glusterd/peers/a70858c6-cac3-43fa-b743-4d2226f8a433
>>>>> uuid=a70858c6-cac3-43fa-b743-4d2226f8a433
>>>>> state=3
>>>>> hostname1=jc1letgfs14-pfs1
>>>>>
>>>>> I hope this is of some help.
>>>>>
>>>>> James Burnash
>>>>> Unix Engineer
>>>>>
>>>>>
>>>>> -----Original Message-----
>>>>> From: gluster-users-bounces at gluster.org
>>>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Karim Latouche
>>>>> Sent: Monday, May 23, 2011 1:02 PM
>>>>> To: Mohit Anchlia
>>>>> Cc: gluster-users at gluster.org
>>>>> Subject: Re: [Gluster-users] Trouble to add brick
>>>>>
>>>>> hi
>>>>>
>>>>> Here is the output of gluster peer status  command of the four nodes
>>>>>
>>>>> mysqldccbackup mysqldccbackupmirror are the two first nodes
>>>>>
>>>>> I got issues when I tried to add gfsbrick1 and two to the already
>>>>> working
>>>>> cluster
>>>>>
>>>>>
>>>>>
>>>>>> [root at gfsbrick1 ~]# gluster peer status
>>>>>> Number of Peers: 1
>>>>>>
>>>>>> Hostname: 10.0.4.225
>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>> State: Establishing Connection (Connected)
>>>>>> [root at gfsbrick2 ~]# gluster peer status
>>>>>> Number of Peers: 2
>>>>>>
>>>>>> Hostname: 10.0.4.225
>>>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6
>>>>>> State: Peer in Cluster (Connected)
>>>>>>
>>>>>> Hostname: mysqldccbackupmirror
>>>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939
>>>>>> State: Peer Rejected (Connected)
>>>>>> [root at mysqldccbackup ~]# gluster peer status
>>>>>> Number of Peers: 3
>>>>>>
>>>>>> Hostname: mysqldccbackupmirror
>>>>>> Uuid: 0961e0d2-e2e8-497d-9c51-4b96c061f939
>>>>>> State: Peer in Cluster (Connected)
>>>>>>
>>>>>> Hostname: gfsbrick2
>>>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f
>>>>>> State: Peer in Cluster (Connected)
>>>>>>
>>>>>> Hostname: gfsbrick1
>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>> State: Establishing Connection (Connected)
>>>>>> root at mysqldccbackupmirror ~]# gluster peer status
>>>>>> Number of Peers: 1
>>>>>>
>>>>>> Hostname: 10.0.4.225
>>>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6
>>>>>> State: Peer in Cluster (Connected)
>>>>>> [root at mysqldccbackupmirror ~]# gluster peer status
>>>>>> Number of Peers: 1
>>>>>>
>>>>>> Hostname: 10.0.4.225
>>>>>> Uuid: 2f4a1869-8c7e-49ea-a7b1-d9b2d364e5d6
>>>>>> State: Peer in Cluster (Connected)
>>>>> Thx a lot
>>>>>
>>>>> KL
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> Please send the complete output of gluster peer status from both the
>>>>>> nodes.
>>>>>>
>>>>>> On Mon, May 23, 2011 at 9:44 AM, Karim Latouche
>>>>>> <klatouche at digitalchocolate.com>       wrote:
>>>>>>> hi
>>>>>>>
>>>>>>> Thx for your answer.
>>>>>>>
>>>>>>> I should have mention that I already set up a proper hosts file to
>>>>>>> avoid
>>>>>>> DNS
>>>>>>> resolution issue but I still have same issue.
>>>>>>>
>>>>>>> The last brick is allways the one having issues to be added . I tried
>>>>>>> adding
>>>>>>> gfsbrick2 first then gfsbrick1 and the result is the same
>>>>>>>
>>>>>>> I get
>>>>>>>
>>>>>>>> Hostname: gfsbrick2
>>>>>>>> Uuid: 14f07412-b2e5-4b34-8980-d180a64b720f
>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>
>>>>>>>> Hostname: gfsbrick1
>>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>>>> State: Establishing Connection (Connected)
>>>>>>> instead of
>>>>>>>
>>>>>>>
>>>>>>>> Hostname: gfsbrick2
>>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>>>> State: Establishing Connection (Connected)
>>>>>>> It 's really strange.
>>>>>>>
>>>>>>>
>>>>>>> Thx
>>>>>>>
>>>>>>> KL
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> Karim,
>>>>>>>>       gfsbrick1 is not able to contact gfsbrick2 yet. Generally
>>>>>>>> happens
>>>>>>>> if
>>>>>>>> the DNS resolutions dont happen as expected. You can map them
>>>>>>>> manually
>>>>>>>> in
>>>>>>>> /etc/hosts for both the machines?.
>>>>>>>>
>>>>>>>> something like:
>>>>>>>> on gfsbrick1
>>>>>>>> <ip-of-brick2>         gfsbrick2
>>>>>>>> on gfsbrick2
>>>>>>>> <ip-of-brick1>         gfsbrick1
>>>>>>>>
>>>>>>>> Pranith.
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>> From: "Karim Latouche"<klatouche at digitalchocolate.com>
>>>>>>>> To: gluster-users at gluster.org
>>>>>>>> Sent: Monday, May 23, 2011 7:56:22 PM
>>>>>>>> Subject: [Gluster-users] Trouble to add brick
>>>>>>>>
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> I got a strange problems
>>>>>>>>
>>>>>>>> I run a two bricks replicated volume and I tried to add 2 more bricks
>>>>>>>>
>>>>>>>> The first one went ok
>>>>>>>>
>>>>>>>>> gluster peer probe gfsbrick1
>>>>>>>>> Probe successful
>>>>>>>> It seems I can't add the second one
>>>>>>>>
>>>>>>>> I get a
>>>>>>>>
>>>>>>>>> Hostname: gfsbrick2
>>>>>>>>> Uuid: 00000000-0000-0000-0000-000000000000
>>>>>>>>> State: Establishing Connection (Connected)
>>>>>>>> Did anyone encounter this issue ?
>>>>>>>>
>>>>>>>> Thx
>>>>>>>>
>>>>>>>> KL
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>
>>>>>
>>>>> DISCLAIMER:
>>>>> This e-mail, and any attachments thereto, is intended only for use by
>>>>> the
>>>>> addressee(s) named herein and may contain legally privileged and/or
>>>>> confidential information. If you are not the intended recipient of this
>>>>> e-mail, you are hereby notified that any dissemination, distribution or
>>>>> copying of this e-mail, and any attachments thereto, is strictly
>>>>> prohibited.
>>>>> If you have received this in error, please immediately notify me and
>>>>> permanently delete the original and any copy of any e-mail and any
>>>>> printout
>>>>> thereof. E-mail transmission cannot be guaranteed to be secure or
>>>>> error-free. The sender therefore does not accept liability for any
>>>>> errors or
>>>>> omissions in the contents of this message which arise as a result of
>>>>> e-mail
>>>>> transmission.
>>>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may,
>>>>> at
>>>>> its discretion, monitor and review the content of all e-mail
>>>>> communications.
>>>>> http://www.knight.com
>>

More information about the Gluster-users mailing list