[Gluster-users] add bricks on distributed replicated volume failed

Mohit Anchlia mohitanchlia at gmail.com
Thu Sep 1 16:54:52 UTC 2011 

Can you ping node05 from node07 from where you are trying to do the
add? Also, try restarting gluster process on every node and try again.

On Thu, Sep 1, 2011 at 9:39 AM, Laurent DOUCHY <Laurent.Douchy at unige.ch> wrote:
> see below
>
> Cheers,
> Laurent DOUCHY.
>
>
> On 9/1/11 6:01 PM, Mohit Anchlia wrote:
>>
>> You can check few things on 5 and 6:
>>
>> 1) gluster processes are running on node5 and 6
>
> yes:
>
> node05
>
> root      4902     1  0 Aug31 ?        00:00:00
> /opt/glusterfs/3.2.2/sbin/glusterd
> root      9626     1  0 19:55 ?        00:00:00
> /opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p
> /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
> root      9690  9686  0 20:04 ?        00:00:00 bash -c ps -edf | grep
> gluster
> root      9704  9690  0 20:04 ?        00:00:00 grep gluster
>
> node06
>
> root      4441     1  0 Aug31 ?        00:00:00
> /opt/glusterfs/3.2.2/sbin/glusterd
> root      9178     1  0 19:55 ?        00:00:00
> /opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p
> /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
> root      9242  9238  0 20:04 ?        00:00:00 bash -c ps -edf | grep
> gluster
> root      9256  9242  0 20:04 ?        00:00:00 grep gluster
>
>> 2) both nodes are able to see each other
>
> yes:
>
> [root at node05 ~]# ping node06
> PING node06.isdc.unige.ch (129.194.168.70) 56(84) bytes of data.
> 64 bytes from node06.isdc.unige.ch (129.194.168.70): icmp_seq=1 ttl=64
> time=0.376 ms
>
> [root at node06 ~]# ping node05
> PING node05.isdc.unige.ch (129.194.168.69) 56(84) bytes of data.
> 64 bytes from node05.isdc.unige.ch (129.194.168.69): icmp_seq=1 ttl=64
> time=0.337 ms
>>
>> 3) do gluster peer status on both the nodes and see what you see
>
> node 5 trust node 6 and node 6 trust node 5
>
> [root at node05 ~]# gluster peer status
> Number of Peers: 5
>
> Hostname: node08
> Uuid: dd114546-5b94-4a62-9301-260703bf5707
> State: Peer in Cluster (Connected)
>
> Hostname: node06
> Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
> State: Peer in Cluster (Connected)
>
> Hostname: node10
> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
> State: Peer in Cluster (Connected)
>
> Hostname: 129.194.168.71
> Uuid: a35fb0a1-af35-4a04-b38a-434f68369508
> State: Peer in Cluster (Connected)
>
> Hostname: node09
> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
> State: Peer in Cluster (Connected)
>
>
>
>
> [root at node06 ~]# gluster peer status
> Number of Peers: 5
>
> Hostname: node08
> Uuid: dd114546-5b94-4a62-9301-260703bf5707
> State: Peer in Cluster (Connected)
>
> Hostname: node09
> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
> State: Peer in Cluster (Connected)
>
> Hostname: node05
> Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
> State: Peer in Cluster (Connected)
>
> Hostname: node10
> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
> State: Peer in Cluster (Connected)
>
> Hostname: 129.194.168.71
> Uuid: a35fb0a1-af35-4a04-b38a-434f68369508
> State: Peer in Cluster (Connected)
>
>
>> 4) check iptables
>
> same file on each node (the installation is manage by puppet)
>>
>> On Thu, Sep 1, 2011 at 8:57 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch>
>>  wrote:
>>>
>>> It works ...
>>>
>>> [root at node07 ~]# gluster volume add-brick cluster node09:/gluster3
>>> node10:/gluster3
>>> Add Brick successful
>>>
>>>
>>> On 9/1/11 5:39 PM, Mohit Anchlia wrote:
>>>>
>>>> Can you try with node09:/gluster3 and node10:gluster3 instead?
>>>>
>>>> On Thu, Sep 1, 2011 at 2:49 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch>
>>>>  wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> I work on the node7 so it doesn't appear on the list.
>>>>>
>>>>> I create a folder /gluster3 on node5 and node6 and try to add them to
>>>>> my
>>>>> volume but it failed with the same message :(
>>>>>
>>>>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster3
>>>>> node06:/gluster3
>>>>> Operation failed on node05
>>>>>
>>>>> next step is to reinstall from scratch the node I hope I can avoid
>>>>> this.
>>>>>
>>>>> On 8/31/11 9:08 PM, Mohit Anchlia wrote:
>>>>>>
>>>>>> I don't see node07 in above output of gluster peer status.
>>>>>>
>>>>>> Can you try to add bricks on the hosts that gluster1, gluster2? So add
>>>>>> gluster3 and see if that works.
>>>>>>
>>>>>> On Wed, Aug 31, 2011 at 11:56 AM, Laurent DOUCHY
>>>>>> <Laurent.Douchy at unige.ch>      wrote:
>>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I try to add 2 bricks or 4 bricks for the same effect.
>>>>>>>
>>>>>>> I try to reinstall gluster without success.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Laurent DOUCHY.
>>>>>>>
>>>>>>>
>>>>>>> On 8/31/11 8:07 PM, Burnash, James wrote:
>>>>>>>>
>>>>>>>> Hi Laurent.
>>>>>>>>
>>>>>>>> Since your configuration specifies replication, you must add bricks
>>>>>>>> in
>>>>>>>> the
>>>>>>>> same number as your number of replicas.
>>>>>>>>
>>>>>>>> For instance - if you have 2 replicas (most normal case), you would
>>>>>>>> need
>>>>>>>> to do something like this:
>>>>>>>>
>>>>>>>> gluster volume add-brick cluster node05:/gluster1 node06:/gluster1
>>>>>>>>
>>>>>>>> James Burnash
>>>>>>>> Unix Engineer
>>>>>>>> Knight Capital Group
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: gluster-users-bounces at gluster.org
>>>>>>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Laurent
>>>>>>>> DOUCHY
>>>>>>>> Sent: Wednesday, August 31, 2011 12:49 PM
>>>>>>>> To: gluster-users at gluster.org
>>>>>>>> Subject: [Gluster-users] add bricks on distributed replicated volume
>>>>>>>> failed
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I'm using gluster 3.2.2 on 10 nodes. Each node have 2x2 TB disk for
>>>>>>>> gluster.
>>>>>>>>
>>>>>>>> I manage to configure a distributed and replicated volume on 4 nodes
>>>>>>>> :
>>>>>>>>
>>>>>>>> [root at node07 ~]# gluster volume info cluster
>>>>>>>>
>>>>>>>> Volume Name: cluster
>>>>>>>> Type: Distributed-Replicate
>>>>>>>> Status: Started
>>>>>>>> Number of Bricks: 4 x 2 = 8
>>>>>>>> Transport-type: tcp
>>>>>>>> Bricks:
>>>>>>>> Brick1: node09:/gluster1
>>>>>>>> Brick2: node10:/gluster1
>>>>>>>> Brick3: node09:/gluster2
>>>>>>>> Brick4: node10:/gluster2
>>>>>>>> Brick5: node07:/gluster1
>>>>>>>> Brick6: node08:/gluster1
>>>>>>>> Brick7: node07:/gluster2
>>>>>>>> Brick8: node08:/gluster2
>>>>>>>>
>>>>>>>> But I can't add new nodes to this volume
>>>>>>>>
>>>>>>>> [root at node07 ~]# gluster peer status
>>>>>>>> Number of Peers: 5
>>>>>>>>
>>>>>>>> Hostname: node10
>>>>>>>> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>
>>>>>>>> Hostname: node08
>>>>>>>> Uuid: dd114546-5b94-4a62-9301-260703bf5707
>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>
>>>>>>>> Hostname: node09
>>>>>>>> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>
>>>>>>>> Hostname: node06
>>>>>>>> Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>
>>>>>>>> Hostname: node05
>>>>>>>> Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster1
>>>>>>>> node06:/gluster1 node05:/gluster2 node06:/gluster2 Operation failed
>>>>>>>> on
>>>>>>>> node05
>>>>>>>>
>>>>>>>> I try to detach nodes 5 and 6, restart glusterd do the probe and the
>>>>>>>> add-brick but still nothing ...
>>>>>>>>
>>>>>>>> Did some one have any idea to fix this ?
>>>>>>>>
>>>>>>>> Thanks in advance,
>>>>>>>> Laurent.
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>
>>>>>>>>
>>>>>>>> DISCLAIMER:
>>>>>>>> This e-mail, and any attachments thereto, is intended only for use
>>>>>>>> by
>>>>>>>> the
>>>>>>>> addressee(s) named herein and may contain legally privileged and/or
>>>>>>>> confidential information. If you are not the intended recipient of
>>>>>>>> this
>>>>>>>> e-mail, you are hereby notified that any dissemination, distribution
>>>>>>>> or
>>>>>>>> copying of this e-mail, and any attachments thereto, is strictly
>>>>>>>> prohibited.
>>>>>>>> If you have received this in error, please immediately notify me and
>>>>>>>> permanently delete the original and any copy of any e-mail and any
>>>>>>>> printout
>>>>>>>> thereof. E-mail transmission cannot be guaranteed to be secure or
>>>>>>>> error-free. The sender therefore does not accept liability for any
>>>>>>>> errors or
>>>>>>>> omissions in the contents of this message which arise as a result of
>>>>>>>> e-mail
>>>>>>>> transmission.
>>>>>>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group
>>>>>>>> may,
>>>>>>>> at
>>>>>>>> its discretion, monitor and review the content of all e-mail
>>>>>>>> communications.
>>>>>>>> http://www.knight.com
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>
>

More information about the Gluster-users mailing list