[Gluster-users] add bricks on distributed replicated volume failed
Laurent DOUCHY
Laurent.Douchy at unige.ch
Thu Sep 1 16:58:26 UTC 2011
ping is ok
restart of gluster done :
[root at node00 ~]# for i in `seq -w 1 10` ; do echo ; echo node$i ; echo ;
ssh node$i "service glusterd restart";done
node01
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node02
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node03
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node04
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node05
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node06
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node07
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node08
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node09
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
node10
Stopping glusterd:[ OK ]
Starting glusterd:[ OK ]
but same error message ...
[root at node07 ~]# gluster volume add-brick cluster node05:/gluster2
node06:/gluster2
Operation failed on node05
Cheers,
Laurent DOUCHY.
On 9/1/11 6:54 PM, Mohit Anchlia wrote:
> Can you ping node05 from node07 from where you are trying to do the
> add? Also, try restarting gluster process on every node and try again.
>
> On Thu, Sep 1, 2011 at 9:39 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch> wrote:
>> see below
>>
>> Cheers,
>> Laurent DOUCHY.
>>
>>
>> On 9/1/11 6:01 PM, Mohit Anchlia wrote:
>>> You can check few things on 5 and 6:
>>>
>>> 1) gluster processes are running on node5 and 6
>> yes:
>>
>> node05
>>
>> root 4902 1 0 Aug31 ? 00:00:00
>> /opt/glusterfs/3.2.2/sbin/glusterd
>> root 9626 1 0 19:55 ? 00:00:00
>> /opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p
>> /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
>> root 9690 9686 0 20:04 ? 00:00:00 bash -c ps -edf | grep
>> gluster
>> root 9704 9690 0 20:04 ? 00:00:00 grep gluster
>>
>> node06
>>
>> root 4441 1 0 Aug31 ? 00:00:00
>> /opt/glusterfs/3.2.2/sbin/glusterd
>> root 9178 1 0 19:55 ? 00:00:00
>> /opt/glusterfs/3.2.2/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p
>> /etc/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log
>> root 9242 9238 0 20:04 ? 00:00:00 bash -c ps -edf | grep
>> gluster
>> root 9256 9242 0 20:04 ? 00:00:00 grep gluster
>>
>>> 2) both nodes are able to see each other
>> yes:
>>
>> [root at node05 ~]# ping node06
>> PING node06.isdc.unige.ch (129.194.168.70) 56(84) bytes of data.
>> 64 bytes from node06.isdc.unige.ch (129.194.168.70): icmp_seq=1 ttl=64
>> time=0.376 ms
>>
>> [root at node06 ~]# ping node05
>> PING node05.isdc.unige.ch (129.194.168.69) 56(84) bytes of data.
>> 64 bytes from node05.isdc.unige.ch (129.194.168.69): icmp_seq=1 ttl=64
>> time=0.337 ms
>>> 3) do gluster peer status on both the nodes and see what you see
>> node 5 trust node 6 and node 6 trust node 5
>>
>> [root at node05 ~]# gluster peer status
>> Number of Peers: 5
>>
>> Hostname: node08
>> Uuid: dd114546-5b94-4a62-9301-260703bf5707
>> State: Peer in Cluster (Connected)
>>
>> Hostname: node06
>> Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
>> State: Peer in Cluster (Connected)
>>
>> Hostname: node10
>> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
>> State: Peer in Cluster (Connected)
>>
>> Hostname: 129.194.168.71
>> Uuid: a35fb0a1-af35-4a04-b38a-434f68369508
>> State: Peer in Cluster (Connected)
>>
>> Hostname: node09
>> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
>> State: Peer in Cluster (Connected)
>>
>>
>>
>>
>> [root at node06 ~]# gluster peer status
>> Number of Peers: 5
>>
>> Hostname: node08
>> Uuid: dd114546-5b94-4a62-9301-260703bf5707
>> State: Peer in Cluster (Connected)
>>
>> Hostname: node09
>> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
>> State: Peer in Cluster (Connected)
>>
>> Hostname: node05
>> Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
>> State: Peer in Cluster (Connected)
>>
>> Hostname: node10
>> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
>> State: Peer in Cluster (Connected)
>>
>> Hostname: 129.194.168.71
>> Uuid: a35fb0a1-af35-4a04-b38a-434f68369508
>> State: Peer in Cluster (Connected)
>>
>>
>>> 4) check iptables
>> same file on each node (the installation is manage by puppet)
>>> On Thu, Sep 1, 2011 at 8:57 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch>
>>> wrote:
>>>> It works ...
>>>>
>>>> [root at node07 ~]# gluster volume add-brick cluster node09:/gluster3
>>>> node10:/gluster3
>>>> Add Brick successful
>>>>
>>>>
>>>> On 9/1/11 5:39 PM, Mohit Anchlia wrote:
>>>>> Can you try with node09:/gluster3 and node10:gluster3 instead?
>>>>>
>>>>> On Thu, Sep 1, 2011 at 2:49 AM, Laurent DOUCHY<Laurent.Douchy at unige.ch>
>>>>> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I work on the node7 so it doesn't appear on the list.
>>>>>>
>>>>>> I create a folder /gluster3 on node5 and node6 and try to add them to
>>>>>> my
>>>>>> volume but it failed with the same message :(
>>>>>>
>>>>>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster3
>>>>>> node06:/gluster3
>>>>>> Operation failed on node05
>>>>>>
>>>>>> next step is to reinstall from scratch the node I hope I can avoid
>>>>>> this.
>>>>>>
>>>>>> On 8/31/11 9:08 PM, Mohit Anchlia wrote:
>>>>>>> I don't see node07 in above output of gluster peer status.
>>>>>>>
>>>>>>> Can you try to add bricks on the hosts that gluster1, gluster2? So add
>>>>>>> gluster3 and see if that works.
>>>>>>>
>>>>>>> On Wed, Aug 31, 2011 at 11:56 AM, Laurent DOUCHY
>>>>>>> <Laurent.Douchy at unige.ch> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I try to add 2 bricks or 4 bricks for the same effect.
>>>>>>>>
>>>>>>>> I try to reinstall gluster without success.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Laurent DOUCHY.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 8/31/11 8:07 PM, Burnash, James wrote:
>>>>>>>>> Hi Laurent.
>>>>>>>>>
>>>>>>>>> Since your configuration specifies replication, you must add bricks
>>>>>>>>> in
>>>>>>>>> the
>>>>>>>>> same number as your number of replicas.
>>>>>>>>>
>>>>>>>>> For instance - if you have 2 replicas (most normal case), you would
>>>>>>>>> need
>>>>>>>>> to do something like this:
>>>>>>>>>
>>>>>>>>> gluster volume add-brick cluster node05:/gluster1 node06:/gluster1
>>>>>>>>>
>>>>>>>>> James Burnash
>>>>>>>>> Unix Engineer
>>>>>>>>> Knight Capital Group
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: gluster-users-bounces at gluster.org
>>>>>>>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Laurent
>>>>>>>>> DOUCHY
>>>>>>>>> Sent: Wednesday, August 31, 2011 12:49 PM
>>>>>>>>> To: gluster-users at gluster.org
>>>>>>>>> Subject: [Gluster-users] add bricks on distributed replicated volume
>>>>>>>>> failed
>>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm using gluster 3.2.2 on 10 nodes. Each node have 2x2 TB disk for
>>>>>>>>> gluster.
>>>>>>>>>
>>>>>>>>> I manage to configure a distributed and replicated volume on 4 nodes
>>>>>>>>> :
>>>>>>>>>
>>>>>>>>> [root at node07 ~]# gluster volume info cluster
>>>>>>>>>
>>>>>>>>> Volume Name: cluster
>>>>>>>>> Type: Distributed-Replicate
>>>>>>>>> Status: Started
>>>>>>>>> Number of Bricks: 4 x 2 = 8
>>>>>>>>> Transport-type: tcp
>>>>>>>>> Bricks:
>>>>>>>>> Brick1: node09:/gluster1
>>>>>>>>> Brick2: node10:/gluster1
>>>>>>>>> Brick3: node09:/gluster2
>>>>>>>>> Brick4: node10:/gluster2
>>>>>>>>> Brick5: node07:/gluster1
>>>>>>>>> Brick6: node08:/gluster1
>>>>>>>>> Brick7: node07:/gluster2
>>>>>>>>> Brick8: node08:/gluster2
>>>>>>>>>
>>>>>>>>> But I can't add new nodes to this volume
>>>>>>>>>
>>>>>>>>> [root at node07 ~]# gluster peer status
>>>>>>>>> Number of Peers: 5
>>>>>>>>>
>>>>>>>>> Hostname: node10
>>>>>>>>> Uuid: 212ce5a0-de51-4a98-9262-ae071c2d63a0
>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>
>>>>>>>>> Hostname: node08
>>>>>>>>> Uuid: dd114546-5b94-4a62-9301-260703bf5707
>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>
>>>>>>>>> Hostname: node09
>>>>>>>>> Uuid: f73fee83-8d47-4f07-bfac-b8a8592eff04
>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>
>>>>>>>>> Hostname: node06
>>>>>>>>> Uuid: 3142fb9a-0a6b-46ec-9262-ede95e8f798a
>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>>
>>>>>>>>> Hostname: node05
>>>>>>>>> Uuid: 13ffcf87-6e8d-4c6b-814a-cbc14d15d88b
>>>>>>>>> State: Peer in Cluster (Connected)
>>>>>>>>> [root at node07 ~]# gluster volume add-brick cluster node05:/gluster1
>>>>>>>>> node06:/gluster1 node05:/gluster2 node06:/gluster2 Operation failed
>>>>>>>>> on
>>>>>>>>> node05
>>>>>>>>>
>>>>>>>>> I try to detach nodes 5 and 6, restart glusterd do the probe and the
>>>>>>>>> add-brick but still nothing ...
>>>>>>>>>
>>>>>>>>> Did some one have any idea to fix this ?
>>>>>>>>>
>>>>>>>>> Thanks in advance,
>>>>>>>>> Laurent.
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> DISCLAIMER:
>>>>>>>>> This e-mail, and any attachments thereto, is intended only for use
>>>>>>>>> by
>>>>>>>>> the
>>>>>>>>> addressee(s) named herein and may contain legally privileged and/or
>>>>>>>>> confidential information. If you are not the intended recipient of
>>>>>>>>> this
>>>>>>>>> e-mail, you are hereby notified that any dissemination, distribution
>>>>>>>>> or
>>>>>>>>> copying of this e-mail, and any attachments thereto, is strictly
>>>>>>>>> prohibited.
>>>>>>>>> If you have received this in error, please immediately notify me and
>>>>>>>>> permanently delete the original and any copy of any e-mail and any
>>>>>>>>> printout
>>>>>>>>> thereof. E-mail transmission cannot be guaranteed to be secure or
>>>>>>>>> error-free. The sender therefore does not accept liability for any
>>>>>>>>> errors or
>>>>>>>>> omissions in the contents of this message which arise as a result of
>>>>>>>>> e-mail
>>>>>>>>> transmission.
>>>>>>>>> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group
>>>>>>>>> may,
>>>>>>>>> at
>>>>>>>>> its discretion, monitor and review the content of all e-mail
>>>>>>>>> communications.
>>>>>>>>> http://www.knight.com
>>>>>>>>> _______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list