[Gluster-users] Adding a 3.6.3 peer to a 3.6.2 trusted pool fails

Wed May 13 08:07:20 UTC 2015

Hi Kaushal,

Volume info:

> gluster02:~# gluster volume info all
>
> Volume Name: ovirt1
> Type: Replicate
> Volume ID: 05171231-a4a5-48ae-bcfe-63408bc0fc0c
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: gluster01.stic.ull.es:/gluster/ovirt1_brick_01/brick
> Brick2: gluster02.stic.ull.es:/gluster/ovirt1_brick_01/brick
> Options Reconfigured:
> storage.owner-gid: 36
> storage.owner-uid: 36
> cluster.server-quorum-type: none
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
>
> Volume Name: ovirt2
> Type: Replicate
> Volume ID: 0c0be558-9d98-4403-82d4-faafa2a3630a
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: gluster01.stic.ull.es:/gluster/ovirt2_brick_01/brick
> Brick2: gluster02.stic.ull.es:/gluster/ovirt2_brick_01/brick
> Options Reconfigured:
> storage.owner-gid: 36
> storage.owner-uid: 36
> cluster.server-quorum-type: none
> network.remote-dio: enable
> cluster.eager-lock: enable
> performance.stat-prefetch: off
> performance.io-cache: off
> performance.read-ahead: off
> performance.quick-read: off
>

And attached you can find the debug log for gluster03.

Gluster01 is still re-healing so I am not sure if it is safe to restart
glusterd in DEBUG mode to obtain logs....

Regards.

2015-05-13 8:41 GMT+01:00 Kaushal M <kshlmster at gmail.com>:

> Hey Adrian,
> Can you list your volume infos? Most of the times the reason for peer
> state being stuck is because of some volume information being incorrect and
> the volume infos can help debug the problem. Could you also start glusterd
> in debug mode (`glusterd -LDEBUG`) and give logs?
>
> ~kaushal
>  On May 13, 2015 9:13 AM, "Adrián Santos Marrero" <asmarre at ull.edu.es>
> wrote:
>
>> Hi again,
>>
>> I upgraded to 3.6.3 all my gluster servers but peer probes still get
>> stuck in "Accepted peer request".
>>
>> Any hint?
>>
>> Regards.
>>
>> 2015-05-11 13:55 GMT+01:00 Adrián Santos Marrero <asmarre at ull.edu.es>:
>>
>>> Hi, I've the same error with gluster03 (first installed with 3.6.2
>>> version, but upgraded to 3.6.3 before any peer probe) and gluster04 (fresh
>>> install with Gluster 3.6.3). I attach the whole content of
>>> /var/log/glusterfs/ directory from gluster03.
>>>
>>> Regards.
>>>
>>> 2015-05-11 12:06 GMT+01:00 Gaurav Garg <ggarg at redhat.com>:
>>>
>>>> Hi Adrian,
>>>>
>>>> Did you try to upgrade/downgrade gluster03 before performing any
>>>> operation. If you attach fresh node (3.6.3 version) to (3.6.2) glusterfs
>>>> version then this problems is not coming. we need to know what operation
>>>> you performed on gluster03 node before peer probe. if possible could you
>>>> attach cmd_history.log and glusterd log.
>>>>
>>>> ~ Gaurav
>>>>
>>>> ----- Original Message -----
>>>> From: "Adrián Santos Marrero" <asmarre at ull.edu.es>
>>>> To: "Gluster-users" <gluster-users at gluster.org>
>>>> Sent: Monday, May 11, 2015 12:39:33 PM
>>>> Subject: [Gluster-users] Adding a 3.6.3 peer to a 3.6.2 trusted pool
>>>> fails
>>>>
>>>> Hi,
>>>>
>>>> I've been trying to add a third node to my two-node trusted pool with
>>>> gluster peer probe:
>>>>
>>>>
>>>>
>>>> gluster02:~# gluster peer probe gluster03.stic.ull.es
>>>> peer probe: success.
>>>>
>>>> The problem is that this node remains in "Accepted peer request" state:
>>>>
>>>>
>>>>
>>>> Hostname: gluster03.stic.ull.es
>>>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539
>>>> State: Accepted peer request (Connected)
>>>>
>>>> If I restart glusterd in gluster02, the state changes to "Sent and
>>>> Received peer request":
>>>>
>>>>
>>>>
>>>> Hostname: gluster03.stic.ull.es
>>>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539
>>>> State: Sent and Received peer request (Connected)
>>>>
>>>> And if I restart glusterd in gluster03 it doesn't start because is
>>>> "Unable to find friend: gluster01.stic.ull.es ":
>>>>
>>>>
>>>>
>>>> [2015-05-11 06:59:39.035184] D
>>>> [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname]
>>>> 0-management: Unable to find friend: gluster01.stic.ull.es
>>>> [2015-05-11 06:59:39.037942] D [common-utils.c:2897:gf_is_local_addr]
>>>> 0-management: 10.107.34.17
>>>> [2015-05-11 06:59:39.039392] D [common-utils.c:2897:gf_is_local_addr]
>>>> 0-management: 10.107.34.17
>>>> [2015-05-11 06:59:39.040584] D [common-utils.c:2897:gf_is_local_addr]
>>>> 0-management: 10.107.34.17
>>>> [2015-05-11 06:59:39.041790] D [common-utils.c:2913:gf_is_local_addr]
>>>> 0-management: gluster01.stic.ull.es is not local
>>>>
>>>> In fact, the only peer that gluster03 knows is gluster02:
>>>>
>>>>
>>>>
>>>> gluster03:~# ls -l /var/lib/glusterd/peers/
>>>> total 4
>>>> -rw-------. 1 root root 73 may 11 07:48
>>>> 739d7c3f-99d5-4c8b-ac28-f85163e26322
>>>> gluster03:~# cat
>>>> /var/lib/glusterd/peers/739d7c3f-99d5-4c8b-ac28-f85163e26322
>>>> uuid=739d7c3f-99d5-4c8b-ac28-f85163e26322
>>>> state=5
>>>> hostname1=10.107.34.18
>>>>
>>>> 10.107.34.18 == gluster02
>>>>
>>>>
>>>>
>>>> gluster03:~# gluster --version
>>>> glusterfs 3.6.3 built on Apr 23 2015 16:11:45
>>>> Repository revision: git:// git.gluster.com/glusterfs.git
>>>> Copyright (c) 2006-2011 Gluster Inc. < http://www.gluster.com >
>>>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>>>> You may redistribute copies of GlusterFS under the terms of the GNU
>>>> General Public License.
>>>>
>>>> All servers are running Centos 7, gluster01 and gluster02 have Gluster
>>>> 3.6.2 version and gluster03 3.6.3.
>>>>
>>>> Am I forced to upgrade gluster01 and gluster02 to 3.6.3?
>>>>
>>>> Regards.
>>>>
>>>> --
>>>> ----------------------------------------------------------------
>>>> Adrián Santos Marrero
>>>> Técnico de Sistemas - Área de Infraestructuras TIC
>>>> Servicios de Tecnologías de la Información y Comunicación (STIC)
>>>> Universidad de La Laguna (ULL)
>>>> Teléfono/Phone: +34 922 845089
>>>> ----------------------------------------------------------------
>>>>
>>>> Este mensaje puede contener información confidencial y/o privilegiada.
>>>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo
>>>> inmediatamente.
>>>> Está estrictamente prohibido por la legislación vigente realizar
>>>> cualquier copia, revelación o distribución del contenido de este mensaje
>>>> sin autorización expresa.
>>>>
>>>> This e-mail may contain confidential and/or privileged information.
>>>> If you are not the intended recipient or have received this e-mail in
>>>> error you must destroy it.
>>>> Any unauthorised copying, disclosure or distribution of the material in
>>>> this e-mail is strictly forbidden by current legislation.
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
>>>
>>>
>>> --
>>> ----------------------------------------------------------------
>>>  Adrián Santos Marrero
>>>  Técnico de Sistemas - Área de Infraestructuras TIC
>>>  Servicios de Tecnologías de la Información y Comunicación (STIC)
>>>  Universidad de La Laguna (ULL)
>>>  Teléfono/Phone: +34 922 845089
>>> ----------------------------------------------------------------
>>>
>>> Este mensaje puede contener información confidencial y/o privilegiada.
>>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo
>>> inmediatamente.
>>> Está estrictamente prohibido por la legislación vigente realizar
>>> cualquier copia, revelación o distribución del contenido de este mensaje
>>> sin autorización expresa.
>>>
>>> This e-mail may contain confidential and/or privileged information.
>>> If you are not the intended recipient or have received this e-mail in
>>> error you must destroy it.
>>> Any unauthorised copying, disclosure or distribution of the material in
>>> this e-mail is strictly forbidden by current legislation.
>>>
>>
>>
>>
>> --
>> ----------------------------------------------------------------
>>  Adrián Santos Marrero
>>  Técnico de Sistemas - Área de Infraestructuras TIC
>>  Servicios de Tecnologías de la Información y Comunicación (STIC)
>>  Universidad de La Laguna (ULL)
>>  Teléfono/Phone: +34 922 845089
>> ----------------------------------------------------------------
>>
>> Este mensaje puede contener información confidencial y/o privilegiada.
>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo
>> inmediatamente.
>> Está estrictamente prohibido por la legislación vigente realizar
>> cualquier copia, revelación o distribución del contenido de este mensaje
>> sin autorización expresa.
>>
>> This e-mail may contain confidential and/or privileged information.
>> If you are not the intended recipient or have received this e-mail in
>> error you must destroy it.
>> Any unauthorised copying, disclosure or distribution of the material in
>> this e-mail is strictly forbidden by current legislation.
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>
>

-- 
----------------------------------------------------------------
 Adrián Santos Marrero
 Técnico de Sistemas - Área de Infraestructuras TIC
 Servicios de Tecnologías de la Información y Comunicación (STIC)
 Universidad de La Laguna (ULL)
 Teléfono/Phone: +34 922 845089
----------------------------------------------------------------

Este mensaje puede contener información confidencial y/o privilegiada.
Si usted no es el destinatario o lo ha recibido por error debe borrarlo
inmediatamente.
Está estrictamente prohibido por la legislación vigente realizar
cualquier copia, revelación o distribución del contenido de este mensaje
sin autorización expresa.

This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient or have received this e-mail in
error you must destroy it.
Any unauthorised copying, disclosure or distribution of the material in
this e-mail is strictly forbidden by current legislation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150513/54ec683f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: etc-glusterfs-glusterd.vol.log
Type: text/x-log
Size: 309033 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150513/54ec683f/attachment-0001.bin>