[Gluster-users] Adding a 3.6.3 peer to a 3.6.2 trusted pool fails
Adrián Santos Marrero
asmarre at ull.edu.es
Thu May 14 08:08:36 UTC 2015
Hi,
Finally, this was a network problem. A switch was mangling jumbo frames.
Regards.
2015-05-13 9:07 GMT+01:00 Adrián Santos Marrero <asmarre at ull.edu.es>:
>
> Hi Kaushal,
>
> Volume info:
>
>
>> gluster02:~# gluster volume info all
>>
>> Volume Name: ovirt1
>> Type: Replicate
>> Volume ID: 05171231-a4a5-48ae-bcfe-63408bc0fc0c
>> Status: Started
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster01.stic.ull.es:/gluster/ovirt1_brick_01/brick
>> Brick2: gluster02.stic.ull.es:/gluster/ovirt1_brick_01/brick
>> Options Reconfigured:
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> cluster.server-quorum-type: none
>> network.remote-dio: enable
>> cluster.eager-lock: enable
>> performance.stat-prefetch: off
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>>
>> Volume Name: ovirt2
>> Type: Replicate
>> Volume ID: 0c0be558-9d98-4403-82d4-faafa2a3630a
>> Status: Started
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster01.stic.ull.es:/gluster/ovirt2_brick_01/brick
>> Brick2: gluster02.stic.ull.es:/gluster/ovirt2_brick_01/brick
>> Options Reconfigured:
>> storage.owner-gid: 36
>> storage.owner-uid: 36
>> cluster.server-quorum-type: none
>> network.remote-dio: enable
>> cluster.eager-lock: enable
>> performance.stat-prefetch: off
>> performance.io-cache: off
>> performance.read-ahead: off
>> performance.quick-read: off
>>
>
>
> And attached you can find the debug log for gluster03.
>
> Gluster01 is still re-healing so I am not sure if it is safe to restart
> glusterd in DEBUG mode to obtain logs....
>
> Regards.
>
>
> 2015-05-13 8:41 GMT+01:00 Kaushal M <kshlmster at gmail.com>:
>
>> Hey Adrian,
>> Can you list your volume infos? Most of the times the reason for peer
>> state being stuck is because of some volume information being incorrect and
>> the volume infos can help debug the problem. Could you also start glusterd
>> in debug mode (`glusterd -LDEBUG`) and give logs?
>>
>> ~kaushal
>> On May 13, 2015 9:13 AM, "Adrián Santos Marrero" <asmarre at ull.edu.es>
>> wrote:
>>
>>> Hi again,
>>>
>>> I upgraded to 3.6.3 all my gluster servers but peer probes still get
>>> stuck in "Accepted peer request".
>>>
>>> Any hint?
>>>
>>> Regards.
>>>
>>> 2015-05-11 13:55 GMT+01:00 Adrián Santos Marrero <asmarre at ull.edu.es>:
>>>
>>>> Hi, I've the same error with gluster03 (first installed with 3.6.2
>>>> version, but upgraded to 3.6.3 before any peer probe) and gluster04 (fresh
>>>> install with Gluster 3.6.3). I attach the whole content of
>>>> /var/log/glusterfs/ directory from gluster03.
>>>>
>>>> Regards.
>>>>
>>>> 2015-05-11 12:06 GMT+01:00 Gaurav Garg <ggarg at redhat.com>:
>>>>
>>>>> Hi Adrian,
>>>>>
>>>>> Did you try to upgrade/downgrade gluster03 before performing any
>>>>> operation. If you attach fresh node (3.6.3 version) to (3.6.2) glusterfs
>>>>> version then this problems is not coming. we need to know what operation
>>>>> you performed on gluster03 node before peer probe. if possible could you
>>>>> attach cmd_history.log and glusterd log.
>>>>>
>>>>> ~ Gaurav
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Adrián Santos Marrero" <asmarre at ull.edu.es>
>>>>> To: "Gluster-users" <gluster-users at gluster.org>
>>>>> Sent: Monday, May 11, 2015 12:39:33 PM
>>>>> Subject: [Gluster-users] Adding a 3.6.3 peer to a 3.6.2 trusted pool
>>>>> fails
>>>>>
>>>>> Hi,
>>>>>
>>>>> I've been trying to add a third node to my two-node trusted pool with
>>>>> gluster peer probe:
>>>>>
>>>>>
>>>>>
>>>>> gluster02:~# gluster peer probe gluster03.stic.ull.es
>>>>> peer probe: success.
>>>>>
>>>>> The problem is that this node remains in "Accepted peer request" state:
>>>>>
>>>>>
>>>>>
>>>>> Hostname: gluster03.stic.ull.es
>>>>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539
>>>>> State: Accepted peer request (Connected)
>>>>>
>>>>> If I restart glusterd in gluster02, the state changes to "Sent and
>>>>> Received peer request":
>>>>>
>>>>>
>>>>>
>>>>> Hostname: gluster03.stic.ull.es
>>>>> Uuid: 9ca23fb2-cd76-4d1c-8a5f-f93306119539
>>>>> State: Sent and Received peer request (Connected)
>>>>>
>>>>> And if I restart glusterd in gluster03 it doesn't start because is
>>>>> "Unable to find friend: gluster01.stic.ull.es ":
>>>>>
>>>>>
>>>>>
>>>>> [2015-05-11 06:59:39.035184] D
>>>>> [glusterd-peer-utils.c:121:glusterd_peerinfo_find_by_hostname]
>>>>> 0-management: Unable to find friend: gluster01.stic.ull.es
>>>>> [2015-05-11 06:59:39.037942] D [common-utils.c:2897:gf_is_local_addr]
>>>>> 0-management: 10.107.34.17
>>>>> [2015-05-11 06:59:39.039392] D [common-utils.c:2897:gf_is_local_addr]
>>>>> 0-management: 10.107.34.17
>>>>> [2015-05-11 06:59:39.040584] D [common-utils.c:2897:gf_is_local_addr]
>>>>> 0-management: 10.107.34.17
>>>>> [2015-05-11 06:59:39.041790] D [common-utils.c:2913:gf_is_local_addr]
>>>>> 0-management: gluster01.stic.ull.es is not local
>>>>>
>>>>> In fact, the only peer that gluster03 knows is gluster02:
>>>>>
>>>>>
>>>>>
>>>>> gluster03:~# ls -l /var/lib/glusterd/peers/
>>>>> total 4
>>>>> -rw-------. 1 root root 73 may 11 07:48
>>>>> 739d7c3f-99d5-4c8b-ac28-f85163e26322
>>>>> gluster03:~# cat
>>>>> /var/lib/glusterd/peers/739d7c3f-99d5-4c8b-ac28-f85163e26322
>>>>> uuid=739d7c3f-99d5-4c8b-ac28-f85163e26322
>>>>> state=5
>>>>> hostname1=10.107.34.18
>>>>>
>>>>> 10.107.34.18 == gluster02
>>>>>
>>>>>
>>>>>
>>>>> gluster03:~# gluster --version
>>>>> glusterfs 3.6.3 built on Apr 23 2015 16:11:45
>>>>> Repository revision: git:// git.gluster.com/glusterfs.git
>>>>> Copyright (c) 2006-2011 Gluster Inc. < http://www.gluster.com >
>>>>> GlusterFS comes with ABSOLUTELY NO WARRANTY.
>>>>> You may redistribute copies of GlusterFS under the terms of the GNU
>>>>> General Public License.
>>>>>
>>>>> All servers are running Centos 7, gluster01 and gluster02 have Gluster
>>>>> 3.6.2 version and gluster03 3.6.3.
>>>>>
>>>>> Am I forced to upgrade gluster01 and gluster02 to 3.6.3?
>>>>>
>>>>> Regards.
>>>>>
>>>>> --
>>>>> ----------------------------------------------------------------
>>>>> Adrián Santos Marrero
>>>>> Técnico de Sistemas - Área de Infraestructuras TIC
>>>>> Servicios de Tecnologías de la Información y Comunicación (STIC)
>>>>> Universidad de La Laguna (ULL)
>>>>> Teléfono/Phone: +34 922 845089
>>>>> ----------------------------------------------------------------
>>>>>
>>>>> Este mensaje puede contener información confidencial y/o privilegiada.
>>>>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo
>>>>> inmediatamente.
>>>>> Está estrictamente prohibido por la legislación vigente realizar
>>>>> cualquier copia, revelación o distribución del contenido de este
>>>>> mensaje
>>>>> sin autorización expresa.
>>>>>
>>>>> This e-mail may contain confidential and/or privileged information.
>>>>> If you are not the intended recipient or have received this e-mail in
>>>>> error you must destroy it.
>>>>> Any unauthorised copying, disclosure or distribution of the material in
>>>>> this e-mail is strictly forbidden by current legislation.
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> ----------------------------------------------------------------
>>>> Adrián Santos Marrero
>>>> Técnico de Sistemas - Área de Infraestructuras TIC
>>>> Servicios de Tecnologías de la Información y Comunicación (STIC)
>>>> Universidad de La Laguna (ULL)
>>>> Teléfono/Phone: +34 922 845089
>>>> ----------------------------------------------------------------
>>>>
>>>> Este mensaje puede contener información confidencial y/o privilegiada.
>>>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo
>>>> inmediatamente.
>>>> Está estrictamente prohibido por la legislación vigente realizar
>>>> cualquier copia, revelación o distribución del contenido de este mensaje
>>>> sin autorización expresa.
>>>>
>>>> This e-mail may contain confidential and/or privileged information.
>>>> If you are not the intended recipient or have received this e-mail in
>>>> error you must destroy it.
>>>> Any unauthorised copying, disclosure or distribution of the material in
>>>> this e-mail is strictly forbidden by current legislation.
>>>>
>>>
>>>
>>>
>>> --
>>> ----------------------------------------------------------------
>>> Adrián Santos Marrero
>>> Técnico de Sistemas - Área de Infraestructuras TIC
>>> Servicios de Tecnologías de la Información y Comunicación (STIC)
>>> Universidad de La Laguna (ULL)
>>> Teléfono/Phone: +34 922 845089
>>> ----------------------------------------------------------------
>>>
>>> Este mensaje puede contener información confidencial y/o privilegiada.
>>> Si usted no es el destinatario o lo ha recibido por error debe borrarlo
>>> inmediatamente.
>>> Está estrictamente prohibido por la legislación vigente realizar
>>> cualquier copia, revelación o distribución del contenido de este mensaje
>>> sin autorización expresa.
>>>
>>> This e-mail may contain confidential and/or privileged information.
>>> If you are not the intended recipient or have received this e-mail in
>>> error you must destroy it.
>>> Any unauthorised copying, disclosure or distribution of the material in
>>> this e-mail is strictly forbidden by current legislation.
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>
>
> --
> ----------------------------------------------------------------
> Adrián Santos Marrero
> Técnico de Sistemas - Área de Infraestructuras TIC
> Servicios de Tecnologías de la Información y Comunicación (STIC)
> Universidad de La Laguna (ULL)
> Teléfono/Phone: +34 922 845089
> ----------------------------------------------------------------
>
> Este mensaje puede contener información confidencial y/o privilegiada.
> Si usted no es el destinatario o lo ha recibido por error debe borrarlo
> inmediatamente.
> Está estrictamente prohibido por la legislación vigente realizar
> cualquier copia, revelación o distribución del contenido de este mensaje
> sin autorización expresa.
>
> This e-mail may contain confidential and/or privileged information.
> If you are not the intended recipient or have received this e-mail in
> error you must destroy it.
> Any unauthorised copying, disclosure or distribution of the material in
> this e-mail is strictly forbidden by current legislation.
>
--
----------------------------------------------------------------
Adrián Santos Marrero
Técnico de Sistemas - Área de Infraestructuras TIC
Servicios de Tecnologías de la Información y Comunicación (STIC)
Universidad de La Laguna (ULL)
Teléfono/Phone: +34 922 845089
----------------------------------------------------------------
Este mensaje puede contener información confidencial y/o privilegiada.
Si usted no es el destinatario o lo ha recibido por error debe borrarlo
inmediatamente.
Está estrictamente prohibido por la legislación vigente realizar
cualquier copia, revelación o distribución del contenido de este mensaje
sin autorización expresa.
This e-mail may contain confidential and/or privileged information.
If you are not the intended recipient or have received this e-mail in
error you must destroy it.
Any unauthorised copying, disclosure or distribution of the material in
this e-mail is strictly forbidden by current legislation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150514/1f9d628a/attachment.html>
More information about the Gluster-users
mailing list