[Gluster-users] op-version for reset-brick (Was: Re: [ovirt-users] Upgrading HC from 4.0 to 4.1)

[Atin Mukherjee]

On Fri, Jul 7, 2017 at 2:23 PM, Gianluca Cecchi <gianluca.cecchi at gmail.com>
wrote:

> On Thu, Jul 6, 2017 at 3:22 PM, Gianluca Cecchi <gianluca.cecchi at gmail.com
> > wrote:
>
>> On Thu, Jul 6, 2017 at 2:16 PM, Atin Mukherjee <amukherj at redhat.com>
>> wrote:
>>
>>>
>>>
>>> On Thu, Jul 6, 2017 at 5:26 PM, Gianluca Cecchi <
>>> gianluca.cecchi at gmail.com> wrote:
>>>
>>>> On Thu, Jul 6, 2017 at 8:38 AM, Gianluca Cecchi <
>>>> gianluca.cecchi at gmail.com> wrote:
>>>>
>>>>>
>>>>> Eventually I can destroy and recreate this "export" volume again with
>>>>> the old names (ovirt0N.localdomain.local) if you give me the sequence of
>>>>> commands, then enable debug and retry the reset-brick command
>>>>>
>>>>> Gianluca
>>>>>
>>>>
>>>>
>>>> So it seems I was able to destroy and re-create.
>>>> Now I see that the volume creation uses by default the new ip, so I
>>>> reverted the hostnames roles in the commands after putting glusterd in
>>>> debug mode on the host where I execute the reset-brick command (do I have
>>>> to set debug for the the nodes too?)
>>>>
>>>
>>> You have to set the log level to debug for glusterd instance where the
>>> commit fails and share the glusterd log of that particular node.
>>>
>>>
>>
>> Ok, done.
>>
>> Command executed on ovirt01 with timestamp "2017-07-06 13:04:12" in
>> glusterd log files
>>
>> [root at ovirt01 export]# gluster volume reset-brick export
>> gl01.localdomain.local:/gluster/brick3/export start
>> volume reset-brick: success: reset-brick start operation successful
>>
>> [root at ovirt01 export]# gluster volume reset-brick export
>> gl01.localdomain.local:/gluster/brick3/export
>> ovirt01.localdomain.local:/gluster/brick3/export commit force
>> volume reset-brick: failed: Commit failed on ovirt02.localdomain.local.
>> Please check log file for details.
>> Commit failed on ovirt03.localdomain.local. Please check log file for
>> details.
>> [root at ovirt01 export]#
>>
>> See glusterd log files for the 3 nodes in debug mode here:
>> ovirt01: https://drive.google.com/file/d/0BwoPbcrMv8mvY1RTTG
>> p3RUhScm8/view?usp=sharing
>> ovirt02: https://drive.google.com/file/d/0BwoPbcrMv8mvSVpJUH
>> NhMzhMSU0/view?usp=sharing
>> ovirt03: https://drive.google.com/file/d/0BwoPbcrMv8mvT2xiWE
>> dQVmJNb0U/view?usp=sharing
>>
>> HIH debugging
>> Gianluca
>>
>>
> Hi Atin,
> did you have time to see the logs?
> Comparing debug enabled messages with previous ones, I see these added
> lines on nodes where commit failed after running the commands
>
> gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export
> start
> gluster volume reset-brick export gl01.localdomain.local:/gluster/brick3/export
> ovirt01.localdomain.local:/gluster/brick3/export commit force
>
>
> [2017-07-06 13:04:30.221872] D [MSGID: 0] [glusterd-peer-utils.c:674:gd_peerinfo_find_from_hostname]
> 0-management: Friend ovirt01.localdomain.local found.. state: 3
> [2017-07-06 13:04:30.221882] D [MSGID: 0] [glusterd-peer-utils.c:167:glusterd_hostname_to_uuid]
> 0-management: returning 0
> [2017-07-06 13:04:30.221888] D [MSGID: 0] [glusterd-utils.c:1039:glusterd_resolve_brick]
> 0-management: Returning 0
> [2017-07-06 13:04:30.221908] D [MSGID: 0] [glusterd-utils.c:998:glusterd_brickinfo_new]
> 0-management: Returning 0
> [2017-07-06 13:04:30.221915] D [MSGID: 0] [glusterd-utils.c:1195:glusterd_brickinfo_new_from_brick]
> 0-management: Returning 0
> [2017-07-06 13:04:30.222187] D [MSGID: 0] [glusterd-peer-utils.c:167:glusterd_hostname_to_uuid]
> 0-management: returning 0
> [2017-07-06 13:04:30.222201] D [MSGID: 0] [glusterd-utils.c:1486:glusterd_volume_brickinfo_get]
> 0-management: Returning -1
>

The above log entry is the reason of the failure. GlusterD is unable to
find the old brick (src_brick) from its volinfo structure. FWIW, would you
be able to share the content of 'gluster get-state' output & gluster volume
info output after running reset-brick start? I'd need to check why glusterd
is unable to find out the old brick's details from its volinfo post
reset-brick start.



> [2017-07-06 13:04:30.222207] D [MSGID: 0] [store.c:459:gf_store_handle_destroy]
> 0-: Returning 0
> [2017-07-06 13:04:30.222242] D [MSGID: 0] [glusterd-utils.c:1512:gluster
> d_volume_brickinfo_get_by_brick] 0-glusterd: Returning -1
> [2017-07-06 13:04:30.222250] D [MSGID: 0] [glusterd-replace-brick.c:416:
> glusterd_op_perform_replace_brick] 0-glusterd: Returning -1
> [2017-07-06 13:04:30.222257] C [MSGID: 106074]
> [glusterd-reset-brick.c:372:glusterd_op_reset_brick] 0-management: Unable
> to add dst-brick: ovirt01.localdomain.local:/gluster/brick3/export to
> volume: export
>
>
> Does it share up more light?
>
> Thanks,
> Gianluca
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170710/18f5e51e/attachment.html>