[Gluster-users] Issue in Adding/Removing the gluster node

Fri Feb 19 11:51:09 UTC 2016

Hi Gaurav,

Both are the board connect through the backplane using ethernet.

Even this inconsistency also occurs when I am trying to bringing back the
node in slot. Means some time add-brick executes without failure but some
time following error occurs.

volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick
force : FAILED : Another transaction is in progress for c_glusterfs. Please
try again after sometime.

You can also see the attached logs for add-brick failure scenario.

Please let me know if you need more logs.

Regards,
Abhishek

On Fri, Feb 19, 2016 at 5:03 PM, Gaurav Garg <ggarg at redhat.com> wrote:

> Hi Abhishek,
>
> How are you connecting two board, and how are you removing it manually
> that need to know because if you are removing your 2nd board from the
> cluster (abrupt shutdown) then you can't perform remove brick operation in
> 2nd node from first node and its happening successfully in your case. could
> you ensure your network connection once again while removing and bringing
> back your node again.
>
> Thanks,
> Gaurav
>
> ------------------------------
> *From: *"ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> *To: *"Gaurav Garg" <ggarg at redhat.com>
> *Cc: *gluster-users at gluster.org
> *Sent: *Friday, February 19, 2016 3:36:21 PM
>
> *Subject: *Re: [Gluster-users] Issue in Adding/Removing the gluster node
>
> Hi Gaurav,
>
> Thanks for reply
>
> 1. Here, I removed the board manually here but this time it works fine
>
> [2016-02-18 10:03:40.601472]  : volume remove-brick c_glusterfs replica 1
> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
> [2016-02-18 10:03:40.885973]  : peer detach 10.32.1.144 : SUCCESS
>
> Yes this time board is reachable but how? don't know because board is
> detached.
>
> 2. Here, I attached the board this time its works fine in add-bricks
>
> 2016-02-18 10:03:42.065038]  : peer probe 10.32.1.144 : SUCCESS
> [2016-02-18 10:03:44.563546]  : volume add-brick c_glusterfs replica 2
> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
>
> 3.Here, again I removed the board this time failed occur
>
> [2016-02-18 10:37:02.816089]  : volume remove-brick c_glusterfs replica 1
> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick
> 10.32.1.144:/opt
> /lvmdir/c2/brick for volume c_glusterfs
>
> but here board is not reachable.
>
> why this inconsistency is there while doing the same step multiple time.
>
> Hope you are getting my point.
>
> Regards,
> Abhishek
>
> On Fri, Feb 19, 2016 at 3:25 PM, Gaurav Garg <ggarg at redhat.com> wrote:
>
>> Abhishek,
>>
>> when sometime its working fine means 2nd board network connection is
>> reachable to first node. you can conform this by executing same #gluster
>> peer status command.
>>
>> Thanks,
>> Gaurav
>>
>> ----- Original Message -----
>> From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
>> To: "Gaurav Garg" <ggarg at redhat.com>
>> Cc: gluster-users at gluster.org
>> Sent: Friday, February 19, 2016 3:12:22 PM
>> Subject: Re: [Gluster-users] Issue in Adding/Removing the gluster node
>>
>> Hi Gaurav,
>>
>> Yes, you are right actually I am force fully detaching the node from the
>> slave and when we removed the board it disconnected from the another
>> board.
>>
>> but my question is I am doing this process multiple time some time it
>> works
>> fine but some time it gave these errors.
>>
>>
>> you can see the following logs from cmd_history.log file
>>
>> [2016-02-18 10:03:34.497996]  : volume set c_glusterfs nfs.disable on :
>> SUCCESS
>> [2016-02-18 10:03:34.915036]  : volume start c_glusterfs force : SUCCESS
>> [2016-02-18 10:03:40.250326]  : volume status : SUCCESS
>> [2016-02-18 10:03:40.273275]  : volume status : SUCCESS
>> [2016-02-18 10:03:40.601472]  : volume remove-brick c_glusterfs replica 1
>> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
>> [2016-02-18 10:03:40.885973]  : peer detach 10.32.1.144 : SUCCESS
>> [2016-02-18 10:03:42.065038]  : peer probe 10.32.1.144 : SUCCESS
>> [2016-02-18 10:03:44.563546]  : volume add-brick c_glusterfs replica 2
>> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
>> [2016-02-18 10:30:53.297415]  : volume status : SUCCESS
>> [2016-02-18 10:30:53.313096]  : volume status : SUCCESS
>> [2016-02-18 10:37:02.748714]  : volume status : SUCCESS
>> [2016-02-18 10:37:02.762091]  : volume status : SUCCESS
>> [2016-02-18 10:37:02.816089]  : volume remove-brick c_glusterfs replica 1
>> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick
>> 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs
>>
>>
>> On Fri, Feb 19, 2016 at 3:05 PM, Gaurav Garg <ggarg at redhat.com> wrote:
>>
>> > Hi Abhishek,
>> >
>> > Seems your peer 10.32.1.144 have disconnected while doing remove brick.
>> > see the below logs in glusterd:
>> >
>> > [2016-02-18 10:37:02.816009] E [MSGID: 106256]
>> > [glusterd-brick-ops.c:1047:__glusterd_handle_remove_brick] 0-management:
>> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs
>> > [Invalid argument]
>> > [2016-02-18 10:37:02.816061] E [MSGID: 106265]
>> > [glusterd-brick-ops.c:1088:__glusterd_handle_remove_brick] 0-management:
>> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs
>> > The message "I [MSGID: 106004]
>> > [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management: Peer
>> > <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state <Peer
>> in
>> > Cluster>, has disconnected from glusterd." repeated 25 times between
>> > [2016-02-18 10:35:43.131945] and [2016-02-18 10:36:58.160458]
>> >
>> >
>> >
>> > If you are facing the same issue now, could you paste your # gluster
>> peer
>> > status     command output here.
>> >
>> > Thanks,
>> > ~Gaurav
>> >
>> > ----- Original Message -----
>> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
>> > To: gluster-users at gluster.org
>> > Sent: Friday, February 19, 2016 2:46:35 PM
>> > Subject: [Gluster-users] Issue in Adding/Removing the gluster node
>> >
>> > Hi,
>> >
>> >
>> > I am working on two board setup connecting to each other. Gluster
>> version
>> > 3.7.6 is running and added two bricks in replica 2 mode but when I
>> manually
>> > removed (detach) the one board from the setup I am getting the following
>> > error.
>> >
>> > volume remove-brick c_glusterfs replica 1 10.32.1.144:
>> /opt/lvmdir/c2/brick
>> > force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for
>> > volume c_glusterfs
>> >
>> > Please find the logs file as an attachment.
>> >
>> >
>> > Regards,
>> > Abhishek
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > http://www.gluster.org/mailman/listinfo/gluster-users
>> >
>>
>>
>>
>> --
>>
>>
>>
>>
>> Regards
>> Abhishek Paliwal
>>
>
>
>
> --
>
>
>
>
> Regards
> Abhishek Paliwal
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cmd_history.log
Type: text/x-log
Size: 21439 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cli.log
Type: text/x-log
Size: 71308 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: etc-glusterfs-glusterd.vol.log
Type: text/x-log
Size: 111061 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160219/9d0565c7/attachment-0005.bin>