[Gluster-users] Issue in Adding/Removing the gluster node

Mon Feb 22 04:48:21 UTC 2016

Hi Abhishek,

>>  I'll provide the required log to you.

sure

on both node. do "pkill glusterd" and then start glusterd services.

Thanks,

~Gaurav

----- Original Message -----
From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
To: "Gaurav Garg" <ggarg at redhat.com>
Cc: gluster-users at gluster.org
Sent: Monday, February 22, 2016 10:11:48 AM
Subject: Re: [Gluster-users] Issue in Adding/Removing the gluster node

Hi Gaurav,

Thanks for your prompt reply.

I'll provide the required log to you.

As a workaround you suggested that restart the glusterd service. Could you
please tell me the point where I can do this?

Regards,
Abhishek

On Fri, Feb 19, 2016 at 6:11 PM, Gaurav Garg <ggarg at redhat.com> wrote:

> Hi Abhishek,
>
> Peer status output looks interesting where it have stale entry,
> technically it should not happen. Here few thing need to ask
>
> Did you perform any manual operation with GlusterFS configuration file
> which resides in /var/lib/glusterd/* folder.
>
> Can you provide output of "ls /var/lib/glusterd/peers"  from both of your
> nodes.
>
> Could you provide output of #gluster peer status command when 2nd node is
> down
>
> Can you provide output of #gluster volume info command
>
> Can you provide full logs details of cmd_history.log and
> etc-glusterfs-glusterd.vol.log from both the nodes.
>
>
> You can restart your glusterd as of now as a workaround but we need to
> analysis this issue further.
>
> Thanks,
> Gaurav
>
> ----- Original Message -----
> From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> To: "Gaurav Garg" <ggarg at redhat.com>
> Cc: gluster-users at gluster.org
> Sent: Friday, February 19, 2016 5:27:21 PM
> Subject: Re: [Gluster-users] Issue in Adding/Removing the gluster node
>
> Hi Gaurav,
>
> After the failure of add-brick following is outcome "gluster peer status"
> command
>
> Number of Peers: 2
>
> Hostname: 10.32.1.144
> Uuid: bbe2a458-ad3d-406d-b233-b6027c12174e
> State: Peer in Cluster (Connected)
>
> Hostname: 10.32.1.144
> Uuid: bbe2a458-ad3d-406d-b233-b6027c12174e
> State: Peer in Cluster (Connected)
>
> Regards,
> Abhishek
>
> On Fri, Feb 19, 2016 at 5:21 PM, ABHISHEK PALIWAL <abhishpaliwal at gmail.com
> >
> wrote:
>
> > Hi Gaurav,
> >
> > Both are the board connect through the backplane using ethernet.
> >
> > Even this inconsistency also occurs when I am trying to bringing back the
> > node in slot. Means some time add-brick executes without failure but some
> > time following error occurs.
> >
> > volume add-brick c_glusterfs replica 2 10.32.1.144:/opt/lvmdir/c2/brick
> > force : FAILED : Another transaction is in progress for c_glusterfs.
> Please
> > try again after sometime.
> >
> >
> > You can also see the attached logs for add-brick failure scenario.
> >
> > Please let me know if you need more logs.
> >
> > Regards,
> > Abhishek
> >
> >
> > On Fri, Feb 19, 2016 at 5:03 PM, Gaurav Garg <ggarg at redhat.com> wrote:
> >
> >> Hi Abhishek,
> >>
> >> How are you connecting two board, and how are you removing it manually
> >> that need to know because if you are removing your 2nd board from the
> >> cluster (abrupt shutdown) then you can't perform remove brick operation
> in
> >> 2nd node from first node and its happening successfully in your case.
> could
> >> you ensure your network connection once again while removing and
> bringing
> >> back your node again.
> >>
> >> Thanks,
> >> Gaurav
> >>
> >> ------------------------------
> >> *From: *"ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> >> *To: *"Gaurav Garg" <ggarg at redhat.com>
> >> *Cc: *gluster-users at gluster.org
> >> *Sent: *Friday, February 19, 2016 3:36:21 PM
> >>
> >> *Subject: *Re: [Gluster-users] Issue in Adding/Removing the gluster node
> >>
> >> Hi Gaurav,
> >>
> >> Thanks for reply
> >>
> >> 1. Here, I removed the board manually here but this time it works fine
> >>
> >> [2016-02-18 10:03:40.601472]  : volume remove-brick c_glusterfs replica
> 1
> >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
> >> [2016-02-18 10:03:40.885973]  : peer detach 10.32.1.144 : SUCCESS
> >>
> >> Yes this time board is reachable but how? don't know because board is
> >> detached.
> >>
> >> 2. Here, I attached the board this time its works fine in add-bricks
> >>
> >> 2016-02-18 10:03:42.065038]  : peer probe 10.32.1.144 : SUCCESS
> >> [2016-02-18 10:03:44.563546]  : volume add-brick c_glusterfs replica 2
> >> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
> >>
> >> 3.Here, again I removed the board this time failed occur
> >>
> >> [2016-02-18 10:37:02.816089]  : volume remove-brick c_glusterfs replica
> 1
> >> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick
> >> 10.32.1.144:/opt
> >> /lvmdir/c2/brick for volume c_glusterfs
> >>
> >> but here board is not reachable.
> >>
> >> why this inconsistency is there while doing the same step multiple time.
> >>
> >> Hope you are getting my point.
> >>
> >> Regards,
> >> Abhishek
> >>
> >> On Fri, Feb 19, 2016 at 3:25 PM, Gaurav Garg <ggarg at redhat.com> wrote:
> >>
> >>> Abhishek,
> >>>
> >>> when sometime its working fine means 2nd board network connection is
> >>> reachable to first node. you can conform this by executing same
> #gluster
> >>> peer status command.
> >>>
> >>> Thanks,
> >>> Gaurav
> >>>
> >>> ----- Original Message -----
> >>> From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> >>> To: "Gaurav Garg" <ggarg at redhat.com>
> >>> Cc: gluster-users at gluster.org
> >>> Sent: Friday, February 19, 2016 3:12:22 PM
> >>> Subject: Re: [Gluster-users] Issue in Adding/Removing the gluster node
> >>>
> >>> Hi Gaurav,
> >>>
> >>> Yes, you are right actually I am force fully detaching the node from
> the
> >>> slave and when we removed the board it disconnected from the another
> >>> board.
> >>>
> >>> but my question is I am doing this process multiple time some time it
> >>> works
> >>> fine but some time it gave these errors.
> >>>
> >>>
> >>> you can see the following logs from cmd_history.log file
> >>>
> >>> [2016-02-18 10:03:34.497996]  : volume set c_glusterfs nfs.disable on :
> >>> SUCCESS
> >>> [2016-02-18 10:03:34.915036]  : volume start c_glusterfs force :
> SUCCESS
> >>> [2016-02-18 10:03:40.250326]  : volume status : SUCCESS
> >>> [2016-02-18 10:03:40.273275]  : volume status : SUCCESS
> >>> [2016-02-18 10:03:40.601472]  : volume remove-brick c_glusterfs
> replica 1
> >>> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
> >>> [2016-02-18 10:03:40.885973]  : peer detach 10.32.1.144 : SUCCESS
> >>> [2016-02-18 10:03:42.065038]  : peer probe 10.32.1.144 : SUCCESS
> >>> [2016-02-18 10:03:44.563546]  : volume add-brick c_glusterfs replica 2
> >>> 10.32.1.144:/opt/lvmdir/c2/brick force : SUCCESS
> >>> [2016-02-18 10:30:53.297415]  : volume status : SUCCESS
> >>> [2016-02-18 10:30:53.313096]  : volume status : SUCCESS
> >>> [2016-02-18 10:37:02.748714]  : volume status : SUCCESS
> >>> [2016-02-18 10:37:02.762091]  : volume status : SUCCESS
> >>> [2016-02-18 10:37:02.816089]  : volume remove-brick c_glusterfs
> replica 1
> >>> 10.32.1.144:/opt/lvmdir/c2/brick force : FAILED : Incorrect brick
> >>> 10.32.1.144:/opt/lvmdir/c2/brick for volume c_glusterfs
> >>>
> >>>
> >>> On Fri, Feb 19, 2016 at 3:05 PM, Gaurav Garg <ggarg at redhat.com> wrote:
> >>>
> >>> > Hi Abhishek,
> >>> >
> >>> > Seems your peer 10.32.1.144 have disconnected while doing remove
> brick.
> >>> > see the below logs in glusterd:
> >>> >
> >>> > [2016-02-18 10:37:02.816009] E [MSGID: 106256]
> >>> > [glusterd-brick-ops.c:1047:__glusterd_handle_remove_brick]
> >>> 0-management:
> >>> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume
> >>> c_glusterfs
> >>> > [Invalid argument]
> >>> > [2016-02-18 10:37:02.816061] E [MSGID: 106265]
> >>> > [glusterd-brick-ops.c:1088:__glusterd_handle_remove_brick]
> >>> 0-management:
> >>> > Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick for volume
> >>> c_glusterfs
> >>> > The message "I [MSGID: 106004]
> >>> > [glusterd-handler.c:5065:__glusterd_peer_rpc_notify] 0-management:
> Peer
> >>> > <10.32.1.144> (<6adf57dc-c619-4e56-ae40-90e6aef75fe9>), in state
> <Peer
> >>> in
> >>> > Cluster>, has disconnected from glusterd." repeated 25 times between
> >>> > [2016-02-18 10:35:43.131945] and [2016-02-18 10:36:58.160458]
> >>> >
> >>> >
> >>> >
> >>> > If you are facing the same issue now, could you paste your # gluster
> >>> peer
> >>> > status     command output here.
> >>> >
> >>> > Thanks,
> >>> > ~Gaurav
> >>> >
> >>> > ----- Original Message -----
> >>> > From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com>
> >>> > To: gluster-users at gluster.org
> >>> > Sent: Friday, February 19, 2016 2:46:35 PM
> >>> > Subject: [Gluster-users] Issue in Adding/Removing the gluster node
> >>> >
> >>> > Hi,
> >>> >
> >>> >
> >>> > I am working on two board setup connecting to each other. Gluster
> >>> version
> >>> > 3.7.6 is running and added two bricks in replica 2 mode but when I
> >>> manually
> >>> > removed (detach) the one board from the setup I am getting the
> >>> following
> >>> > error.
> >>> >
> >>> > volume remove-brick c_glusterfs replica 1 10.32.1.144:
> >>> /opt/lvmdir/c2/brick
> >>> > force : FAILED : Incorrect brick 10.32.1.144:/opt/lvmdir/c2/brick
> for
> >>> > volume c_glusterfs
> >>> >
> >>> > Please find the logs file as an attachment.
> >>> >
> >>> >
> >>> > Regards,
> >>> > Abhishek
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Gluster-users mailing list
> >>> > Gluster-users at gluster.org
> >>> > http://www.gluster.org/mailman/listinfo/gluster-users
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>>
> >>>
> >>>
> >>>
> >>> Regards
> >>> Abhishek Paliwal
> >>>
> >>
> >>
> >>
> >> --
> >>
> >>
> >>
> >>
> >> Regards
> >> Abhishek Paliwal
> >>
> >>
> >
> >
> >
> >
>
>
> --
>
>
>
>
> Regards
> Abhishek Paliwal
>