[Gluster-devel] Thoughts on enhancing replace-brick

[Joe Julian]

I completely agree that we should not have to kill (never -9 as this 
will cause the clients to wait for ping-timeout) brick processes 
directly. There should be a cli command to stop an individual brick.

The "Today" method should be corrected as follows:

  1. ssh root at node-1
  2. kill $(gluster volume status example-vol | awk '/srv.brick-02.brick.0/{print $3}')
  3. # replace physical disk
  4. remount /srv/brick-02
  5. gluster volume start example-vol force # no replace-brick necessary
  6. gluster volume heal example-vol full

I'd love to see step 2 replaced with

gluster volume stop example-vol node-1:/srv/brick-02/brick.0

This command should gracefully stop the brick, closing tcp connections. 
It should fail if there are pending-heals with that brick as the source 
unless you add the "force" keyword at the end.

This would turn your hypothetical example into

  1. ssh root at node-1
  2. gluster volume stop example-vol node-1:/srv/brick-02/brick.0
  3. # replace physical disk
  4. remount /srv/brick-02
  5. gluster volume start example-vol force
  6. gluster volume heal example-vol full




On 03/18/2016 08:11 AM, Peter Portante wrote:
> Hi Folks,
>
> There is a review posted, http://review.gluster.org/#/c/12250, to 
> which I tacked on a review comment for an update to the replace-brick 
> command. The gist of it is at 
> https://gist.github.com/portante/248407dbfb29c2515fc3
>
gist included for completeness:

  > I like where this is going.  Perhaps we could consider the steps an
  > admin has to take and orient the commands to address what they need
  > to do.
  >
  > First, having to "kill -9" a process as an interface to stopping a
  > brick seems dicey.
  >
  > So to that end, perhaps we could two sets of commands, the first to
  > tell gluster to take the brick out of service, allowing the admin
  > to then replace it, and a second command to brick the brick back
  > into service.
  >
  > For example:
  >
  > gluster volume name: example-vol
  > 6 node cluster, each member providing 12 bricks, three way replicated, distributed
  > members are named node-0 ... node-5
  > disks are named /srv/brick-00/brick.0 .. /srv/brick-11/brick.0
  >
  > Lets say the disk for /srv/brick-02/brick.0 on node-1 goes bad.
  >
  > Today I believe I have to:
  >
  > 1. ssh root at node-1
  > 2. kill -9 $(gluster volume status | grep /srv/brick-02/brick | awk '{print $3}')
  > 3. # replace physical disk
  > 4. remount /srv/brick-02
  > 5. mkdir /srv/brick-02/brick.1
  > 6. # on a FUSE mnt, do the directory dance and the xattr dance
  > 7. # ensure other node brick trusted IDs for healing are correct
  > 8. gluster volume replace-brick example-vol node-1:/srv/brick-02/brick.0 node-1:/srv/brick-02/brick.1 commit force
  >
  > Perhaps we could do the following instead:
  >
  > 1. ssh root at node-1
  > 2. gluster volume replace-brick example-vol node-1:/srv/brick-02/brick.0 offline
  > 3. # replace physical disk
  > 4. remount /srv/brick-02
  > 5. mkdir /srv/brick-02/brick.0
  > 6. gluster volume replace-brick example-vol node-1:/srv/brock-02/brick.0 online
  >
  > Then the new step #6 would take care of the old steps #6 & #7. We
  > would lose the "kill -9" and replace with a declaration of
  > intention command that tells gluster to "take this brick off line,
  > stopping the brick process, and removing references to the mount
  > point".
  >
  > What do you all think?
  >
  > Please excuse my ignorance of the ins-and-outs of gluster commands
  > if I have the above steps off a bit.


> What do folks think of such a proposal?
>
> Thanks!
>
> -peter
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160318/37838d2c/attachment.html>