[Gluster-devel] Backup support for GlusterFS

[Niels de Vos]

On Fri, Aug 19, 2016 at 02:53:57PM +0530, Alok Srivastava wrote:
> On Wed, Aug 10, 2016 at 1:35 AM, Vijay Bellur <vbellur at redhat.com> wrote:
> 
> > On 08/08/2016 04:57 AM, Alok Srivastava wrote:
> >
> >> Reviving the old thread..
> >>
> >> I was thinking about a possible NDMP based migration tool built into
> >> GlusterFS.
> >> Migration from any other file system storage (source) to gluster storage
> >> (target) can be challenging because of lack of a tool built into
> >> GlusterFS. Host based migration is a common approach but it can be
> >> painful for large datasets.
> >>
> >> The proposed migration tool will  transfer data over the Network Data
> >> Management Protocol (NDMP). This tool will make use of the snapshots
> >> feature on the source system to ensure that a point-in- time copy of the
> >> file system is transferred to the target. The proposed tool can also be
> >> utilized to migrate from gluster on old hardware to gluster on new
> >> hardware.
> >>
> >> Following is the proposed workflow:
> >>
> >> *1) **First copy*
> >>
> >> Migrate recursively from source directory to the gluster cluster. This
> >> is expected to be a time consuming step.
> >>
> >> * 2)      Incremental copy*
> >>
> >> Migrating any changes that occur after the first copy.
> >>
> >>   3)      Cutover
> >>
> >> Restrict clients from accessing data on the source , migrate remaining
> >> changes , and redirect clients to access their data on the gluster
> >> cluster.
> >>
> >>
> >> Note that there may be issues with migrating existing file permissions,
> >> quota enforcements, etc on the source.
> >>
> >> Missing pieces in gluster
> >>
> >>
> >> -NDMP support
> >>
> >> -May have to revive “ndmp-server” project?
> >>
> >>
> >>
> >
> > Why is NDMP better than rsync from NFS (src) to FUSE (destination)?. Is it
> > owing to the incremental copy that NDMP can handle better?
> >
> 
> Right, Determining the files for incremental  migration will take longer
> with rsync. Also, the proposed approach is based on NDMP + Snapshots, Hence
> it's not a one size fits all approach. However, Making use of the snapshots
> will ensure that  a point in time copy is migrated and the in-flight
> directories are also accessible to the clients connected to the source.
> 
> NDMP support with gluster will help in integration with other backup
> utilities as well.

Do you have examples of backup utilities/communities that would like to
see NDMP support? I have never seen any requests from users about this.
We already have a nice integration with Bareos, and that supports NDMP I
think.

Maybe it would be a better approach to let backup software handle the
NDMP part, and Gluster just offers a nice integrated way to consume
point-in-time snapshots?

Niels


> 
> >
> > I have not seen the ndmp-server project in action or the code to comment
> > on how functional it was. We might have to implement from scratch if we
> > decide to go down this route.
> >
> > Aravinda - do you remember the state of this project?
> >
> > Thanks,
> > Vijay
> >
> >
> >

> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160819/1a9a7758/attachment-0001.sig>