[Gluster-devel] Data classification infrastructure

[Venky Shankar]

> It looks like a lot of work went into this.  Kudos for that.  Here are
> some quick thoughts.

Thanks!

> 
> * I wouldn't worry too much about NSR in this design.  NSR is evolving
> toward a full-data-logging design.  I don't think changelog should (or
> is likely to) evolve in that same direction.  As noted in the document,
> NSR is also unique in other ways such as durability requirements, so I
> think it makes sense to exclude it from the list of valid changelog use
> cases.

Hmm. Full data logging is something that changelog would try to avoid. But having both data and metadata journaling translator in the stack seems to be redundant to some extent.

> 
> * For putting changelog on its own SSD, how do the changelog translator
> and libgfchangelog each know where that is?  The first seems to be a
> simple translator option.  The second, and particularly coordination
> between the two, might require a bit more effort.

Well, libgfchangelog does not need to know the journal path. Clients register themselves with the changelog translator to receive updates. This does put a good amount of load on changelog translator. Another option could be libgfchangelog reading the changelog itself (till the last "valid" written offset). fd could be open()'ed by changelog translator (during client register) and handed over back to the client (unix fd pass magic). This way, non-root clients can still read the journal even tough the permission are restrictive.

> 
> * One of the key issues here is multiple consumers, particularly issues
> such as backpressure and garbage collection in the presence of same.
> 
> * Is the LRU/LFU cache really part of changelog, or should it be
> separate?  Either way, we probably need a lot more detail to address
> similar issues of currency, space usage, garbage collection, etc.

I tried to keep anything that's dealing with "what-has-changed" related to changelog. Suggestions are welcome. As far as space usage and GC are concerned -- those still need thinking. Regarding LRU/LFU snap, a scalable "single writer-multi reader" algorithm with minimal write starvation needs to be chosen. I haven't thought much regarding this as of now, but this part needs the utmost care.

Thoughts?

> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>