[Gluster-devel] Data classification proposal

[Krishnan Parthasarathi]

Jeff,

----- Original Message -----
> > Am I right if I understood that the value for media-type is not
> > interpreted beyond the scope of matching rules? That is to say, we
> > don't need/have any notion of media-types that type check internally
> > for forming (sub)volumes using the rules specified.
> 
> Exactly.  To us it's just an opaque ID.

OK. That makes sense.

> 
> > Should the no. of bricks or lower-level subvolumes that match the rule
> > be an exact multiple of group-size?
> 
> Good question.  I think users see the current requirement to add bricks
> in multiples of the replica/stripe size as an annoyance.  This will only
> get worse with erasure coding where the group size is larger.  On the
> other hand, we do need to make sure that members of a group are on
> different machines.  This is why I think we need to be able to split
> bricks, so that we can use overlapping replica/erasure sets.  For
> example, if we have five bricks and two-way replication, we can split
> bricks to get a multiple of two and life's good again.  So *long term* I
> think we can/should remove any restriction on users, but there are a
> whole bunch of unsolved issues around brick splitting.  I'm not sure
> what to do in the short term.

For the short-term, wouldn't it be OK to disallow adding bricks that is not
a multiple of group-size?

> 
> > > Here's a more complex example that adds replication and erasure
> > > coding to the mix.
> > >
> > >     # Assume 20 hosts, four fast and sixteen slow (named
> > >     appropriately).
> > >
> > >     rule tier-1
> > >             select *fast*
> > >             group-size 2
> > >             type cluster/afr
> > >
> > >     rule tier-2
> > >             # special pattern matching otherwise-unused bricks
> > >             select %{unclaimed}
> > >             group-size 8
> > >             type cluster/ec parity=2
> > >             # i.e. two groups, each six data plus two parity
> > >
> > >     rule all
> > >             select tier-1
> > >             select tier-2
> > >             type features/tiering
> > >
> >
> > In the above example we would have 2 subvolumes each containing 2
> > bricks that would be aggregated by rule tier-1. Lets call those
> > subvolumes as tier-1-fast-0 and tier-fast-1.  Both of these subvolumes
> > are afr based two-way replicated subvolumes.  Are these instances of
> > tier-1-* composed using cluster/dht by the default semantics?
> 
> Yes.  Any time we have multiple subvolumes and no other specified way to
> combine them into one, we just slap DHT on top.  We do this already at
> the top level; with data classification we might do it at lower levels
> too.
> 

thanks,
Krish