[Gluster-devel] ZkFarmer

[Ian Latter]

> > Is there anything written up on why you/all want every
> > node to be completely conscious of every other node?
> > 
> > I could see a couple of architectures that might work
> > better (be more scalable) if the config minutiae were 
> > either not necessary to be shared or shared in only 
> > cases where the config minutiae were a dependency.
> 
> Well, these aren't exactly minutiae.  Everything at file
or directory level is
> fully distributed and will remain so.  We're talking only
about stuff at the
> volume or server level, which is very little data but very
broad in scope.
> Trying to segregate that only adds complexity and
subtracts convenience,
> compared to having it equally accessible to (or through)
any server.

Sorry, I didn't have time this morning to add more detail.

Note that my concern isn't bandwidth, its flexibility; the
less knowledge needed the more I can do crazy things 
in user land, like running boxes in different data centres 
and randomly power things up and down, randomly re-
address, randomly replace in-box hardware, load 
balance, NAT, etc.  It makes a dynamic environment 
difficult to construct, for example, when Gluster rejects
the same volume-id being presented to an existing 
cluster from a new GFID.

But there's no need to go even that complicated, let 
me pull out an example of where shared knowledge 
may be unnecessary;

The work that I was doing in Gluster (pre glusterd) drove
out one primary "server" which fronted a Replicate 
volume of both its own Distribute volume and that of 
another server or two - themselves serving a single 
Distribute volume.  So the client connected to one 
server for one volume and the rest was black box /
magic (from the client's perspective - big fast storage
in many locations); in that case it could be said that 
servers needed some shared knowledge, while the 
clients didn't.

The equivalent configuration in a glusterd world (from
my experiments) pushed all of the distribute knowledge 
out to the client and I haven't had a response as to how
to add a replicate on distributed volumes in this model, 
so I've lost replicate.  But in this world, the client must 
know about everything and the server is simply a set
of served/presented disks (as volumes).  In this 
glusterd world, then, why does any server need to 
know of any other server, if the clients are doing all of 
the heavy lifting?  

The additional consideration is where the server both
consumes and presents, but this would be captured in
the client side view.  i.e. given where glusterd seems
to be driving, this knowledge seems to be needed on 
the client side (within glusterfs, not glusterfsd).

To my mind this breaks the gluster architecture that I
read about 2009, but I need to stress that I didn't get 
a reply to the glusterd architecture question that I 
posted about a month ago;  so I don't know if glusterd
is currently limiting deployment options because;
  - there is an intention to drive the heavy lifting to the
    client (for example for performance reasons in big
    deployments), or;
  - there are known limitations in the existing bricks/
    modules (for example moving files thru distribute), 
    or;
  - there is ultimately (long term) more flexibility seen
    in this model (and we're at a midway point between 
    pre glusterd and post so it doesn't feel that way 
    yet), or;
  - there is an intent to drive out a particular market 
    outcome or match an existing storage model (the 
    gluster presentation was driving towards cloud,
    and maybe those vendors don't use server side
    implementations), etc.

As I don't have a clear/big picture in my mind; if I'm 
not considering all of the impacts, then my apologies.


> > RE ZK, I have an issue with it not being a binary at
> > the linux distribution level.  This is the reason I don't
> > currently have Gluster's geo replication module in
> > place ..
> 
> What exactly is your objection to interpreted or JIT
compiled languages?
> Performance?  Security?  It's an unusual position, to say
the least.
> 

Specifically, primarily, space.  Saturn builds GlusterFS
capacity from a 48 Megabyte Linux distribution and 
adding many Megabytes of Perl and/or Python and/or 
PHP and/or Java for a single script is impractical.

My secondary concern is licensing (specifically in the 
Java run-time environment case).  Hadoop forced my
hand; GNU's JRE/compiler wasn't up to the task of 
running Hadoop when I last looked at it (about 2 or 3 
years ago now) - well, it could run a 2007 or so 
version but not current ones at that time - so now I 
work with Gluster ..



Going back to ZkFarmer;

Considering other architectures; it depends on how 
you slice and dice the problem as to how much 
external support you need;
  > I've long felt that our ways of dealing with cluster 
  > membership and staging of config changes is not 
  > quite as robust and scalable as we might want.

By way of example;
  The openMosix kernel extensions maintained their
own information exchange between cluster nodes; if
a node (ip) was added via the /proc interface, it was
"in" the cluster.  Therefore cluster membership was
the hand-off/interface.  
  It could be as simple as a text list on each node, or 
it could be left to a user space daemon which could 
then gate cluster membership - this suited everyone
with a small cluster.
  The native daemon (omdiscd) used multicast 
packets to find nodes and then stuff those IP's into
the /proc interface - this suited everyone with a 
private/dedicated cluster.
  A colleague and I wrote a TCP variation to allow 
multi-site discovery with SSH public key exchanges 
and IPSEC tunnel establishment as part of the 
gating process - this suited those with a distributed/
part-time cluster.  To ZooKeeper's point 
(http://zookeeper.apache.org/), the discovery 
protocol that we created was weak and I've since 
found a model/algorithm that allows for far more 
robust discovery.

  The point being that, depending on the final cluster
architecture for gluster (i.e. all are nodes are peers
and thus all are cluster members, nodes are client 
or server and both are cluster members, nodes are 
client or server and only clients [or servers] are 
cluster members, etc) there may be simpler cluster 
management options .. 


Cheers,




--
Ian Latter
Late night coder ..
http://midnightcode.org/