[Gluster-devel] break glusterd into small parts (Re: good job on fixing heavy hitters in spurious regressions)

[Pranith Kumar Karampuri]

On 05/09/2015 03:19 PM, Krishnan Parthasarathi wrote:
>> Why not break glusterd into small parts and distribute the load to
>> different people? Did you guys plan anything for 4.0 for breaking glusterd?
>> It is going to be a maintenance hell if we don't break it sooner.
> Good idea. We have thought about it. Just re-architecting glusterd doesn't
> (and will not) solve the division of responsibility issue that is being discussed here.
> It's already difficult to maintain glusterd. I have already explained the reasons
> in the previous thread.
I was thinking *-cli xlators could be maintained by the respective fs 
team itself. It is easier to maintain it this way because each of those 
xls can be put in xlators/cluster/afr/cli, xlators/cluster/dht/cli, etc. 
There will be clear demarcation of who owns what this way is my feeling. 
Even the tests can be organized to tests/afr-cli, tests/dht-cli etc etc.

>
>> Glusterd does a lot of things: Lets see how we can break things up one
>> thing at a time. I would love to spend some quality time thinking about
>> this problem once I am done with ec work, but this is a rough idea I
>> have for glusterd.
>>
>> 1) CLI handling:
>> Glusterd-cli-xlator should act something like fuse in fs. It just gets
>> the commands and passes it down, just like fuse gets the fops and passes
>> it down. In glusterd process there should be snapshot.so, afr-cli.so,
>> ec-cli.so, dht-cli.so loaded as management-xlators.
>> Just like we have fops lets have mops (management operations).
>> LOCK/STAGE/BRICK-OP/COMMIT-OP if there are more add them as well. Every
>> time the top xlator in glusterd receives commands from cli, it converts
>> the params into the arguments (req, op, dict etc) which are needed to
>> carryout the cli. Now it winds the fop to all its children. One of the
>> children is going to handle it locally, while the other child will send
>> the cli to different glusterds that are in cluster. Second child of
>> gluster-cli-xlator (give it a better name, but for now lets call it:
>> mgmtcluster) will collate the responses and give the list of responses
>> to glusterd-cli-xlator, it will call COLLATE mop on the first-child(lets
>> call it local-handler) to collate the responses, i.e. logic for
>> collating responses should also be in snapshot.so, afr-cli.so,
>> dht-cli.so etc etc. Once the top translator does LOCK, STAGE, BRICK-OP,
>> COMMIT-OP send response to CLI.
>>
>> 2) Volinfo should become more like inode_t in fs where each *-cli xlator
>> can store their own ctx like snapshot-cli can store all snapshot related
>> info for that volume in that context and afr can store afr-related info
>> in the ctx. Volinfo data strcuture should have very minimal information.
>> Maybe name, bricks etc.
>>
>> 3) Daemon handling:
>>        Daemon-manager xlator should have MOPS like START/STOP/INFO and
>> this xlator should be accessible for all the -cli xlators which want to
>> do their own management of the daemons. i.e. ec-cli/afr-cli should do
>> self-heal-daemon handling. dht should do rebalance process handling etc.
>> to give an example:
>> while winding START mop it has to specify the daemon as
>> "self-heal-daemon" and give enough info etc.
>>
>> 4) Peer handling:
>>       mgmtcluster(second child of top-xlator) should have MOPS like
>> PEER_ADD/PEER_DEL/PEER_UPDATE etc to do the needful. top xlator is going
>> to wind these operations based on the peer-cli-commands to this xlator.
>>
>> 5) volgen:
>>       top xlator is going to wind MOP called GET_NODE_LINKS, which takes
>> the type of volfile (i.e. mount/nfs/shd/brick etc) on which each *-cli
>> will construct its node(s), stuff options and tell the parent xl-name to
>> which it needs to be linked to. Top xlator is going to just link the
>> nodes to construct the graph and does graph_print to generate the volfile.
>>
>> I am pretty sure I forgot some more aspects of what glusterd does but
>> you get the picture right? Break each aspect into different xlator and
>> have MOPS to solve them.
> We have some initial ideas on how glusterd for 4.0 would look like. We won't be
> continuing with glusterd is also a translator model. The above model would
> work well only if we stuck with the stack of translators approach.
Oh nice, I might have missed the mails. Do you mind sharing the plan for 
4.0? Any reason why you guys do not want to continue glusterd as 
translator model?

Pranith