[Gluster-devel] break glusterd into small parts (Re: good job on fixing heavy hitters in spurious regressions)

[Pranith Kumar Karampuri]

On 05/09/2015 11:08 AM, Krishnan Parthasarathi wrote:
>> Ah! now I understood the confusion. I never said maintainer should fix
>> all the bugs in tests. I am only saying that they maintain tests, just
>> like we maintain code. Whether you personally work on it or not, you at
>> least have an idea of what is the problem and what is the solution so
>> someone can come and ask you and you know the status of it. Expectation
>> is not to fix every test failure that comes maintainer's way by
>> maintainer alone. But he/she would know about problem/solution because
>> he/she at least reviews it and merges it. We want to make sure that the
>> tests are in good quality as well just like we make sure code is of good
>> quality. Core is a special case. We will handle it separately.
> Glusterd is also a 'special' case. As a glusterd maintainer, I am _not_ maintaining
> <insert-your-favourite-gluster-command-here>'s implementation. So, I don't
> 'know'/'understand' how it has been implemented and by extension I wouldn't be able
> to fix it (forget maintaining it :-) ). Given the no. of gluster commands, I won't be
> surprised if I didn't have an inkling on how <your-favourite-gluster-command> worked ;-)
> I hope this encourages other contributors, i.e, any gluster (feature) contributor,
> to join Kaushal and me in maintaining glusterd.
I understand the frustration kp :-). Human brain can only take so much. 
I think we are solving wrong problem by putting more people on the code. 
Why not break glusterd into small parts and distribute the load to 
different people? Did you guys plan anything for 4.0 for breaking glusterd?
It is going to be a maintenance hell if we don't break it sooner.

Glusterd does a lot of things: Lets see how we can break things up one 
thing at a time. I would love to spend some quality time thinking about 
this problem once I am done with ec work, but this is a rough idea I 
have for glusterd.

1) CLI handling:
Glusterd-cli-xlator should act something like fuse in fs. It just gets 
the commands and passes it down, just like fuse gets the fops and passes 
it down. In glusterd process there should be snapshot.so, afr-cli.so, 
ec-cli.so, dht-cli.so loaded as management-xlators.
Just like we have fops lets have mops (management operations). 
LOCK/STAGE/BRICK-OP/COMMIT-OP if there are more add them as well. Every 
time the top xlator in glusterd receives commands from cli, it converts 
the params into the arguments (req, op, dict etc) which are needed to 
carryout the cli. Now it winds the fop to all its children. One of the 
children is going to handle it locally, while the other child will send 
the cli to different glusterds that are in cluster. Second child of 
gluster-cli-xlator (give it a better name, but for now lets call it: 
mgmtcluster) will collate the responses and give the list of responses 
to glusterd-cli-xlator, it will call COLLATE mop on the first-child(lets 
call it local-handler) to collate the responses, i.e. logic for 
collating responses should also be in snapshot.so, afr-cli.so, 
dht-cli.so etc etc. Once the top translator does LOCK, STAGE, BRICK-OP, 
COMMIT-OP send response to CLI.

2) Volinfo should become more like inode_t in fs where each *-cli xlator 
can store their own ctx like snapshot-cli can store all snapshot related 
info for that volume in that context and afr can store afr-related info 
in the ctx. Volinfo data strcuture should have very minimal information. 
Maybe name, bricks etc.

3) Daemon handling:
      Daemon-manager xlator should have MOPS like START/STOP/INFO and 
this xlator should be accessible for all the -cli xlators which want to 
do their own management of the daemons. i.e. ec-cli/afr-cli should do 
self-heal-daemon handling. dht should do rebalance process handling etc. 
to give an example:
while winding START mop it has to specify the daemon as 
"self-heal-daemon" and give enough info etc.

4) Peer handling:
     mgmtcluster(second child of top-xlator) should have MOPS like 
PEER_ADD/PEER_DEL/PEER_UPDATE etc to do the needful. top xlator is going 
to wind these operations based on the peer-cli-commands to this xlator.

5) volgen:
     top xlator is going to wind MOP called GET_NODE_LINKS, which takes 
the type of volfile (i.e. mount/nfs/shd/brick etc) on which each *-cli 
will construct its node(s), stuff options and tell the parent xl-name to 
which it needs to be linked to. Top xlator is going to just link the 
nodes to construct the graph and does graph_print to generate the volfile.

I am pretty sure I forgot some more aspects of what glusterd does but 
you get the picture right? Break each aspect into different xlator and 
have MOPS to solve them.

Pranith