[Gluster-devel] break glusterd into small parts (Re: good job on fixing heavy hitters in spurious regressions)

Sat May 9 10:41:25 UTC 2015

On 05/09/2015 03:04 PM, Kaushal M wrote:
> Modularising GlusterD is something we plan to do.  As of now, it's
> just that a plan. We don't yet have a design to achieve it yet.
>
> What Atin mentioned and what you've mentioned seem to be the same at a
> high level. The core of GlusterD will be a co-ordinating engine, which
> defines an interface for commands to use to do their work. The
> commands will each be a seperate module implementing this interface.
> Depending on how we implement, the actual names will be different.
Yes, this is a nice approach. It would be nice if there is a clear 
demarcation as well for the code, so there won't be any dependency with 
merging dht changes vs say afr changes in cli. That is why I was 
suggesting xlator based solution. But other ways of doing it where there 
is clear demarcation is welcome as well. Would love to know more about 
the other approaches :-).

Pranith
>
>
> On Sat, May 9, 2015 at 2:24 PM, Pranith Kumar Karampuri
> <pkarampu at redhat.com> wrote:
>> On 05/09/2015 02:21 PM, Atin Mukherjee wrote:
>>>
>>> On 05/09/2015 01:36 PM, Pranith Kumar Karampuri wrote:
>>>> On 05/09/2015 11:08 AM, Krishnan Parthasarathi wrote:
>>>>>> Ah! now I understood the confusion. I never said maintainer should fix
>>>>>> all the bugs in tests. I am only saying that they maintain tests, just
>>>>>> like we maintain code. Whether you personally work on it or not, you at
>>>>>> least have an idea of what is the problem and what is the solution so
>>>>>> someone can come and ask you and you know the status of it. Expectation
>>>>>> is not to fix every test failure that comes maintainer's way by
>>>>>> maintainer alone. But he/she would know about problem/solution because
>>>>>> he/she at least reviews it and merges it. We want to make sure that the
>>>>>> tests are in good quality as well just like we make sure code is of
>>>>>> good
>>>>>> quality. Core is a special case. We will handle it separately.
>>>>> Glusterd is also a 'special' case. As a glusterd maintainer, I am
>>>>> _not_ maintaining
>>>>> <insert-your-favourite-gluster-command-here>'s implementation. So, I
>>>>> don't
>>>>> 'know'/'understand' how it has been implemented and by extension I
>>>>> wouldn't be able
>>>>> to fix it (forget maintaining it :-) ). Given the no. of gluster
>>>>> commands, I won't be
>>>>> surprised if I didn't have an inkling on how
>>>>> <your-favourite-gluster-command> worked ;-)
>>>>> I hope this encourages other contributors, i.e, any gluster (feature)
>>>>> contributor,
>>>>> to join Kaushal and me in maintaining glusterd.
>>>> I understand the frustration kp :-). Human brain can only take so much.
>>>> I think we are solving wrong problem by putting more people on the code.
>>>> Why not break glusterd into small parts and distribute the load to
>>>> different people? Did you guys plan anything for 4.0 for breaking
>>>> glusterd?
>>>> It is going to be a maintenance hell if we don't break it sooner.
>>>>
>>>> Glusterd does a lot of things: Lets see how we can break things up one
>>>> thing at a time. I would love to spend some quality time thinking about
>>>> this problem once I am done with ec work, but this is a rough idea I
>>>> have for glusterd.
>>>>
>>>> 1) CLI handling:
>>>> Glusterd-cli-xlator should act something like fuse in fs. It just gets
>>>> the commands and passes it down, just like fuse gets the fops and passes
>>>> it down. In glusterd process there should be snapshot.so, afr-cli.so,
>>>> ec-cli.so, dht-cli.so loaded as management-xlators.
>>>> Just like we have fops lets have mops (management operations).
>>>> LOCK/STAGE/BRICK-OP/COMMIT-OP if there are more add them as well. Every
>>>> time the top xlator in glusterd receives commands from cli, it converts
>>>> the params into the arguments (req, op, dict etc) which are needed to
>>>> carryout the cli. Now it winds the fop to all its children. One of the
>>>> children is going to handle it locally, while the other child will send
>>>> the cli to different glusterds that are in cluster. Second child of
>>>> gluster-cli-xlator (give it a better name, but for now lets call it:
>>>> mgmtcluster) will collate the responses and give the list of responses
>>>> to glusterd-cli-xlator, it will call COLLATE mop on the first-child(lets
>>>> call it local-handler) to collate the responses, i.e. logic for
>>>> collating responses should also be in snapshot.so, afr-cli.so,
>>>> dht-cli.so etc etc. Once the top translator does LOCK, STAGE, BRICK-OP,
>>>> COMMIT-OP send response to CLI.
>>>>
>>>> 2) Volinfo should become more like inode_t in fs where each *-cli xlator
>>>> can store their own ctx like snapshot-cli can store all snapshot related
>>>> info for that volume in that context and afr can store afr-related info
>>>> in the ctx. Volinfo data strcuture should have very minimal information.
>>>> Maybe name, bricks etc.
>>>>
>>>> 3) Daemon handling:
>>>>        Daemon-manager xlator should have MOPS like START/STOP/INFO and
>>>> this xlator should be accessible for all the -cli xlators which want to
>>>> do their own management of the daemons. i.e. ec-cli/afr-cli should do
>>>> self-heal-daemon handling. dht should do rebalance process handling etc.
>>>> to give an example:
>>>> while winding START mop it has to specify the daemon as
>>>> "self-heal-daemon" and give enough info etc.
>>>>
>>>> 4) Peer handling:
>>>>       mgmtcluster(second child of top-xlator) should have MOPS like
>>>> PEER_ADD/PEER_DEL/PEER_UPDATE etc to do the needful. top xlator is going
>>>> to wind these operations based on the peer-cli-commands to this xlator.
>>>>
>>>> 5) volgen:
>>>>       top xlator is going to wind MOP called GET_NODE_LINKS, which takes
>>>> the type of volfile (i.e. mount/nfs/shd/brick etc) on which each *-cli
>>>> will construct its node(s), stuff options and tell the parent xl-name to
>>>> which it needs to be linked to. Top xlator is going to just link the
>>>> nodes to construct the graph and does graph_print to generate the
>>>> volfile.
>>>>
>>>> I am pretty sure I forgot some more aspects of what glusterd does but
>>>> you get the picture right? Break each aspect into different xlator and
>>>> have MOPS to solve them.
>>> Sounds interesting but needs to be thought out in details. For 4.0,wWe
>>> do have a plan to make core glusterd algorithms work as a glusterd
>>> engine and other features will work have interfaces to connect to it.
>>> Your proposal looks another alternative. I would like to hear from the
>>> community about it.
>> Whatever works to reduce the burden on you folks :-).
>>
>> Pranith
>>
>>>> Pranith
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel