[Gluster-devel] Some updates on the eventing framework for Gluster

[Samikshan Bairagya]

On 12/02/2015 06:47 AM, Nagaprasad Sathyanarayana wrote:
> Any specific reasons for going with Kafka? What is the advantage of using Kafka over RabbitMQ?
>

Hi,

I do not actually have much grounds for my bias towards Kafka and could 
use either RabbitMQ or Kafka for the purpose of cluster-wide messaging. 
Its also true that it would not require a lot of effort to shift from 
Kafka to RabbitMQ at this point of the project. Having said that, Kafka 
apparently does have better performance than RabbitMQ, but again it is 
doubtful if we might need that scale of performance to be achieved.

Thanks and Regards,

Samikshan

> Thanks
> Naga
>
>
>> On Dec 2, 2015, at 6:09 AM, Samikshan Bairagya <@redhat.com> wrote:
>>
>> Hi,
>>
>> The updates for the eventing framework for gluster can be divided into the following two parts.
>>
>> 1. Bubbling out notifications through dbus signals from every gluster node.
>>
>> * The 'glusterfs' module in storaged [1] exports objects on the system bus for every gluster volume. These objects hold the following properties:
>> - Name
>> - Id
>> - Status (0 = Created, 1 = Started, 2 = Stopped)
>> - Brickcount
>>
>> * A singleton dbus object corresponding to glusterd is also exported by storaged on the system bus. This object holds properties to track the state of glusterd (LoadState and ActiveState).
>>
>> 2. Aggregating all these signals from each node over an entire cluster.
>>
>> * Using Kafka [2] for messaging over a cluster: Implementing a (dbus signal) listener in python that converts these dbus signals from objects to 'keyed messages' in Kafka under a particular 'topic'.
>>
>> For example, if a volume 'testvol' is started, a message is published under topic 'testvol', with 'status' as the 'key' and the changed status ('1' in this case) as the 'value'.
>>
>>
>> *** Near term plans:
>> - Export dbus objects corresponding to bricks.
>> - Figure out how to map the path to the brick directory to the block device and consequently the drive object. The 'SmartFailing' property from org.storaged.Storaged.Drive.Ata [3] interface can then be used to track brick failures.
>> - Make the framework work over a multi-node cluster with possibly a multi-broker kafka setup to identify redundancies as well as to keep consistent information across the cluster.
>>
>> Views/feedback/queries are welcome.
>>
>> [1] https://github.com/samikshan/storaged/tree/glusterfs
>> [2] http://kafka.apache.org/documentation.html#introduction
>> [3] http://storaged-project.github.io/doc/latest/gdbus-org.storaged.Storaged.Drive.Ata.html#gdbus-property-org-storaged-Storaged-Drive-Ata.SmartFailing
>>
>> Thanks and Regards,
>>
>> Samikshan
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>
>>