[Gluster-devel] HTIME API

[Venky Shankar]

On 10/31/2014 12:28 PM, Aravinda wrote:
> Since HTIME file is available we need not maintain changelogs in two 
> different places as we do now(backend and working dir), We can provide 
> backend changelog names to consumer instead of working dir names. One 
> problem with this is for parsing these changelogs they have to use one 
> more RPC call. How about returning FOP details directly instead of 
> changelog file names?

Nicely put Aravinda.

That's exactly what's planned for Gnotify (lowlevel APIs), where the 
consumer does not have to _know_ the changelog format. The fops are 
notified (via callback) in a way similar to inotify(7) in an event 
buffer. Avoiding maintaining state given that we keep performance on par 
would be a huge plus point. Further, we could relax ordering and have 
callback invoked from multiple threads for cases when ordering does not 
really matter.

>
> If we use db(sqlite) internally in API itself then we can do multiple 
> optimization.
>
> 1. If multiple data operations in requested time for a file then 
> return only one
> 2. If a file/gfid is created and deleted in the requested time range 
> then skip from the output.
> 3. If a file/gfid is deleted in the given time then do not return data 
> fops.
>
> Other challenge is getting cluster view of these notifications.(and 
> RENAME issues)

Correct. Cluster view needs a good amount of thinking w.r.t 
functionality, performance, etc.

>
> -- 
> regards
> Aravinda
>
> On 10/31/2014 11:10 AM, Kotresh Hiremath Ravishankar wrote:
>> Hi Venky,
>>
>> If I understand correctly, the idea is to provide RPC based history API?
>> If yes, how can that solve the different clients/consumers having to 
>> maintain
>> state of how much it has processed?
>>
>> Thanks and Regards,
>> Kotresh H R
>>
>> ----- Original Message -----
>> From: "Venky Shankar" <vshankar at redhat.com>
>> To: "Ajeet Jha" <ajha at redhat.com>
>> Cc: "Kotresh Hiremath Ravishankar" <khiremat at redhat.com>, 
>> gluster-devel at gluster.org
>> Sent: Wednesday, October 29, 2014 6:23:40 PM
>> Subject: HTIME API
>>
>> Hey Ajeet,
>>
>> Since changelog translator maintains a meta file (HTIME) containing a 
>> list of consumable changelogs, we could leverage this to provide an 
>> API invokable by libgfchangelog (once the interaction is RPC based) 
>> to get a list of changelogs for a given time period (similar to what 
>> history API does today).
>>
>> The benefit that this provides is that the consumers[1] need not 
>> maintain state (the API takes care of that) of the amount of 
>> changelogs [to be] processed and make use of this API to get freshly 
>> consumable changelogs.
>>
>> Thoughts?
>>
>> [1]: esp for lowlevel API 
>> (http://gluster.org/community/documentation/index.php/Features/Gnotify)
>>
>>      Venky
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-devel
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-devel