[Gluster-devel] Reg Upcall Cache Invalidation: asynchronous notification + Cleanup of expired states

[Soumya Koduri]

Hi Shyam/Niels,

To re-iterate the issues,

a) at present, when two clients access same file, we send 
'cache_invalidation' upcall notification to the first client in the fop 
cbk path of the second client. This may affect brick latency esp., for 
the directories (where there are more chances of multiple clients 
accessing them at the same time.).

b) clean-up the expired client state which no longer access any of the 
files/directories.


Proposed solution -
a) I initially intend to maintain a list of all upcall notifications and 
spawn a thread which reads the list and sends notifications sequentially.

But the issue I foresee with this approach is that, since there will be 
a single thread sending notifications (which could be in huge number 
depending on the client I/O workloads), there may be delay in clients 
receiving this notification which may break the whole purpose of sending 
these notifications. Instead, after having a discussion with KP, plan to 
create synctasks to send upcall notifications. If there are any issues 
with it, maybe we need to use thread_pool.

Any thoughts?

b) approach taken here is that -
   * maintain a global list to contain all the upcall_inode_ctx allocated
   * Every time a upcall_inode_ctx is allocated, it is added to the 
global list
   * during inode_forget, that upcall_inode_ctx is marked for destroy
   * created a reaper thread which scans through that list
          * cleans up expired client entries
          * frees the inode_ctx with destroy_mode set.

patch URL: http://review.gluster.org/#/c/10342/

Kindly review the changes and provide your inputs on the same.

Thanks,
Soumya