[Gluster-devel] managing of THIS

[Raghavendra Bhat]

Hi,

In glusterfs at the time of the process coming up, it creates 5 pthread 
keys (for saving THIS, syncop, uuid buf, lkowner buf and syncop ctx).  
Even gfapi does the same thing in its glfs_new function. But with User 
Serviceable Snapshots (where a glusterfs process spawns multiple gfapi 
instances for a snapshot) this will lead to more and more consumption of 
pthread keys. In fact the old keys are lost (as same variables are used 
for creating the keys) and eventually the process will run out of 
pthread keys after 203 snapshots (maximum allowed number is 1024 per 
process.). So to avoid it, pthread keys creation can be done only once 
(using pthread_once in which, the globals_init function is called).

But now a new problem arises. Say, from snapview-server xlator glfs_new 
was called (or glfs_init etc). Now gfapi wants calls THIS for some of 
its operations such properly accounting the memory within the xlator 
while allocating a new structure. But when gfapi calls THIS it gets 
snapview-server's pointer. Since snapview-server does not know about 
gfapi's internal structure, it asserts at the time of allocating.

For now, a patch has been sent to handle the issue by turning off 
memory-accounting for snapshot daemon.
(http://review.gluster.org/#/c/9430).

But if memory-accounting has to be turned on for snapshot daemon, then 
the above problem has to be fixed.
2 ways that can be used for fixing the issue are:

1) Add the datastructures that are used by gfapi to libglusterfs (and 
hence their mem-types as well), so that any xlator that is calling gfapi 
functions (such as snapview-server as of now) will be aware of the 
memory types used by gfapi and hence will not cause problems, when 
memory accounting has to be done as part of allocations and frees.

OR

2) Properly manage THIS by introducing a new macro similar to STACK_WIND 
(for now it can be called STACK_API_WIND). The macro will be much 
simpler than STACK_WIND as it need not create new frames before handing 
over the call to the next layer. Before handing over the call to gfapi 
(any call, such as glfs_new or fops such as glfs_h_open), saves THIS in 
a variable and calls the gfapi function given as an argument. After the 
function returns it again sets THIS back the value before the gfapi 
function was called.

Ex:

STACK_API_WIND (this, fn, ret, params....)
do {
     xlator_t *old_THIS = NULL;

     old_THIS = this;
     ret = fn (params);
     THIS = old_THIS;
} while (0);

a caller (as of now snapview-server xlator) would call the macro like this
STACK_API_WIND (this, glfs_h_open, glfd, fs, object, flags);


Please provide feedback and any suggestions or solutions to handle the 
mentioned issue are welcome.

Regards,
Raghavendra Bhat