[Gluster-devel] glusterd crashes when synctask is used in conjunction with inner functions.

[Jeff Darcy]

On 10/08/2012 01:21 PM, Krishnan Parthasarathi wrote:
> FWIW, inner functions also make it harder to debug with gdb (read breakpoints).
> I am yet to demonstrably establish that inner functions' would
> crash when run in synctask. I hope some of you might be able to shed
> light on how I could resolve that.

I think to answer that, we'd need to examine how inner functions are (or at
least might be) implemented.  The fundamental problem is how the inner function
actually accesses variables within the outer function's local scope.  For a
direct call from outer to inner, it's pretty easy - it's just another set of
stack offsets.  Otherwise, the inner function needs a pointer to the outer
function's stack frame.  How does it get that?  Functions in between might be
totally obvlivious to this pointer (it's not in their argument lists) so you
can't just pass it directly.  They can't be stored directly in function
pointers either, for similar reasons.  The only seemingly workable option would
be to generate a "thunk" dynamically, something like this:

	int
	real_inner_function (void *outer_frame, int arg)
	{
		/* Same as user's function but with extra arg. */
	}

	int
	inner_function (int arg)
	{
		real_inner_function(COMPILE_TIME_CONSTANT,arg);
	}

So now a pointer to inner_function is actually a pointer to a specific
invocation within the context of an outer-function call.  What happens when
that outer function returns?  Uh oh.  Is any of this safe in the presence of
both pthreads and ucontext threads?  Uh oh again, and we do use both in our
synctask implementation.  Hmmm.  In a way, we're doing almost everything we can
to trip up gcc's inner-function implementation, so it's kind of no surprise
that we've succeeded.

I haven't actually examined the gcc code to see if that's how inner functions
are implemented.  There are other possibilities, but most are equally
susceptible to our threading shenanigans.  The real proof point is whether the
problems go away if we stop using inner functions.  There aren't too many uses
currently, so it shouldn't be a prohibitively difficult experiment, and if
you're right that inner functions to blame then we don't need to figure out
why.  We should just stop using them.