[Gluster-devel] debugging syntasks

[Anand Avati]

Summarizing some steps which might help others who are debugging synctask
related issues. Since synctasks uses swapcontext() to switch among
themselves, the pthread stack would only be representing the current
executing tasks stack alone (which usually would be the syncenv_processor()
scheduler.) To inspect the "stack backtrace" of other synctasks:

0. Attach gdb to the live running (possibly deadlocked or hung) process.
This approach does not work on a core dump (there might be a way to apply
the same principles on a core dump, any suggestions welcome)

1. Get the address of their 'struct synctask'. All the yielded synctasks
are linked in a list originating at syncenv->waitq. The addresses of the
links can be typecasted into 'struct synctask *' (as long as we are keeping
the first member as 'all_tasks' in the structure definition).

2. Once you have the address of the 'struct synctask' you want to inspect
the backtrace, inspect basic sanity of the structure with:

(gdb) p *(struct synctask *) 0xaddre55

Verify that the structure looks reasonably alright.

3. Perform a quick "sneak peek" into that synctask's backtrace at the time
it yielded away:

(gdb) set $rsp = ((struct synctask *) 0xaddre55)->ctx.uc_mcontext.gregs[15]
(gdb) set $pc = ((struct synctask *) 0xaddre55)->ctx.uc_mcontext.gregs[16]
(gdb) bt

Note that you are essentially "sacrificing" the current thread in order to
inspect the backtrace. So to be safe, you would want to

a) save the original $rsp and $pc values
b) NOT execute "continue" after setting the synctask's saved values.
Instead just inspect the stack with commands like "bt", "frame" etc. and
revert back the orig $pc and $rsp variables.

4. The above step is for x86_64. For other architectures you will need to
find out the appropriate names of registers in gdb for the stack pointer
and program counter, and also the index number of those saved register
values inside gregs[].

For i386:

(gdb) set $esp = ((struct synctask *) 0xaddre55)->ctx.uc_mcontext.gregs[7]
(gdb) set $pc = ((struct synctask *) 0xaddre55)->ctx.uc_mcontext.gregs[14]

For your architecture, you will need to confirm by inspecting
/usr/include/sys/ucontext.h for the actual offsets of the saved register
values.

5. Note that this procedure is not really a "complete context switch". For
a full context switch you will need to load all the saved register values
necessary. This is because the compiler may
chose to maintain variables completely in the registers and not on the
stack - more so common in x86_64 than i386. Compiling with -O0 should
"improve the situation" most of the times, but not a guarantee. You can
load the other registers from the saved values too (look up the appropriate
index into gregs[] for the value from /usr/include/sys/ucontext.h), but
remember to save the current register values if you care about restoring
and resuming the stalled process after inspection.

Happy hacking!
Avati
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20130504/24534b94/attachment-0001.html>