[Gluster-devel] client crash report

[Emmanuel Dreyfus]

Pavan T C <tcp at gluster.com> wrote:

> Might be obvious, but did you see what address caused the seg fault?
> You could do that by disassembling code around  0xbbb63960.
> In your gdb prompt, try:

(gdb) x/12i pthread_mutex_lock
0xbbb63940 <pthread_mutex_lock>:        push   %esi
0xbbb63941 <pthread_mutex_lock+1>:      push   %ebx
0xbbb63942 <pthread_mutex_lock+2>:      
    call   0xbbb63947 <pthread_mutex_lock+7>
0xbbb63947 <pthread_mutex_lock+7>:      pop    %ebx
0xbbb63948 <pthread_mutex_lock+8>:      add    $0x4829,%ebx
0xbbb6394e <pthread_mutex_lock+14>:     mov    0xc(%esp),%ecx
0xbbb63952 <pthread_mutex_lock+18>:     mov    %esp,%esi
0xbbb63954 <pthread_mutex_lock+20>:     mov    0xffffffa0(%ebx),%edx
0xbbb6395a <pthread_mutex_lock+26>:     and    (%edx),%esi
0xbbb6395c <pthread_mutex_lock+28>:     xor    %edx,%edx
0xbbb6395e <pthread_mutex_lock+30>:     mov    %edx,%eax
0xbbb63960 <pthread_mutex_lock+32>:     lock cmpxchg %esi,0xc(%ecx)

(gdb) info reg esi ecx
esi            0xb9200000       -1189085184
ecx            0x313032a3       825242275

ecx holds an invalid address. 
(gdb) x/10w $ecx
0x313032a3:     Cannot access memory at address 0x313032a3


> Also print the ioc_inode structure that is passed to ioc_inode_destroy 
> and see if the values of its members make sense. You might have NULL 
> values where you don't expect them. That might throw some light on the 
> root cause.

(gdb) print *((ioc_inode_t *)0xb5af7100)
$2 = {table = 0x3130325b, ia_size = 2321940053660544305, cache = {
    page_table = 0x323a3331, page_lru = {next = 0x39333a31, 
      prev = 0x3437322e}, mtime = 1563440438, mtime_nsec = 1528846112, tv = {
      tv_sec = 1702065510, tv_usec = 1769103917}}, inode_list = {
    next = 0x2e656764, prev = 0x31353a63}, inode_lru = {next = 0x75663a31, 
    prev = 0x675f6573}, waitq = 0x74617465, inode_lock = {
    ptm_magic = 542995060, ptm_errorcheck = 48 '0', ptm_pad1 = "-gl", 
    ptm_interlock = 117 'u', ptm_pad2 = "ste", ptm_owner = 0x2d736672, 
    ptm_waiters = 0x65737566, ptm_recursed = 8250, ptm_spare2 = 0x0}, 
  weight = 1}

table is invalid, and  pthread_mutex_lock() is called with table->table_lock

(gdb) print *((ioc_inode_t *)0xb5af7100)->table
Cannot access memory at address 0x3130325b

But in fact that looks like a string, and indeed...
(gdb) x/1s 0xb5af7100
0xb5af7100:      "[2011-08-09 13:21:39.274650] W
[fuse-bridge.c:511:fuse_getattr] 0-glusterfs-fuse: "

Wow, there is something seriously rotten.

-- 
Emmanuel Dreyfus
http://hcpnet.free.fr/pubz
manu at netbsd.org