[Gluster-devel] Gluster3.2.5 crash

Tomoaki Sato tsato at valinux.co.jp
Mon Jan 30 01:13:50 UTC 2012 

Hi,

3 gluster processes crashed in my scalable NAS environment. 
Any insight into this is much appreciated.
please find attached log files & configureation files.
Thanks,

tomo sato
--
# gluster volume info

Volume Name: two
Type: Distribute
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: two-1-private:/mnt/brick
Brick2: two-2-private:/mnt/brick

1/3:

# strings /core.5606 | head
CORE
d-ahead 
CORE
glusterfs
/opt/glusterfs/3.2.5/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc
CORE
CORE
CORE
CORE
CORE
# gdb /opt/glusterfs/3.2.5/sbin/glusterfs /core.5606
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/glusterfs/3.2.5/sbin/glusterfs...(no debugging symbols found)...done.
[New Thread 5627]
[New Thread 5633]
[New Thread 5607]
[New Thread 5606]
Reading symbols from /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/read-ahead.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/read-ahead.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/io-cache.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/io-cache.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/quick-read.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/quick-read.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/debug/io-stats.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/debug/io-stats.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/nfs/server.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/nfs/server.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff92bfc000
Core was generated by `/opt/glusterfs/3.2.5/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc'.
Program terminated with signal 11, Segmentation fault.
#0  ioc_writev_cbk (frame=0x2b1d5583555c, cookie=<value optimized out>, this=0x1eb230b0, op_ret=-1, op_errno=9, prebuf=0x0, postbuf=0x0) at io-cache.c:1193
1193	io-cache.c: No such file or directory.
	in io-cache.c
(gdb) bt
#0  ioc_writev_cbk (frame=0x2b1d5583555c, cookie=<value optimized out>, this=0x1eb230b0, op_ret=-1, op_errno=9, prebuf=0x0, postbuf=0x0) at io-cache.c:1193
#1  0x00002aaaab12a0b9 in ra_writev_cbk (frame=0x2b1d55835600, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=9, prebuf=0x0, 
    postbuf=0x0) at read-ahead.c:685
#2  0x00002aaaaaf19096 in wb_writev_cbk (frame=0x2b1d558356a4, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=116, 
    prebuf=0x0, postbuf=0x0) at write-behind.c:2055
#3  0x00002aaaaacf744f in dht_writev_cbk (frame=0x2b1d55835748, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=116, 
    prebuf=0x0, postbuf=0x0) at dht-common.c:2685
#4  0x00002aaaaaac815d in client3_1_writev (frame=0x2b1d558357ec, this=0x1eb1f1b0, data=<value optimized out>) at client3_1-fops.c:3623
#5  0x00002aaaaaab964b in client_writev (frame=0x2b1d558357ec, this=0x1eb1f1b0, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, off=420155392, 
    iobref=0x2aaab0019da0) at client.c:817
#6  0x00002aaaaacfc690 in dht_writev (frame=0x2b1d55835748, this=<value optimized out>, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, off=420155392, 
    iobref=0x2aaab0019da0) at dht-common.c:2721
#7  0x00002aaaaaf1f24a in wb_writev (frame=0x2b1d558356a4, this=0x1eb20e90, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, offset=420155392, 
    iobref=0x2aaab0019da0) at write-behind.c:2134
#8  0x00002aaaab129e57 in ra_writev (frame=0x2b1d55835600, this=0x1eb21f60, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, offset=420155392, 
    iobref=0x2aaab0019da0) at read-ahead.c:718
#9  0x00002aaaab3364f0 in ioc_writev (frame=0x2b1d5583555c, this=0x1eb230b0, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, offset=420155392, 
    iobref=0x2aaab0019da0) at io-cache.c:1237
#10 0x00002aaaab54eb53 in qr_writev (frame=0x2b1d558354b8, this=0x1eb24100, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, off=420155392, 
    iobref=0x2aaab0019da0) at quick-read.c:1504
#11 0x00002aaaab75ed0a in io_stats_writev (frame=0x2b1d55835414, this=0x1eb251d0, fd=0x2aaaad1672ec, vector=0x2aaaad3c9a68, count=1, offset=420155392, 
    iobref=0x2aaab0019da0) at io-stats.c:1982
#12 0x00002aaaab981dcc in nfs_fop_write (nfsx=<value optimized out>, xl=0x1eb251d0, nfu=<value optimized out>, fd=0x2aaaad1672ec, 
    srciob=<value optimized out>, vector=0x2aaaad3c9a68, count=1, offset=420155392, cbk=0x2aaaab9956a0 <nfs3svc_write_cbk>, local=0x2aaaad3c9698)
    at nfs-fops.c:1260
#13 0x00002aaaab9900c9 in __nfs3_write_resume (cs=0x2aaaad3c9698) at nfs3.c:2025
#14 0x00002aaaab993d95 in nfs3_write_resume (carg=0x2b1d5583555c) at nfs3.c:2045
#15 0x00002aaaab9a4004 in nfs3_file_open_and_resume (cs=0x2aaaad3c9698, resume=<value optimized out>) at nfs3-helpers.c:2136
#16 0x00002aaaab993ccc in nfs3_write_open_resume (carg=0x2b1d5583555c) at nfs3.c:2072
#17 0x00002aaaab99e794 in nfs3_fh_resolve_inode_done (cs=0x2aaaad3c9698, inode=<value optimized out>) at nfs3-helpers.c:2421
#18 0x00002aaaab99e813 in nfs3_fh_resolve_inode (cs=0x2aaaad3c9698) at nfs3-helpers.c:3018
#19 0x00002aaaab99e8e5 in nfs3_fh_resolve_resume (cs=0x2b1d5583555c) at nfs3-helpers.c:3050
#20 0x00002aaaab99e9a8 in nfs3_fh_resolve_root (cs=0x2aaaad3c9698) at nfs3-helpers.c:3104
#21 0x00002aaaab99b506 in nfs3_write (req=0x1ed76ee0, fh=0x40320fc0, offset=420155392, count=65536, stable=UNSTABLE, payload=..., iob=0x2aaab00808c0)
    at nfs3.c:2119
#22 0x00002aaaab99b780 in nfs3svc_write_vec (req=0x1ed76ee0, iob=0x2aaab00808c0) at nfs3.c:2207
#23 0x00002aaaab9a9e5a in nfs_rpcsvc_record_vectored_call_actor (conn=0x1eb27bec) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2326
#24 0x00002aaaab9aa72e in nfs_rpcsvc_update_vectored_state (conn=0x1eb27bec) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2385
#25 0x00002aaaab9aab18 in nfs_rpcsvc_record_update_state (conn=0x1eb27bec, dataread=52688) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2481
#26 0x00002aaaab9ab4ff in nfs_rpcsvc_conn_data_handler (fd=<value optimized out>, idx=514994352, data=0x1eb27bec, poll_in=1, poll_out=9, poll_err=0)
    at ../../../../xlators/nfs/lib/src/rpcsvc.c:2701
#27 0x00002b1d5491c9a1 in event_dispatch_epoll_handler (event_pool=0x1eb41980) at event.c:794
#28 event_dispatch_epoll (event_pool=0x1eb41980) at event.c:856
#29 0x00002aaaab9a90c2 in nfs_rpcsvc_stage_proc (arg=<value optimized out>) at ../../../../xlators/nfs/lib/src/rpcsvc.c:64
#30 0x00000035e760673d in start_thread () from /lib64/libpthread.so.0
#31 0x00000035e72d40cd in clone () from /lib64/libc.so.6
(gdb) 

2/3:

# strings /core.5629 | head -20
 ;Z3+
0;Z3+
p<Z3+
p\Z3+
\Z3+
CORE
\Z3+
0E_a
\Z3+
CORE
glusterd
/opt/glusterfs/3.2.5/sbin/glusterd 
CORE
CORE
CORE
\Z3+
CORE
CORE
\Z3+
CORE
# gdb /opt/glusterfs/3.2.5/sbin/glusterd /core.5629
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/glusterfs/3.2.5/sbin/glusterd...(no debugging symbols found)...done.
[New Thread 5629]
[New Thread 5638]
[New Thread 5630]
Reading symbols from /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/mgmt/glusterd.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/mgmt/glusterd.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff617f5000
Core was generated by `/opt/glusterfs/3.2.5/sbin/glusterd'.
Program terminated with signal 11, Segmentation fault.
#0  glusterd_friend_sm () at glusterd-sm.c:934
934	glusterd-sm.c: No such file or directory.
	in glusterd-sm.c
(gdb) bt
#0  glusterd_friend_sm () at glusterd-sm.c:934
#1  0x00002aaaaaac7a04 in glusterd_peer_rpc_notify (rpc=0x519dc50, mydata=0x519e4b0, event=RPC_CLNT_DISCONNECT, data=<value optimized out>)
    at glusterd-handler.c:3883
#2  0x00002b335a1ab8a7 in rpc_clnt_start (rpc=<value optimized out>) at rpc-clnt.c:1022
#3  0x00002aaaaaac874f in glusterd_rpc_create (rpc=0x7fff615f4630, options=<value optimized out>, notify_fn=0x2aaaaaac7850 <glusterd_peer_rpc_notify>, 
    notify_data=0x519e4b0) at glusterd-handler.c:3279
#4  0x00002aaaaaac8ad7 in glusterd_friend_add (hoststr=0x519e420 "two-2-private", port=0, state=GD_FRIEND_STATE_BEFRIENDED, uuid=<value optimized out>, 
    rpc=0x519dc50, friend=0x7fff615f6710, restore=_gf_true, args=0x7fff615f66d0) at glusterd-handler.c:3378
#5  0x00002aaaaab0d4c2 in glusterd_store_retrieve_peers (this=<value optimized out>) at glusterd-store.c:2248
#6  0x00002aaaaab0edd8 in glusterd_restore () at glusterd-store.c:2310
#7  0x00002aaaaaac4312 in init (this=0x5190c20) at glusterd.c:695
#8  0x00002b3359f46488 in __xlator_init (xl=0x5190c20) at xlator.c:1418
#9  xlator_init (xl=0x5190c20) at xlator.c:1441
#10 0x00002b3359f75501 in glusterfs_graph_init (graph=<value optimized out>) at graph.c:345
#11 0x00002b3359f75688 in glusterfs_graph_activate (graph=0x518c900, ctx=0x518b010) at graph.c:524
#12 0x0000000000404033 in glusterfs_process_volfp ()
#13 0x000000000040424b in glusterfs_volumes_init ()
#14 0x0000000000405299 in main ()
(gdb) 

3/3:

# strings /core.6124 | head
CORE
d-ahead 
CORE
glusterfs
/opt/glusterfs/3.2.5/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc
CORE
CORE
CORE
PZm	
CORE
# gdb /opt/glusterfs/3.2.5/sbin/glusterfs /core.6124
GNU gdb (GDB) Red Hat Enterprise Linux (7.0.1-37.el5_7.1)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/glusterfs/3.2.5/sbin/glusterfs...(no debugging symbols found)...done.
[New Thread 6173]
[New Thread 6230]
[New Thread 6125]
[New Thread 6124]
Reading symbols from /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libglusterfs.so.0
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfrpc.so.0
Reading symbols from /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/libgfxdr.so.0
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/protocol/client.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/cluster/distribute.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/write-behind.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/read-ahead.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/read-ahead.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/io-cache.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/io-cache.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/quick-read.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/performance/quick-read.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/debug/io-stats.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/debug/io-stats.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/nfs/server.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/xlator/nfs/server.so
Reading symbols from /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so...done.
Loaded symbols for /opt/glusterfs/3.2.5/lib64/glusterfs/3.2.5/rpc-transport/socket.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_dns.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1

warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fffca144000
Core was generated by `/opt/glusterfs/3.2.5/sbin/glusterfs -f /etc/glusterd/nfs/nfs-server.vol -p /etc'.
Program terminated with signal 11, Segmentation fault.
#0  ioc_writev_cbk (frame=0x2b299b7b477c, cookie=<value optimized out>, this=0x968e0b0, op_ret=-1, op_errno=9, prebuf=0x0, postbuf=0x0) at io-cache.c:1193
1193	io-cache.c: No such file or directory.
	in io-cache.c
(gdb) bt
#0  ioc_writev_cbk (frame=0x2b299b7b477c, cookie=<value optimized out>, this=0x968e0b0, op_ret=-1, op_errno=9, prebuf=0x0, postbuf=0x0) at io-cache.c:1193
#1  0x00002aaaab12a0b9 in ra_writev_cbk (frame=0x2b299b7b4820, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=9, prebuf=0x0, 
    postbuf=0x0) at read-ahead.c:685
#2  0x00002aaaaaf19096 in wb_writev_cbk (frame=0x2b299b7b48c4, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=116, 
    prebuf=0x0, postbuf=0x0) at write-behind.c:2055
#3  0x00002aaaaacf744f in dht_writev_cbk (frame=0x2b299b7b4968, cookie=<value optimized out>, this=<value optimized out>, op_ret=-1, op_errno=116, 
    prebuf=0x0, postbuf=0x0) at dht-common.c:2685
#4  0x00002aaaaaac815d in client3_1_writev (frame=0x2b299b7b4a0c, this=0x968a1b0, data=<value optimized out>) at client3_1-fops.c:3623
#5  0x00002aaaaaab964b in client_writev (frame=0x2b299b7b4a0c, this=0x968a1b0, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, off=472158208, 
    iobref=0x2aaab4316420) at client.c:817
#6  0x00002aaaaacfc690 in dht_writev (frame=0x2b299b7b4968, this=<value optimized out>, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, off=472158208, 
    iobref=0x2aaab4316420) at dht-common.c:2721
#7  0x00002aaaaaf1f24a in wb_writev (frame=0x2b299b7b48c4, this=0x968be90, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, offset=472158208, 
    iobref=0x2aaab4316420) at write-behind.c:2134
#8  0x00002aaaab129e57 in ra_writev (frame=0x2b299b7b4820, this=0x968cf60, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, offset=472158208, 
    iobref=0x2aaab4316420) at read-ahead.c:718
#9  0x00002aaaab3364f0 in ioc_writev (frame=0x2b299b7b477c, this=0x968e0b0, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, offset=472158208, 
    iobref=0x2aaab4316420) at io-cache.c:1237
#10 0x00002aaaab54eb53 in qr_writev (frame=0x2b299b7b46d8, this=0x968f100, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, off=472158208, 
    iobref=0x2aaab4316420) at quick-read.c:1504
#11 0x00002aaaab75ed0a in io_stats_writev (frame=0x2b299b7b4634, this=0x96901d0, fd=0x2aaaad166a64, vector=0x2aaab431a970, count=1, offset=472158208, 
    iobref=0x2aaab4316420) at io-stats.c:1982
#12 0x00002aaaab981dcc in nfs_fop_write (nfsx=<value optimized out>, xl=0x96901d0, nfu=<value optimized out>, fd=0x2aaaad166a64, 
    srciob=<value optimized out>, vector=0x2aaab431a970, count=1, offset=472158208, cbk=0x2aaaab9956a0 <nfs3svc_write_cbk>, local=0x2aaab431a5a0)
    at nfs-fops.c:1260
#13 0x00002aaaab9900c9 in __nfs3_write_resume (cs=0x2aaab431a5a0) at nfs3.c:2025
#14 0x00002aaaab993d95 in nfs3_write_resume (carg=0x2b299b7b477c) at nfs3.c:2045
#15 0x00002aaaab9a4004 in nfs3_file_open_and_resume (cs=0x2aaab431a5a0, resume=<value optimized out>) at nfs3-helpers.c:2136
#16 0x00002aaaab993ccc in nfs3_write_open_resume (carg=0x2b299b7b477c) at nfs3.c:2072
#17 0x00002aaaab99e794 in nfs3_fh_resolve_inode_done (cs=0x2aaab431a5a0, inode=<value optimized out>) at nfs3-helpers.c:2421
#18 0x00002aaaab99e813 in nfs3_fh_resolve_inode (cs=0x2aaab431a5a0) at nfs3-helpers.c:3018
#19 0x00002aaaab99e8e5 in nfs3_fh_resolve_resume (cs=0x2b299b7b477c) at nfs3-helpers.c:3050
#20 0x00002aaaab99e9a8 in nfs3_fh_resolve_root (cs=0x2aaab431a5a0) at nfs3-helpers.c:3104
#21 0x00002aaaab99b506 in nfs3_write (req=0x2aaab41dbb34, fh=0x4042cfc0, offset=472158208, count=65536, stable=UNSTABLE, payload=..., iob=0x2aaab42b7f50)
    at nfs3.c:2119
#22 0x00002aaaab99b780 in nfs3svc_write_vec (req=0x2aaab41dbb34, iob=0x2aaab42b7f50) at nfs3.c:2207
#23 0x00002aaaab9a9e5a in nfs_rpcsvc_record_vectored_call_actor (conn=0x9692cb8) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2326
#24 0x00002aaaab9aa72e in nfs_rpcsvc_update_vectored_state (conn=0x9692cb8) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2385
#25 0x00002aaaab9aab18 in nfs_rpcsvc_record_update_state (conn=0x9692cb8, dataread=6352) at ../../../../xlators/nfs/lib/src/rpcsvc.c:2481
#26 0x00002aaaab9ab4ff in nfs_rpcsvc_conn_data_handler (fd=<value optimized out>, idx=157868208, data=0x9692cb8, poll_in=1, poll_out=9, poll_err=0)
    at ../../../../xlators/nfs/lib/src/rpcsvc.c:2701
#27 0x00002b299a8659a1 in event_dispatch_epoll_handler (event_pool=0x96ac980) at event.c:794
#28 event_dispatch_epoll (event_pool=0x96ac980) at event.c:856
#29 0x00002aaaab9a90c2 in nfs_rpcsvc_stage_proc (arg=<value optimized out>) at ../../../../xlators/nfs/lib/src/rpcsvc.c:64
#30 0x00000035e760673d in start_thread () from /lib64/libpthread.so.0
#31 0x00000035e72d40cd in clone () from /lib64/libc.so.6
(gdb) 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: log.tar.gz
Type: application/x-gzip
Size: 179559 bytes
Desc: not available
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20120130/eb011d99/attachment-0003.gz>

More information about the Gluster-devel mailing list