[Gluster-devel] Segfault in glusterfs client

Anand Avati avati at zresearch.com
Thu Jul 12 11:36:27 UTC 2007 

Is there logs at the time when glusterfs loads increased? are there any
steps you have found to reproduce? also have you noticed any network traffic
happening (tcpdump port 6996) when glusterfs load had shot up?

thanks,
avati

2007/7/12, Pooya Woodcock <pooya at packetcloud.net>:
>
> 4 daemons, 4 clients. My configs are in the "no element to prune"
> thread on July 10.
> I am running apache with html and php.
>
> I also just noticed that the load on nodes 2 and 4 of my servers went
> up to over 30 and I am unable to stop apache. I had to kill gluserfs
> and glusterfsd on those two nodes.
>
> I can provide you with a temporary remote login if that would help.
> -pooya
>
> On Jul 12, 2007, at 1:34 AM, Amar S. Tumballi wrote:
>
> > Hi Pooya,
> >  let us know your setup.. ie, what application you were running and
> > howmany clients.
> >
> > -amar
> >
> > On 7/12/07, Pooya Woodcock < pooya at packetcloud.net> wrote:Also on
> > another node, I got this segfault. This one was first then I
> > got the other one on node4 that I emailed eearlier.
> >
> > [root at node02 ~]# gdb glusterfs -c /core.13358
> > GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh)
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and
> > you are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show warranty" for
> > details.
> > This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
> > libthread_db library "/lib64/tls/libthread_db.so.1".
> >
> > Core was generated by `
> > [glusterfs]
> >        '.
> > Program terminated with signal 11, Segmentation fault.
> > Reading symbols from /usr/local/lib/libglusterfs.so.0...done.
> > Loaded symbols for /usr/local/lib/libglusterfs.so.0
> > Reading symbols from /usr/local/lib/libibverbs.so.1...done.
> > Loaded symbols for /usr/local/lib/libibverbs.so.1
> > Reading symbols from /usr/local/lib/libfuse.so.2...done.
> > Loaded symbols for /usr/local/lib/libfuse.so.2
> > Reading symbols from /lib64/libdl.so.2...done.
> > Loaded symbols for /lib64/libdl.so.2
> > Reading symbols from /lib64/tls/libpthread.so.0...done.
> > Loaded symbols for /lib64/tls/libpthread.so.0
> > Reading symbols from /lib64/tls/libc.so.6...done.
> > Loaded symbols for /lib64/tls/libc.so.6
> > Reading symbols from /usr/lib64/libsysfs.so.1...done.
> > Loaded symbols for /usr/lib64/libsysfs.so.1
> > Reading symbols from /lib64/tls/librt.so.1...done.
> > Loaded symbols for /lib64/tls/librt.so.1
> > Reading symbols from /lib64/libselinux.so.1...done.
> > Loaded symbols for /lib64/libselinux.so.1
> > Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> > Loaded symbols for /lib64/ld-linux-x86-64.so.2
> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > protocol/client.so...done.
> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> > protocol/client.so
> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > cluster/afr.so...done.
> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > cluster/afr.so
> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > cluster/unify.so...done.
> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > cluster/unify.so
> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> > performance/io-threads.so...done.
> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > performance/io-threads.so
> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > performance/read-ahead.so...done.
> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > performance/read-ahead.so
> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/transport/
> > tcp/client.so...done.
> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/transport/
> > tcp/client.so
> > Reading symbols from /usr/local/lib/glusterfs/1.3.0-pre5.2/scheduler/
> > rr.so...done.
> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/scheduler/
> > rr.so
> > Reading symbols from /lib64/libgcc_s.so.1...done.
> > Loaded symbols for /lib64/libgcc_s.so.1
> > #0  unify_open (frame=0x2a960d3ed0, this=0x514b70, loc=0x2a960cd178,
> > flags=32770) at unify.c:956
> > 956         loc_t tmp_loc = {
> > (gdb) bt
> > #0  unify_open (frame=0x2a960d3ed0, this=0x514b70, loc=0x2a960cd178,
> > flags=32770) at unify.c:956
> > #1  0x0000002a95bc8121 in iot_open (frame=0x2a960d2190,
> > this=0x515700, loc=0x2a960cd178, flags=32770) at io-threads.c:88
> > #2  0x0000002a95ccc2a7 in ra_open (frame=0x2a960d1800, this=0x5160a0,
> > loc=0x2a960cd178, flags=32770) at read-ahead.c:207
> > #3  0x0000000000405898 in fuse_open (req=Variable "req" is not
> > available.
> > ) at fuse-bridge.c:1017
> > #4  0x0000002a95783ffe in do_open (req=Variable "req" is not
> > available.
> > ) at fuse_lowlevel.c:586
> > #5  0x0000000000407b52 in fuse_transport_notify (xl=Variable "xl" is
> > not available.
> > ) at fuse-bridge.c:1937
> > #6  0x0000002a95563e86 in sys_epoll_iteration (ctx=Variable "ctx" is
> > not available.
> > ) at epoll.c:54
> > #7  0x0000000000402f4e in main (argc=7, argv=0x7fbffff9d8) at
> > glusterfs.c:348
> >
> >
> >
> >
> > On Jul 11, 2007, at 9:58 AM, Pooya Woodcock wrote:
> >
> > > Hi Avati,
> > > Confirmed fixed. Thanks, running great so far.
> > > Pooya
> > >
> > > On Jul 10, 2007, at 11:50 PM, Anand Avati wrote:
> > >
> > >> Pooya,
> > >>  this bug is fixed in patch 295. can you please confirm if some
> > >> operations were performed when one (or more) server was down to
> > >> reproduce this bug?
> > >>
> > >> thanks,
> > >> avati
> > >>
> > >> 2007/7/11, Pooya Woodcock < pooya at packetcloud.net>:This is on
> > >> patch 293 in 2.5 mainline.
> > >> -Pooya
> > >>
> > >> On Jul 10, 2007, at 7:40 PM, Amar S. Tumballi wrote:
> > >>
> > >> > Hi Pooya,
> > >> >  Thanks for the bt, but can you tell us which version (or
> > patch in
> > >> > tla) are you using?
> > >> >
> > >> > -amar
> > >> >
> > >> > On 7/11/07, Pooya Woodcock < pooya at packetcloud.net> wrote:Here is
> > >> > gdb output on the core..
> > >> >
> > >> > [root at node02 ~]# gdb glusterfs -c /core.6612
> > >> > GNU gdb Red Hat Linux (6.3.0.0-1.132.EL4rh)
> > >> > Copyright 2004 Free Software Foundation, Inc.
> > >> > GDB is free software, covered by the GNU General Public License,
> > >> and
> > >> > you are
> > >> > welcome to change it and/or distribute copies of it under certain
> > >> > conditions.
> > >> > Type "show copying" to see the conditions.
> > >> > There is absolutely no warranty for GDB.  Type "show warranty"
> > for
> > >> > details.
> > >> > This GDB was configured as "x86_64-redhat-linux-gnu"...Using host
> > >> > libthread_db library "/lib64/tls/libthread_db.so.1".
> > >> >
> > >> > Core was generated by `
> > >> > [glusterfs]
> > >> >        '.
> > >> > Program terminated with signal 11, Segmentation fault.
> > >> > Reading symbols from /usr/local/lib/libglusterfs.so.0...done.
> > >> > Loaded symbols for /usr/local/lib/libglusterfs.so.0
> > >> > Reading symbols from /usr/local/lib/libibverbs.so.1...done.
> > >> > Loaded symbols for /usr/local/lib/libibverbs.so.1
> > >> > Reading symbols from /usr/local/lib/libfuse.so.2...done.
> > >> > Loaded symbols for /usr/local/lib/libfuse.so.2
> > >> > Reading symbols from /lib64/libdl.so.2...done.
> > >> > Loaded symbols for /lib64/libdl.so.2
> > >> > Reading symbols from /lib64/tls/libpthread.so.0...done.
> > >> > Loaded symbols for /lib64/tls/libpthread.so.0
> > >> > Reading symbols from /lib64/tls/libc.so.6...done.
> > >> > Loaded symbols for /lib64/tls/libc.so.6
> > >> > Reading symbols from /usr/lib64/libsysfs.so.1...done.
> > >> > Loaded symbols for /usr/lib64/libsysfs.so.1
> > >> > Reading symbols from /lib64/tls/librt.so.1...done.
> > >> > Loaded symbols for /lib64/tls/librt.so.1
> > >> > Reading symbols from /lib64/libselinux.so.1...done.
> > >> > Loaded symbols for /lib64/libselinux.so.1
> > >> > Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
> > >> > Loaded symbols for /lib64/ld- linux-x86-64.so.2
> > >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> > xlator/
> > >> > protocol/client.so...done.
> > >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > >> > protocol/client.so
> > >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> > xlator/
> > >> > cluster/afr.so...done.
> > >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> > >> > cluster/afr.so
> > >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> > xlator/
> > >> > cluster/unify.so...done.
> > >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > >> > cluster/unify.so
> > >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> > xlator/
> > >> > performance/io-threads.so...done.
> > >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> > >> > performance/io- threads.so
> > >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> > xlator/
> > >> > performance/read-ahead.so...done.
> > >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/xlator/
> > >> > performance/read- ahead.so
> > >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> > >> transport/
> > >> > tcp/client.so...done.
> > >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/
> > transport/
> > >> > tcp/client.so
> > >> > Reading symbols from /usr/local/lib/glusterfs/1.3.0- pre5.2/
> > >> scheduler/
> > >> > rr.so...done.
> > >> > Loaded symbols for /usr/local/lib/glusterfs/1.3.0-pre5.2/
> > scheduler/
> > >> > rr.so
> > >> > Reading symbols from /lib64/libgcc_s.so.1...done.
> > >> > Loaded symbols for /lib64/libgcc_s.so.1
> > >> > #0  inode_unref (inode=0x0) at inode.c:403
> > >> > 403       inode_table_t *table = inode->table;
> > >> >
> > >> > (gdb) backtrace
> > >> > #0  inode_unref (inode=0x0) at inode.c:403
> > >> > #1  0x0000002a959ad6bc in afr_forget (frame=dwarf2_read_address:
> > >> > Corrupted DWARF expression.
> > >> > ) at afr.c:468
> > >> > #2  0x0000002a95564a44 in __destroy_inode (inode=0x2a9619bc10) at
> > >> > inode.c:250
> > >> > #3  0x0000002a95564cfc in inode_unref (inode=0x2a9619bc10) at
> > >> inode.c:
> > >> > 413
> > >> > #4  0x0000002a95abc289 in unify_forget (frame=Variable "frame"
> > >> is not
> > >> > available.
> > >> > ) at unify.c:407
> > >> > #5  0x0000002a95564a44 in __destroy_inode (inode=0x2a9619b8b0) at
> > >> > inode.c:250
> > >> > #6  0x0000002a95564cfc in inode_unref (inode=0x2a9619b8b0) at
> > >> inode.c:
> > >> > 413
> > >> > #7  0x0000000000407370 in fuse_forget_notify (frame=Variable
> > >> "frame"
> > >> > is not available.
> > >> > ) at fuse-bridge.c:1652
> > >> > #8  0x0000002a95564a44 in __destroy_inode (inode=0x2a9608f0c0) at
> > >> > inode.c:250
> > >> > #9  0x0000002a95564cfc in inode_unref (inode=0x2a9608f0c0) at
> > >> inode.c:
> > >> > 413
> > >> > #10 0x0000000000403993 in fuse_forget (req=0x2a9607de50,
> > >> ino=Variable
> > >> > "ino" is not available.
> > >> > ) at fuse-bridge.c:363
> > >> > #11 0x0000000000407b52 in fuse_transport_notify (xl=Variable
> > >> "xl" is
> > >> > not available.
> > >> > ) at fuse-bridge.c:1937
> > >> > #12 0x0000002a95563e86 in sys_epoll_iteration (ctx=Variable
> > >> "ctx" is
> > >> > not available.
> > >> > ) at epoll.c:54
> > >> > #13 0x0000000000402f4e in main (argc=7, argv=0x7fbffffa08) at
> > >> > glusterfs.c:348
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> > On Jul 10, 2007, at 1:36 PM, Pooya Woodcock wrote:
> > >> >
> > >> > > 2007-07-10 13:30:21 D [ fuse-bridge.c:338:fuse_lookup]
> > glusterfs-
> > >> > > fuse: LOOKUP 6832132/sess_3c1f96159c610b755a7def44f036a11b (/
> > >> > > php_sessions/sess_3c1f96159c610b755a
> > >> > > 7def44f036a11b)
> > >> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] ns/inode:
> > >> create
> > >> > > inode(6832716)
> > >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] ns/inode:
> > >> > > activating inode(6832716), lru=0/1000
> > >> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] unify1/
> > inode:
> > >> > > create inode(6832716)
> > >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] unify1/
> > inode:
> > >> > > activating inode(6832716), lru=0/1000
> > >> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] brick3-afr/
> > >> > > inode: create inode(10535464)
> > >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] brick3-afr/
> > >> > > inode: activating inode(10535464), lru=0/1000
> > >> > > 2007-07-10 13:30:21 D [inode.c:511:__create_inode] afr3/inode:
> > >> > > create inode(10535464)
> > >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] afr3/inode:
> > >> > > activating inode(10535464), lru=0/1000
> > >> > > 2007-07-10 13:30:21 D [ fuse-bridge.c:288:fuse_entry_cbk]
> > >> glusterfs-
> > >> > > fuse: ENTRY => 6832716
> > >> > > 2007-07-10 13:30:21 D [ inode.c:511:__create_inode] fuse/inode:
> > >> > > create inode(6832716)
> > >> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
> > >> > > activating inode(6832716), lru=454/1024
> > >> > > 2007-07-10 13:30:21 D [ inode.c:332:__passive_inode] fuse/
> > inode:
> > >> > > passivating inode(6832716), lru=455/1024
> > >> > > 2007-07-10 13:30:21 D [inode.c:332:__passive_inode] fuse/inode:
> > >> > > passivating inode(6832132), lru=456/1024
> > >> > > 2007-07-10 13:30:21 D [ fuse-bridge.c:775:fuse_unlink]
> > glusterfs-
> > >> > > fuse: UNLINK 6832132/sess_3c1f96159c610b755a7def44f036a11b
> > >> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
> > >> > > activating inode(6832716), lru=455/1024
> > >> > > 2007-07-10 13:30:21 D [inode.c:302:__active_inode] fuse/inode:
> > >> > > activating inode(6832132), lru=454/1024
> > >> > > 2007-07-10 13:30:21 D [inode.c :332:__passive_inode] fuse/
> > inode:
> > >> > > passivating inode(6832716), lru=455/1024
> > >> > > 2007-07-10 13:30:21 D [inode.c:332:__passive_inode] fuse/inode:
> > >> > > passivating inode(6832132), lru=456/1024
> > >> > > 2007-07-10 13:30:21 D [ inode.c:302:__active_inode] fuse/inode:
> > >> > > activating inode(6832716), lru=455/1024
> > >> > > 2007-07-10 13:30:21 C [common-utils.c:208:gf_print_trace]
> > debug-
> > >> > > backtrace: Got signal (11), printing backtrace
> > >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/libglusterfs.so.0(gf_print_trace
> > +0x22)
> > >> > > [0x2a95562782]
> > >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /lib64/tls/libc.so.6 [0x37fbc2e2b0]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x16)
> > >> > > [0x2a95564bc6]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/glusterfs/1.3.0-pre5.2 /xlator/
> > cluster/
> > >> > > afr.so [0x2a959ad6bc]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/libglusterfs.so.0 [0x2a95564a44]
> > >> > > 2007-07-10 13:30:21 C [ common-utils.c :210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x14c)
> > >> > > [0x2a95564cfc]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/glusterfs/1.3.0- pre5.2/xlator/
> > cluster/
> > >> > > unify.so(unify_forget+0x29) [0x2a
> > >> > > 95abc289]
> > >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/libglusterfs.so.0 [0x2a95564a44]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x14c)
> > >> > > [0x2a95564cfc]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace]
> > debug-
> > >> > > backtrace: [glusterfs] [0x407370]
> > >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/libglusterfs.so.0 [0x2a95564a44]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/libglusterfs.so.0(inode_unref+0x14c)
> > >> > > [0x2a95564cfc]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c :210:gf_print_trace]
> > debug-
> > >> > > backtrace: [glusterfs] [0x403993]
> > >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: [glusterfs] [0x407b52]
> > >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /usr/local/lib/libglusterfs.so.0(sys_epoll_iteration
> > >> > > +0xc6) [0x2a95563e86]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: [glusterfs] [0x402f4e]
> > >> > > 2007-07-10 13:30:21 C [ common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: /lib64/tls/libc.so.6(__libc_start_main+0xdb)
> > >> > [0x37fbc1c3fb]
> > >> > > 2007-07-10 13:30:21 C [common-utils.c:210:gf_print_trace]
> > debug-
> > >> > > backtrace: [glusterfs] [0x40290a]
> > >> > >
> > >> > >
> > >> > > _______________________________________________
> > >> > > Gluster-devel mailing list
> > >> > > Gluster-devel at nongnu.org
> > >> > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> > >> >
> > >> >
> > >> >
> > >> > _______________________________________________
> > >> > Gluster-devel mailing list
> > >> > Gluster-devel at nongnu.org
> > >> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Amar Tumballi
> > >> > http://amar.80x25.org
> > >> > [bulde on #gluster/irc.gnu.org]
> > >>
> > >>
> > >>
> > >> _______________________________________________
> > >> Gluster-devel mailing list
> > >> Gluster-devel at nongnu.org
> > >> http://lists.nongnu.org/mailman/listinfo/gluster-devel
> > >>
> > >>
> > >>
> > >> --
> > >> Anand V. Avati
> > >
> > >
> > >
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > Gluster-devel at nongnu.org
> > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> >
> >
> > --
> > Amar Tumballi
> > http://amar.80x25.org
> > [bulde on #gluster/irc.gnu.org]
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>



-- 
Anand V. Avati

More information about the Gluster-devel mailing list