[Gluster-devel] [Gluster-users] Glusterfsd crashed with SIGSEGV
Amar Tumballi Suryanarayan
atumball at redhat.com
Wed Mar 13 04:20:00 UTC 2019
Hi Abhishek,
Few more questions,
> On Tue, Mar 12, 2019 at 10:58 AM ABHISHEK PALIWAL <abhishpaliwal at gmail.com>
> wrote:
>
>> Hi Amar,
>>
>> Below are the requested logs
>>
>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libglusterfs.so
>> not a dynamic executable
>>
>> pabhishe at arn-build3$ldd ./sysroot-destdir/usr/lib64/libgfrpc.so
>> not a dynamic executable
>>
>>
Can you please add a * at the end, so it gets the linked library list from
the actual files (ideally this is a symlink, but I expected it to resolve
like in Fedora).
> root at 128:/# gdb /usr/sbin/glusterd core.1099
>> GNU gdb (GDB) 7.10.1
>> Copyright (C) 2015 Free Software Foundation, Inc.
>> License GPLv3+: GNU GPL version 3 or later <
>> http://gnu.org/licenses/gpl.html>
>> This is free software: you are free to change and redistribute it.
>> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
>> and "show warranty" for details.
>> This GDB was configured as "powerpc64-wrs-linux".
>> Type "show configuration" for configuration details.
>> For bug reporting instructions, please see:
>> <http://www.gnu.org/software/gdb/bugs/>.
>> Find the GDB manual and other documentation resources online at:
>> <http://www.gnu.org/software/gdb/documentation/>.
>> For help, type "help".
>> Type "apropos word" to search for commands related to "word"...
>> Reading symbols from /usr/sbin/glusterd...(no debugging symbols
>> found)...done.
>> [New LWP 1109]
>> [New LWP 1101]
>> [New LWP 1105]
>> [New LWP 1110]
>> [New LWP 1099]
>> [New LWP 1107]
>> [New LWP 1119]
>> [New LWP 1103]
>> [New LWP 1112]
>> [New LWP 1116]
>> [New LWP 1104]
>> [New LWP 1239]
>> [New LWP 1106]
>> [New LWP 1111]
>> [New LWP 1108]
>> [New LWP 1117]
>> [New LWP 1102]
>> [New LWP 1118]
>> [New LWP 1100]
>> [New LWP 1114]
>> [New LWP 1113]
>> [New LWP 1115]
>>
>> warning: Could not load shared library symbols for linux-vdso64.so.1.
>> Do you need "set solib-search-path" or "set sysroot"?
>> [Thread debugging using libthread_db enabled]
>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140
>> --volfile-id gv0.128.224.95.140.tmp-bric'.
>> Program terminated with signal SIGSEGV, Segmentation fault.
>> #0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020,
>> bytes=bytes at entry=36) at malloc.c:3327
>> 3327 {
>> [Current thread is 1 (Thread 0x3fffb1689160 (LWP 1109))]
>> (gdb) bt full
>>
>
This is backtrace of one particular thread. I need output of command
(gdb) thread apply all bt full
Also, considering this is a crash in the malloc library call itself, would
like to know the details of OS, Kernel version and gcc versions.
Regards,
Amar
#0 0x00003fffb76a6d48 in _int_malloc (av=av at entry=0x3fffa8000020,
>> bytes=bytes at entry=36) at malloc.c:3327
>> nb = <optimized out>
>> idx = <optimized out>
>> bin = <optimized out>
>> victim = <optimized out>
>> size = <optimized out>
>> victim_index = <optimized out>
>> remainder = <optimized out>
>> remainder_size = <optimized out>
>> block = <optimized out>
>> bit = <optimized out>
>> map = <optimized out>
>> fwd = <optimized out>
>> bck = <optimized out>
>> errstr = 0x0
>> __func__ = "_int_malloc"
>> #1 0x00003fffb76a93dc in __GI___libc_malloc (bytes=36) at malloc.c:2921
>> ar_ptr = 0x3fffa8000020
>> victim = <optimized out>
>> hook = <optimized out>
>> __func__ = "__libc_malloc"
>> #2 0x00003fffb7764fd0 in x_inline (xdrs=0x3fffb1686d20, len=<optimized
>> out>) at xdr_sizeof.c:89
>> len = 36
>> xdrs = 0x3fffb1686d20
>> #3 0x00003fffb7842488 in .xdr_gfx_iattx () from /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #4 0x00003fffb7842e84 in .xdr_gfx_dirplist () from
>> /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #5 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20,
>> pp=0x3fffa81099f0, size=<optimized out>, proc=<optimized out>) at
>> xdr_ref.c:84
>> loc = 0x3fffa8109aa0 "\265\256\373\200\f\206\361j"
>> stat = <optimized out>
>> #6 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20,
>> objpp=0x3fffa81099f0, obj_size=<optimized out>,
>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at
>> xdr_ref.c:135
>> more_data = 1
>> #7 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from
>> /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #8 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20,
>> pp=0x3fffa8109870, size=<optimized out>, proc=<optimized out>) at
>> xdr_ref.c:84
>> loc = 0x3fffa8109920 "\232\373\377\315\352\325\005\271"
>> stat = <optimized out>
>> #9 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20,
>> objpp=0x3fffa8109870, obj_size=<optimized out>,
>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at
>> xdr_ref.c:135
>> more_data = 1
>> #10 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from
>> /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #11 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20,
>> pp=0x3fffa81096f0, size=<optimized out>, proc=<optimized out>) at
>> xdr_ref.c:84
>> loc = 0x3fffa81097a0 "\241X\372!\216\256=\342"
>> stat = <optimized out>
>> ---Type <return> to continue, or q <return> to quit---
>> #12 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20,
>> objpp=0x3fffa81096f0, obj_size=<optimized out>,
>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at
>> xdr_ref.c:135
>> more_data = 1
>> #13 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from
>> /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #14 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20,
>> pp=0x3fffa8109570, size=<optimized out>, proc=<optimized out>) at
>> xdr_ref.c:84
>> loc = 0x3fffa8109620 "\265\205\003Vu'\002L"
>> stat = <optimized out>
>> #15 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20,
>> objpp=0x3fffa8109570, obj_size=<optimized out>,
>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at
>> xdr_ref.c:135
>> more_data = 1
>> #16 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from
>> /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #17 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20,
>> pp=0x3fffa81093f0, size=<optimized out>, proc=<optimized out>) at
>> xdr_ref.c:84
>> loc = 0x3fffa81094a0 "\200L\027F'\177\366D"
>> stat = <optimized out>
>> #18 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20,
>> objpp=0x3fffa81093f0, obj_size=<optimized out>,
>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at
>> xdr_ref.c:135
>> more_data = 1
>> #19 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from
>> /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #20 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20,
>> pp=0x3fffa8109270, size=<optimized out>, proc=<optimized out>) at
>> xdr_ref.c:84
>> loc = 0x3fffa8109320 "\217{dK(\001E\220"
>> stat = <optimized out>
>> #21 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20,
>> objpp=0x3fffa8109270, obj_size=<optimized out>,
>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at
>> xdr_ref.c:135
>> more_data = 1
>> #22 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from
>> /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #23 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20,
>> pp=0x3fffa81090f0, size=<optimized out>, proc=<optimized out>) at
>> xdr_ref.c:84
>> loc = 0x3fffa81091a0 "\217\275\067\336\232\300(\005"
>> stat = <optimized out>
>> #24 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20,
>> objpp=0x3fffa81090f0, obj_size=<optimized out>,
>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at
>> xdr_ref.c:135
>> more_data = 1
>> #25 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from
>> /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #26 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20,
>> pp=0x3fffa8108f70, size=<optimized out>, proc=<optimized out>) at
>> xdr_ref.c:84
>> loc = 0x3fffa8109020 "\260.\025\b\244\352IT"
>> stat = <optimized out>
>> #27 0x00003fffb7764e04 in __GI_xdr_pointer (xdrs=0x3fffb1686d20,
>> objpp=0x3fffa8108f70, obj_size=<optimized out>,
>> xdr_obj=@0x3fffb785f4b0: 0x3fffb7842dc0 <.xdr_gfx_dirplist>) at
>> xdr_ref.c:135
>> more_data = 1
>> #28 0x00003fffb7842ec0 in .xdr_gfx_dirplist () from
>> /usr/lib64/libgfxdr.so.0
>> No symbol table info available.
>> #29 0x00003fffb7764c28 in __GI_xdr_reference (xdrs=0x3fffb1686d20,
>> pp=0x3fffa8108df0, size=<optimized out>, proc=<optimized out>) at
>> xdr_ref.c:84
>> loc = 0x3fffa8108ea0 "\212GS\203l\035\n\\"
>> ---Type <return> to continue, or q <return> to quit---
>>
>>
>> Regards,
>> Abhishek
>>
>> On Mon, Mar 11, 2019 at 7:10 PM Amar Tumballi Suryanarayan <
>> atumball at redhat.com> wrote:
>>
>>> Hi Abhishek,
>>>
>>> Can you check and get back to us?
>>>
>>> ```
>>> bash# ldd /usr/lib64/libglusterfs.so
>>> bash# ldd /usr/lib64/libgfrpc.so
>>>
>>> ```
>>>
>>> Also considering you have the core, can you do `(gdb) thr apply all bt
>>> full` and pass it on?
>>>
>>> Thanks & Regards,
>>> Amar
>>>
>>> On Mon, Mar 11, 2019 at 3:41 PM ABHISHEK PALIWAL <
>>> abhishpaliwal at gmail.com> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> COuld you please provide some pointer to debug it further.
>>>>
>>>> Regards,
>>>> Abhishek
>>>>
>>>> On Fri, Mar 8, 2019 at 2:19 PM ABHISHEK PALIWAL <
>>>> abhishpaliwal at gmail.com> wrote:
>>>>
>>>>> Hi Team,
>>>>>
>>>>> I am using Glusterfs 5.4, where after setting the gluster mount point
>>>>> when trying to access it, glusterfsd is getting crashed and mount point
>>>>> through the "Transport endpoint is not connected error.
>>>>>
>>>>> Here I are the gdb log for the core file
>>>>>
>>>>> warning: Could not load shared library symbols for linux-vdso64.so.1.
>>>>> Do you need "set solib-search-path" or "set sysroot"?
>>>>> [Thread debugging using libthread_db enabled]
>>>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>>>>> Core was generated by `/usr/sbin/glusterfsd -s 128.224.95.140
>>>>> --volfile-id gv0.128.224.95.140.tmp-bric'.
>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020,
>>>>> bytes=bytes at entry=36) at malloc.c:3327
>>>>> 3327 {
>>>>> [Current thread is 1 (Thread 0x3fff90394160 (LWP 811))]
>>>>> (gdb)
>>>>> (gdb)
>>>>> (gdb) bt
>>>>> #0 0x00003fff95ab1d48 in _int_malloc (av=av at entry=0x3fff7c000020,
>>>>> bytes=bytes at entry=36) at malloc.c:3327
>>>>> #1 0x00003fff95ab43dc in __GI___libc_malloc (bytes=36) at
>>>>> malloc.c:2921
>>>>> #2 0x00003fff95b6ffd0 in x_inline (xdrs=0x3fff90391d20,
>>>>> len=<optimized out>) at xdr_sizeof.c:89
>>>>> #3 0x00003fff95c4d488 in .xdr_gfx_iattx () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #4 0x00003fff95c4de84 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #5 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c132020, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #6 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c132020, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>> #7 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #8 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c131ea0, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #9 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c131ea0, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>> #10 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #11 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c131d20, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #12 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c131d20, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>> #13 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #14 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c131ba0, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #15 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c131ba0, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>> #16 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #17 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c131a20, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #18 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c131a20, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>> #19 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #20 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c1318a0, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #21 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c1318a0, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>> #22 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #23 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c131720, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #24 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c131720, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>> #25 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #26 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c1315a0, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #27 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c1315a0, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>> #28 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #29 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c131420, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #30 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c131420, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>> #31 0x00003fff95c4dec0 in .xdr_gfx_dirplist () from
>>>>> /usr/lib64/libgfxdr.so.0
>>>>> #32 0x00003fff95b6fc28 in __GI_xdr_reference (xdrs=0x3fff90391d20,
>>>>> pp=0x3fff7c1312a0, size=<optimized out>, proc=<optimized out>) at
>>>>> xdr_ref.c:84
>>>>> #33 0x00003fff95b6fe04 in __GI_xdr_pointer (xdrs=0x3fff90391d20,
>>>>> objpp=0x3fff7c1312a0, obj_size=<optimized out>,
>>>>> xdr_obj=@0x3fff95c6a4b0: 0x3fff95c4ddc0 <.xdr_gfx_dirplist>) at
>>>>> xdr_ref.c:135
>>>>>
>>>>> Frames are getting repeated, could any one please me.
>>>>> --
>>>>> Regards
>>>>> Abhishek Paliwal
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>>
>>>> Regards
>>>> Abhishek Paliwal
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>> --
>>> Amar Tumballi (amarts)
>>>
>>
>>
>> --
>>
>>
>>
>>
>> Regards
>> Abhishek Paliwal
>>
>
>
> --
>
>
>
>
> Regards
> Abhishek Paliwal
>
--
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190313/6b7172d3/attachment-0001.html>
More information about the Gluster-devel
mailing list