[Gluster-devel] [Gluster-users] Crash in glusterfs!!!

ABHISHEK PALIWAL abhishpaliwal at gmail.com
Mon Sep 24 08:38:44 UTC 2018


Could you please let me know about the bug in libc which you are talking.

On Mon, Sep 24, 2018 at 2:01 PM Pranith Kumar Karampuri <pkarampu at redhat.com>
wrote:

>
>
> On Mon, Sep 24, 2018 at 1:57 PM ABHISHEK PALIWAL <abhishpaliwal at gmail.com>
> wrote:
>
>> If you see the source code in cleanup_and_exit() we are getting the
>> SIGSEGV crash when 'exit(0)' is triggered.
>>
>
> yes, that is what I was mentioning earlier. It is crashing in libc. So
> either there is a bug in libc (glusterfs actually found 1 bug so far in
> libc, so I wouldn't rule out that possibility) or there is something that
> is happening in glusterfs which is leading to the problem.
> Valgrind/address-sanitizer would help find where the problem could be in
> some cases, so before reaching out libc developers, it is better to figure
> out where the problem is. Do you have steps to recreate it?
>
>
>>
>> On Mon, Sep 24, 2018 at 1:41 PM Pranith Kumar Karampuri <
>> pkarampu at redhat.com> wrote:
>>
>>>
>>>
>>> On Mon, Sep 24, 2018 at 1:36 PM ABHISHEK PALIWAL <
>>> abhishpaliwal at gmail.com> wrote:
>>>
>>>> Hi Sanju,
>>>>
>>>> Do you have any update on this?
>>>>
>>>
>>> This seems to happen while the process is dying, in libc. I am not
>>> completely sure if there is anything glusterfs is contributing to it from
>>> the bt at the moment. Do you have any steps to re-create this problem? It
>>> is probably better to run the steps with valgrind/address-sanitizer and see
>>> if it points to the problem in glusterfs.
>>>
>>>
>>>>
>>>> Regards,
>>>> Abhishek
>>>>
>>>> On Fri, Sep 21, 2018 at 4:07 PM ABHISHEK PALIWAL <
>>>> abhishpaliwal at gmail.com> wrote:
>>>>
>>>>> Hi Sanju,
>>>>>
>>>>> Output of 't a a bt full'
>>>>>
>>>>> (gdb) t a a bt full
>>>>>
>>>>>
>>>>>
>>>>> Thread 7 (LWP 1743):
>>>>>
>>>>> #0  0x00003fffa3ea7e88 in __lll_lock_wait (futex=0x0, private=0) at
>>>>> lowlevellock.c:43
>>>>>
>>>>>         r4 = 128
>>>>>
>>>>>         r7 = 0
>>>>>
>>>>>         arg2 = 128
>>>>>
>>>>>         r5 = 2
>>>>>
>>>>>         r8 = 1
>>>>>
>>>>>         r0 = 221
>>>>>
>>>>>         r3 = 0
>>>>>
>>>>>         r6 = 0
>>>>>
>>>>>         arg1 = 0
>>>>>
>>>>>         __err = 221
>>>>>
>>>>>         __ret = 0
>>>>>
>>>>> #1  0x00003fffa3e9ef64 in __GI___pthread_mutex_lock (mutex=0x100272a8)
>>>>> at ../nptl/pthread_mutex_lock.c:81
>>>>>
>>>>>         __futex = 0x100272a8
>>>>>
>>>>>         __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
>>>>>
>>>>>         type = <optimized out>
>>>>>
>>>>>         id = <optimized out>
>>>>>
>>>>> #2  0x00003fffa3f6ce8c in _gf_msg (domain=0x3fff98006c90
>>>>> "c_glusterfs-client-0", file=0x3fff9fb34de0 "client.c",
>>>>> function=0x3fff9fb34cd8 <__FUNCTION__.18849> "notify",
>>>>>
>>>>>     line=<optimized out>, level=<optimized out>, errnum=<optimized
>>>>> out>, trace=<optimized out>, msgid=114020,
>>>>>
>>>>>     fmt=0x3fff9fb35350 "parent translators are ready, attempting
>>>>> connect on transport") at logging.c:2058
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>>         msgstr = <optimized out>
>>>>>
>>>>>         ap = <optimized out>
>>>>>
>>>>>         this = 0x3fff980061f0
>>>>>
>>>>>         ctx = 0x10027010
>>>>>
>>>>>         callstr = '\000' <repeats 4095 times>
>>>>>
>>>>>         passcallstr = 0
>>>>>
>>>>>         log_inited = 0
>>>>>
>>>>>         __PRETTY_FUNCTION__ = "_gf_msg"
>>>>>
>>>>> #3  0x00003fff9fb084ac in notify (this=0x3fff980061f0,
>>>>> event=<optimized out>, data=0x3fff98008c50) at client.c:2116
>>>>>
>>>>>         conf = 0x3fff98056dd0
>>>>>
>>>>>         __FUNCTION__ = "notify"
>>>>>
>>>>> #4  0x00003fffa3f68ca0 in xlator_notify (xl=0x3fff980061f0,
>>>>> event=<optimized out>, data=<optimized out>) at xlator.c:491
>>>>>
>>>>>         old_THIS = 0x3fff98008c50
>>>>>
>>>>>         ret = 0
>>>>>
>>>>> #5  0x00003fffa3f87700 in default_notify (this=0x3fff98008c50,
>>>>> event=<optimized out>, data=<optimized out>) at defaults.c:2302
>>>>>
>>>>>         list = 0x3fff9800a340
>>>>>
>>>>> #6  0x00003fff9fac922c in afr_notify (this=0x3fff98008c50, event=1,
>>>>> data=0x3fff9800a4c0, data2=<optimized out>) at afr-common.c:3967
>>>>>
>>>>>         priv = 0x3fff98010050
>>>>>
>>>>>         i = <optimized out>
>>>>>
>>>>>         up_children = <optimized out>
>>>>>
>>>>>         down_children = <optimized out>
>>>>>
>>>>>         propagate = 1
>>>>>
>>>>>         had_heard_from_all = <optimized out>
>>>>>
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>
>>>>>         have_heard_from_all = 0
>>>>>
>>>>>         idx = <optimized out>
>>>>>
>>>>>         ret = 0
>>>>>
>>>>>         call_psh = <optimized out>
>>>>>
>>>>>         input = 0x0
>>>>>
>>>>>         output = 0x0
>>>>>
>>>>>         had_quorum = <optimized out>
>>>>>
>>>>>         has_quorum = <optimized out>
>>>>>
>>>>>         __FUNCTION__ = "afr_notify"
>>>>>
>>>>> #7  0x00003fff9fad4994 in notify (this=<optimized out>,
>>>>> event=<optimized out>, data=<optimized out>) at afr.c:38
>>>>>
>>>>>         ret = -1
>>>>>
>>>>>         ap = 0x3fffa034cc58 ""
>>>>>
>>>>>         data2 = <optimized out>
>>>>>
>>>>> #8  0x00003fffa3f68ca0 in xlator_notify (xl=0x3fff98008c50,
>>>>> event=<optimized out>, data=<optimized out>) at xlator.c:491
>>>>>
>>>>>         old_THIS = 0x3fff9800a4c0
>>>>>
>>>>>         ret = 0
>>>>>
>>>>> #9  0x00003fffa3f87700 in default_notify (this=0x3fff9800a4c0,
>>>>> event=<optimized out>, data=<optimized out>) at defaults.c:2302
>>>>>
>>>>>         list = 0x3fff9800b710
>>>>>
>>>>> #10 0x00003fff9fa6b1e4 in notify (this=<optimized out>,
>>>>> event=<optimized out>, data=<optimized out>) at io-stats.c:3064
>>>>>
>>>>>         ret = 0
>>>>>
>>>>>         args = {type = IOS_DUMP_TYPE_NONE, u = {logfp = 0x0, dict =
>>>>> 0x0}}
>>>>>
>>>>>         op = 0
>>>>>
>>>>>         list_cnt = 0
>>>>>
>>>>>         throughput = 0
>>>>>
>>>>>         time = 0
>>>>>
>>>>>         is_peek = _gf_false
>>>>>
>>>>>         ap = 0x3fffa034ce68 ""
>>>>>
>>>>>         __FUNCTION__ = "notify"
>>>>>
>>>>> #11 0x00003fffa3f68ca0 in xlator_notify (xl=0x3fff9800a4c0,
>>>>> event=<optimized out>, data=<optimized out>) at xlator.c:491
>>>>>
>>>>>         old_THIS = 0x3fffa402d290 <global_xlator>
>>>>>
>>>>>         ret = 0
>>>>>
>>>>> #12 0x00003fffa3fbd560 in glusterfs_graph_parent_up (graph=<optimized
>>>>> out>) at graph.c:440
>>>>>
>>>>>         trav = 0x3fff9800a4c0
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>> #13 0x00003fffa3fbdb90 in glusterfs_graph_activate
>>>>> (graph=0x3fff98000af0, ctx=0x10027010) at graph.c:688
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>>         __FUNCTION__ = "glusterfs_graph_activate"
>>>>>
>>>>> #14 0x000000001000a49c in glusterfs_process_volfp (ctx=0x10027010,
>>>>> fp=0x3fff98001cd0) at glusterfsd.c:2221
>>>>>
>>>>>         graph = 0x3fff98000af0
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>>         trav = <optimized out>
>>>>>
>>>>>         __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot
>>>>> access memory at address 0x10010ec0)>
>>>>>
>>>>> #15 0x000000001000fd08 in mgmt_getspec_cbk (req=<optimized out>,
>>>>> iov=<optimized out>, count=<optimized out>, myframe=0x3fffa2bea06c) at
>>>>> glusterfsd-mgmt.c:1561
>>>>>
>>>>>         rsp = {op_ret = 1059, op_errno = 0,
>>>>>
>>>>>           spec = 0x3fff980018a0 "volume c_glusterfs-client-0\n    type
>>>>> protocol/client\n    option password
>>>>> 5fd8d83d-99f3-4630-97c6-965d7a8ead62\n    option username
>>>>> e65687aa-e135-445e-8778-48bb8fb19640\n    option transport-type tcp\n
>>>>> "..., xdata = {xdata_len = 0, xdata_val = 0x0}}
>>>>>
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>
>>>>>         frame = 0x3fffa2bea06c
>>>>>
>>>>>         ctx = 0x10027010
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>>         size = 1059
>>>>>
>>>>>         tmpfp = 0x3fff98001cd0
>>>>>
>>>>>         volfilebuf = 0x0
>>>>>
>>>>>        __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot
>>>>> access memory at address 0x10013570)>
>>>>>
>>>>> #16 0x00003fffa3f21ec4 in rpc_clnt_handle_reply (clnt=0x10089020,
>>>>> pollin=0x3fff98001760) at rpc-clnt.c:775
>>>>>
>>>>>         conn = 0x10089050
>>>>>
>>>>>         saved_frame = <optimized out>
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>>         req = 0x1008931c
>>>>>
>>>>>         xid = 1
>>>>>
>>>>>         __FUNCTION__ = "rpc_clnt_handle_reply"
>>>>>
>>>>> #17 0x00003fffa3f223d0 in rpc_clnt_notify (trans=<optimized out>,
>>>>> mydata=0x10089050, event=<optimized out>, data=<optimized out>) at
>>>>> rpc-clnt.c:933
>>>>>
>>>>>         conn = 0x10089050
>>>>>
>>>>>         clnt = <optimized out>
>>>>>
>>>>>         ret = -1
>>>>>
>>>>>         req_info = 0x0
>>>>>
>>>>>         pollin = <optimized out>
>>>>>
>>>>>         clnt_mydata = 0x0
>>>>>
>>>>>         old_THIS = 0x3fffa402d290 <global_xlator>
>>>>>
>>>>>         __FUNCTION__ = "rpc_clnt_notify"
>>>>>
>>>>> #18 0x00003fffa3f1d4fc in rpc_transport_notify (this=<optimized out>,
>>>>> event=<optimized out>, data=<optimized out>) at rpc-transport.c:546
>>>>>
>>>>>         ret = -1
>>>>>
>>>>>         __FUNCTION__ = "rpc_transport_notify"
>>>>>
>>>>> #19 0x00003fffa0401d44 in socket_event_poll_in (this=this at entry=0x1008ab80)
>>>>> at socket.c:2236
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>>         pollin = 0x3fff98001760
>>>>>
>>>>>         priv = 0x1008b820
>>>>>
>>>>> #20 0x00003fffa040489c in socket_event_handler (fd=<optimized out>,
>>>>> idx=<optimized out>, data=0x1008ab80, poll_in=<optimized out>,
>>>>> poll_out=<optimized out>, poll_err=<optimized out>)
>>>>>
>>>>>     at socket.c:2349
>>>>>
>>>>>         this = 0x1008ab80
>>>>>
>>>>>         priv = 0x1008b820
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>>         __FUNCTION__ = "socket_event_handler"
>>>>>
>>>>> #21 0x00003fffa3fe2874 in event_dispatch_epoll_handler
>>>>> (event=0x3fffa034d6a0, event_pool=0x10045bc0) at event-epoll.c:575
>>>>>
>>>>>         handler = @0x3fffa041f620: 0x3fffa04046f0
>>>>> <socket_event_handler>
>>>>>
>>>>>         gen = 1
>>>>>
>>>>>         slot = 0x1007cd80
>>>>>
>>>>>         data = <optimized out>
>>>>>
>>>>>         ret = -1
>>>>>
>>>>>         fd = 9
>>>>>
>>>>>         ev_data = 0x3fffa034d6a8
>>>>>
>>>>>         idx = 1
>>>>>
>>>>> #22 event_dispatch_epoll_worker (data=0x1008bd50) at event-epoll.c:678
>>>>>
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>
>>>>>         event = {events = 1, data = {ptr = 0x100000001, fd = 1, u32 =
>>>>> 1, u64 = 4294967297}}
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>>         ev_data = 0x1008bd50
>>>>>
>>>>>         event_pool = 0x10045bc0
>>>>>
>>>>>         myindex = <optimized out>
>>>>>
>>>>>         timetodie = 0
>>>>>
>>>>>         __FUNCTION__ = "event_dispatch_epoll_worker"
>>>>>
>>>>> #23 0x00003fffa3e9bb30 in start_thread (arg=0x3fffa034e160) at
>>>>> pthread_create.c:462
>>>>>
>>>>>         pd = 0x3fffa034e160
>>>>>
>>>>>         now = <optimized out>
>>>>>
>>>>>         unwind_buf = {cancel_jmp_buf = {{jmp_buf =
>>>>> {2380233324717718430, 70367199403008, 2380233324703897146, 0, 0,
>>>>> 70367128645632, 70367137030688, 8388608, 70367199363104, 269008208,
>>>>>
>>>>>                 70368094386592, 70367199388632, 70367200825640, 3, 0,
>>>>> 70367199388648, 70368094386240, 70368094386296, 4001536, 70367199364120,
>>>>> 70367137027904, -3187653596,
>>>>>
>>>>>                 0 <repeats 42 times>}, mask_was_saved = 0}}, priv =
>>>>> {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype
>>>>> = 0}}}
>>>>>
>>>>>         not_first_call = <optimized out>
>>>>>
>>>>>         pagesize_m1 = <optimized out>
>>>>>
>>>>>         sp = <optimized out>
>>>>>
>>>>>         freesize = <optimized out>
>>>>>
>>>>>         __PRETTY_FUNCTION__ = "start_thread"
>>>>>
>>>>> #24 0x00003fffa3de60fc in .__clone () at
>>>>> ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
>>>>>
>>>>> No locals.
>>>>>
>>>>>
>>>>>
>>>>> Thread 6 (LWP 1735):
>>>>>
>>>>> #0  0x00003fffa3ea3ccc in __pthread_cond_timedwait (cond=0x10059a98,
>>>>> mutex=0x10059a70, abstime=0x3fffa141f670) at pthread_cond_timedwait.c:198
>>>>>
>>>>>         r4 = 393
>>>>>
>>>>>         r7 = 0
>>>>>
>>>>>         arg5 = 0
>>>>>
>>>>>         arg2 = <optimized out>
>>>>>
>>>>>         r5 = 2
>>>>>
>>>>>         r8 = 4294967295
>>>>>
>>>>>         arg6 = 4294967295
>>>>>
>>>>>         arg3 = 2
>>>>>
>>>>>         r0 = 221
>>>>>
>>>>>         r3 = 516
>>>>>
>>>>>         r6 = 70367154665072
>>>>>
>>>>>         arg4 = 70367154665072
>>>>>
>>>>>         arg1 = 268802716
>>>>>
>>>>>         __err = <optimized out>
>>>>>
>>>>>         __ret = <optimized out>
>>>>>
>>>>>         futex_val = 2
>>>>>
>>>>>         buffer = {__routine = @0x3fffa3ec0b50: 0x3fffa3ea3400
>>>>> <__condvar_cleanup>, __arg = 0x3fffa141f540, __canceltype = 0, __prev = 0x0}
>>>>>
>>>>>         cbuffer = {oldtype = 0, cond = 0x10059a98, mutex = 0x10059a70,
>>>>> bc_seq = 0}
>>>>>
>>>>>         result = 0
>>>>>
>>>>>         pshared = 0
>>>>>
>>>>>         pi_flag = 0
>>>>>
>>>>>         err = <optimized out>
>>>>>
>>>>>         val = <optimized out>
>>>>>
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>
>>>>>         seq = 0
>>>>>
>>>>> #1  0x00003fffa3fc0e74 in syncenv_task (proc=0x10053eb0) at
>>>>> syncop.c:607
>>>>>
>>>>>         env = 0x10053eb0
>>>>>
>>>>>         task = 0x0
>>>>>
>>>>>         sleep_till = {tv_sec = 1536845230, tv_nsec = 0}
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>> #2  0x00003fffa3fc1cdc in syncenv_processor (thdata=0x10053eb0) at
>>>>> syncop.c:699
>>>>>
>>>>>         env = 0x10053eb0
>>>>>
>>>>>         proc = 0x10053eb0
>>>>>
>>>>>         task = <optimized out>
>>>>>
>>>>> #3  0x00003fffa3e9bb30 in start_thread (arg=0x3fffa1420160) at
>>>>> pthread_create.c:462
>>>>>
>>>>>         pd = 0x3fffa1420160
>>>>>
>>>>>         now = <optimized out>
>>>>>
>>>>>         unwind_buf = {cancel_jmp_buf = {{jmp_buf =
>>>>> {2380233324731669406, 70367199403008, 2380233324703897146, 0, 0,
>>>>> 70367146283008, 70367154668064, 8388608, 70367199363104, 268779184,
>>>>>
>>>>>                 268779184, 70367199388632, 70367200820192, 3, 0,
>>>>> 70367199388648, 70368094386080, 70368094386136, 4001536, 70367199364120,
>>>>> 70367154665280, -3187653564,
>>>>>
>>>>>                 0 <repeats 42 times>}, mask_was_saved = 0}}, priv =
>>>>> {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype
>>>>> = 0}}}
>>>>>
>>>>>         not_first_call = <optimized out>
>>>>>
>>>>>         pagesize_m1 = <optimized out>
>>>>>
>>>>>         sp = <optimized out>
>>>>>
>>>>>         freesize = <optimized out>
>>>>>
>>>>>         __PRETTY_FUNCTION__ = "start_thread"
>>>>>
>>>>> #4  0x00003fffa3de60fc in .__clone () at
>>>>> ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
>>>>>
>>>>> No locals.
>>>>>
>>>>>
>>>>>
>>>>> Thread 5 (LWP 1746):
>>>>>
>>>>> #0  0x00003fffa3ea7e38 in __lll_lock_wait (futex=0x100272a8,
>>>>> private=<optimized out>) at lowlevellock.c:46
>>>>>
>>>>>         r4 = 128
>>>>>
>>>>>         r7 = 0
>>>>>
>>>>>         r5 = 2
>>>>>
>>>>>         r8 = 1
>>>>>
>>>>>         arg3 = 2
>>>>>
>>>>>         r0 = 221
>>>>>
>>>>>         r3 = 512
>>>>>
>>>>>         r6 = 0
>>>>>
>>>>>         arg4 = 0
>>>>>
>>>>>         arg1 = 268595880
>>>>>
>>>>>         __err = <optimized out>
>>>>>
>>>>>         __ret = <optimized out>
>>>>>
>>>>> #1  0x00003fffa3e9ef64 in __GI___pthread_mutex_lock (mutex=0x100272a8)
>>>>> at ../nptl/pthread_mutex_lock.c:81
>>>>>
>>>>>         __futex = 0x100272a8
>>>>>
>>>>>         __PRETTY_FUNCTION__ = "__pthread_mutex_lock"
>>>>>
>>>>>         type = <optimized out>
>>>>>
>>>>>         id = <optimized out>
>>>>>
>>>>> #2  0x00003fffa3f6ce8c in _gf_msg (domain=0x3fffa4009e38 "epoll",
>>>>> file=0x3fffa4009e28 "event-epoll.c", function=0x3fffa4009db8
>>>>> <__FUNCTION__.8510> "event_dispatch_epoll_worker",
>>>>>
>>>>>     line=<optimized out>, level=<optimized out>, errnum=<optimized
>>>>> out>, trace=<optimized out>, msgid=101190, fmt=0x3fffa4009f48 "Started
>>>>> thread with index %d") at logging.c:2058
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>
>>>>>         msgstr = <optimized out>
>>>>>
>>>>>         ap = <optimized out>
>>>>>
>>>>>         this = 0x3fffa402d290 <global_xlator>
>>>>>
>>>>>         ctx = 0x10027010
>>>>>
>>>>>         callstr = '\000' <repeats 4095 times>
>>>>>
>>>>>         passcallstr = 0
>>>>>
>>>>>         log_inited = 0
>>>>>
>>>>>         __PRETTY_FUNCTION__ = "_gf_msg"
>>>>>
>>>>> #3  0x00003fffa3fe265c in event_dispatch_epoll_worker
>>>>> (data=0x3fff9802ffd0) at event-epoll.c:631
>>>>>
>>>>>         event = {events = 0, data = {ptr = 0x0, fd = 0, u32 = 0, u64 =
>>>>> 0}}
>>>>>
>>>>>         ret = -1
>>>>>
>>>>>         ev_data = 0x3fff9802ffd0
>>>>>
>>>>>         event_pool = 0x10045bc0
>>>>>
>>>>>         myindex = <optimized out>
>>>>>
>>>>>         timetodie = 0
>>>>>
>>>>>         __FUNCTION__ = "event_dispatch_epoll_worker"
>>>>>
>>>>> #4  0x00003fffa3e9bb30 in start_thread (arg=0x3fff9f37d160) at
>>>>> pthread_create.c:462
>>>>>
>>>>>         pd = 0x3fff9f37d160
>>>>>
>>>>>         now = <optimized out>
>>>>>
>>>>>         unwind_buf = {cancel_jmp_buf = {{jmp_buf =
>>>>> {2380233325439331230, 70367199403008, 2380233324703897146, 0, 269006880,
>>>>> 70367112060928, 70367120445984, 8388608, 70367199363104,
>>>>>
>>>>>                 70366999543760, 70367137025008, 70367199388632,
>>>>> 70367200825640, 3, 0, 70367199388648, 70367137024656, 70367137024712,
>>>>> 4001536, 70367199364120, 70367120443200, -3183328188,
>>>>>
>>>>>                 0 <repeats 42 times>}, mask_was_saved = 0}}, priv =
>>>>> {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype
>>>>> = 0}}}
>>>>>
>>>>>         not_first_call = <optimized out>
>>>>>
>>>>>         pagesize_m1 = <optimized out>
>>>>>
>>>>>         sp = <optimized out>
>>>>>
>>>>>         freesize = <optimized out>
>>>>>
>>>>>         __PRETTY_FUNCTION__ = "start_thread"
>>>>>
>>>>> #5  0x00003fffa3de60fc in .__clone () at
>>>>> ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
>>>>>
>>>>> No locals.
>>>>>
>>>>>
>>>>>
>>>>> Thread 4 (LWP 1733):
>>>>>
>>>>> #0  0x00003fffa3ea9150 in .__nanosleep () at
>>>>> ../sysdeps/unix/syscall-template.S:84
>>>>>
>>>>> No locals.
>>>>>
>>>>> #1  0x00003fffa3f93f28 in gf_timer_proc (ctx=0x10027010) at timer.c:205
>>>>>
>>>>>         now = 20528921491
>>>>>
>>>>>         now_ts = {tv_sec = 20, tv_nsec = 528921491}
>>>>>
>>>>>         reg = 0x100533c0
>>>>>
>>>>>         sleepts = {tv_sec = 0, tv_nsec = 2000000}
>>>>>
>>>>>         event = 0x1008d280
>>>>>
>>>>>         old_THIS = <optimized out>
>>>>>
>>>>>         __FUNCTION__ = "gf_timer_proc"
>>>>>
>>>>> #2  0x00003fffa3e9bb30 in start_thread (arg=0x3fffa2420160) at
>>>>> pthread_create.c:462
>>>>>
>>>>>         pd = 0x3fffa2420160
>>>>>
>>>>>         now = <optimized out>
>>>>>
>>>>>         unwind_buf = {cancel_jmp_buf = {{jmp_buf =
>>>>> {2380233324681337758, 70367199403008, 2380233324703897146, 0, 0,
>>>>> 70367163060224, 70367171445280, 8388608, 70367199363104, 268595216,
>>>>>
>>>>>                 268776384, 70367199388632, 70367200807088, 3, 0,
>>>>> 70367199388648, 70368094385584, 70368094385640, 4001536, 70367199364120,
>>>>> 70367171442496, -3187653564,
>>>>>
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>
>>>>>                 0 <repeats 42 times>}, mask_was_saved = 0}}, priv =
>>>>> {pad = {0x0, 0x0, 0x0, 0x0}, data = {prev = 0x0, cleanup = 0x0, canceltype
>>>>> = 0}}}
>>>>>
>>>>>         not_first_call = <optimized out>
>>>>>
>>>>>         pagesize_m1 = <optimized out>
>>>>>
>>>>>         sp = <optimized out>
>>>>>
>>>>>         freesize = <optimized out>
>>>>>
>>>>>         __PRETTY_FUNCTION__ = "start_thread"
>>>>>
>>>>> #3  0x00003fffa3de60fc in .__clone () at
>>>>> ../sysdeps/unix/sysv/linux/powerpc/powerpc64/clone.S:96
>>>>>
>>>>> No locals.
>>>>>
>>>>>
>>>>>
>>>>> Thread 3 (LWP 1732):
>>>>>
>>>>> #0  0x00003fffa3e9d084 in pthread_join (threadid=70367137030496,
>>>>> thread_return=0x0) at pthread_join.c:90
>>>>>
>>>>>         r4 = 0
>>>>>
>>>>>         r7 = 2
>>>>>
>>>>>         arg2 = 0
>>>>>
>>>>>         r5 = 1743
>>>>>
>>>>>         r8 = 2
>>>>>
>>>>>         arg3 = 1743
>>>>>
>>>>>         r0 = 221
>>>>>
>>>>>         r3 = 512
>>>>>
>>>>>         r6 = 0
>>>>>
>>>>>         arg4 = 0
>>>>>
>>>>>         arg1 = 70367137030704
>>>>>
>>>>>         __err = <optimized out>
>>>>>
>>>>>         __ret = <optimized out>
>>>>>
>>>>>         __tid = 1743
>>>>>
>>>>>         _buffer = {__routine = @0x3fffa3ec0478: 0x3fffa3e9cf70
>>>>> <cleanup>, __arg = 0x3fffa034e588, __canceltype = 16383, __prev = 0x0}
>>>>>
>>>>>         oldtype = 0
>>>>>
>>>>>         self = 0x3fffa405e000
>>>>>
>>>>>         result = 0
>>>>>
>>>>> #1  0x00003fffa3fe2ca0 in event_dispatch_epoll (event_pool=0x10045bc0)
>>>>> at event-epoll.c:762
>>>>>
>>>>>         i = <optimized out>
>>>>>
>>>>>         t_id = 70367137030496
>>>>>
>>>>>         pollercount = 1
>>>>>
>>>>>         ret = 0
>>>>>
>>>>>         ev_data = <optimized out>
>>>>>
>>>>>         __FUNCTION__ = "event_dispatch_epoll"
>>>>>
>>>>> #2  0x00003fffa3fab7e4 in event_dispatch (event_pool=<optimized out>)
>>>>> at event.c:128
>>>>>
>>>>>         ret = -1
>>>>>
>>>>>         __FUNCTION__ = "event_dispatch"
>>>>>
>>>>> #3  0x0000000010005ea8 in main (argc=<optimized out>, argv=<optimized
>>>>> out>) at glusterfsd.c:2380
>>>>>
>>>>>         ctx = 0x10027010
>>>>>
>>>>>         ret = 0
>>>>>
>>>>>         cmdlinestr = "/usr/sbin/glusterfs -s localhost --volfile-id
>>>>> gluster/glustershd -p /system/glusterd/glustershd/run/glustershd.pid -l
>>>>> /var/log/glusterfs/glustershd.log -S
>>>>> /var/run/gluster/868ec0f3795bfb5abd176dadece9"...
>>>>>
>>>>>         cmd = 0x10027010
>>>>>
>>>>>         __FUNCTION__ = <error reading variable __FUNCTION__ (Cannot
>>>>> access memory at address 0x10010f00)>
>>>>>
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>
>>>>> #4  0x00003fffa3cfe318 in generic_start_main (main=0x10025718 <main>,
>>>>> argc=<optimized out>, argv=0x3fffd9450fd8, auxvec=0x3fffd9451068,
>>>>> init=<optimized out>, rtld_fini=<optimized out>,
>>>>>
>>>>>     stack_end=<optimized out>, fini=<optimized out>) at
>>>>> ../csu/libc-start.c:289
>>>>>
>>>>>         self = 0x3fffa405e000
>>>>>
>>>>>         result = <optimized out>
>>>>>
>>>>>         unwind_buf = {cancel_jmp_buf = {{jmp_buf =
>>>>> {2380233326476224334, 70367199177216, 2380233324705844766, 0 <repeats 13
>>>>> times>, 70367201122640, 0, 70367201122648, 0, 268589712,
>>>>>
>>>>>                 -3724539774, 0 <repeats 42 times>}, mask_was_saved =
>>>>> 0}}, priv = {pad = {0x0, 0x0, 0x3fffd9450ed0, 0x0}, data = {prev = 0x0,
>>>>> cleanup = 0x0, canceltype = 16383}}}
>>>>>
>>>>>         not_first_call = <optimized out>
>>>>>
>>>>> #5  0x00003fffa3cfe54c in __libc_start_main (argc=<optimized out>,
>>>>> argv=<optimized out>, ev=<optimized out>, auxvec=<optimized out>,
>>>>> rtld_fini=<optimized out>, stinfo=<optimized out>,
>>>>>
>>>>>     stack_on_entry=<optimized out>) at
>>>>> ../sysdeps/unix/sysv/linux/powerpc/libc-start.c:93
>>>>>
>>>>> No locals.
>>>>>
>>>>> #6  0x0000000000000000 in ?? ()
>>>>>
>>>>> No symbol table info available.
>>>>>
>>>>>
>>>>>
>>>>> Thread 2 (LWP 1736):
>>>>>
>>>>> #0  0x00003fffa3ea3ccc in __pthread_cond_timedwait (cond=0x10059a98,
>>>>> mutex=0x10059a70, abstime=0x3fffa0c1f670) at pthread_cond_timedwait.c:198
>>>>>
>>>>>         r4 = 393
>>>>>
>>>>>         r7 = 0
>>>>>
>>>>>         arg5 = 0
>>>>>
>>>>>         arg2 = <optimized out>
>>>>>
>>>>>         r5 = 2
>>>>>
>>>>>         r8 = 4294967295
>>>>>
>>>>>         arg6 = 4294967295
>>>>>
>>>>>         arg3 = 2
>>>>>
>>>>>         r0 = 221
>>>>>
>>>>>         r3 = 516
>>>>>
>>>>>         r6 = 70367146276464
>>>>>
>>>>>         arg4 = 70367146276464
>>>>>
>>>>>         arg1 = 268802716
>>>>>
>>>>>         __err = <optimized out>
>>>>>
>>>>>         __ret = <optimized out>
>>>>>
>>>>>         futex_val = 2
>>>>>
>>>>>         buffer = {__routine = @0x3fffa3ec0b50: 0x3fffa3ea3400
>>>>> <__condvar_cleanup>, __arg = 0x3fffa0c1f540, __canceltype = 0, __prev = 0x0}
>>>>>
>>>>>         cbuffer = {oldtype = 0, cond = 0x10059a98, mutex = 0x10059a70,
>>>>> bc_seq = 0}
>>>>>
>>>>>         result = 0
>>>>>
>>>>>         pshared = 0
>>>>>
>>>>>         pi_flag = 0
>>>>>
>>>>>         err = <optimized out>
>>>>>
>>>>>         val = <optimized out>
>>>>>
>>>>>         seq = 0
>>>>>
>>>>> #1  0x00003fffa3fc0e74 in syncenv_task (proc=0x10054468) at
>>>>> syncop.c:607
>>>>>
>>>>>         env = 0x10053eb0
>>>>>
>>>>>         task = 0x0
>>>>>
>>>>>         sleep_till = {tv_sec = 1536845230, tv_nsec = 0}
>>>>>
>>>>>         ret = <optimized out>
>>>>>
>>>>> #2  0x00003fffa3fc1cdc in syncenv_processor (thdata=0x10054468) at
>>>>> syncop.c:699
>>>>>
>>>>>         env = 0x10053eb0
>>>>>
>>>>> ---Type <return> to continue, or q <return> to quit---
>>>>>
>>>>>         proc = 0x10054468
>>>>>
>>>>>         task = <optimized out>
>>>>>
>>>>> #3  0x00003fffa3
>>>>>
>>>>

-- 




Regards
Abhishek Paliwal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180924/88068134/attachment-0001.html>


More information about the Gluster-devel mailing list