[Gluster-devel] regarding glusterd crash on NetBSD https://build.gluster.org/job/rackspace-netbsd7-regression-triggered/14100/consoleFull
Pranith Kumar Karampuri
pkarampu at redhat.com
Wed Feb 10 12:46:53 UTC 2016
On 02/10/2016 06:13 PM, Kaushal M wrote:
> On Wed, Feb 10, 2016 at 6:08 PM, Pranith Kumar Karampuri
> <pkarampu at redhat.com> wrote:
>>
>> On 02/10/2016 06:01 PM, Atin Mukherjee wrote:
>>> Not that I am aware of. Do you have backtrace of all the threads?
>>
>> it doesn't seem to give proper output for 'thread apply all bt':
>> (gdb) thread apply all bt
>>
>> Thread 6 (process 2):
>> #0 0xbb354977 in ?? ()
>> #1 0xbb682b67 in ?? ()
>> #2 0xba4fff98 in ?? ()
>> Cannot access memory at address 0xba4fffd4
>>
>> I tried info threads and went to each thread, all of them except the thread
>> that crashed has '??' in all frames. You can check the core on the netbsd
>> machine mentioned in the console output in subject.
>>
>> Pranith
> We've observed these sorts of crashes (segfaults with stack having a
> rpc event happening), before when GlusterD is in the process of
> shutting down and exit() has been called. Since we don't correctly
> call fini() on xlators, rpcsvc continues accepting requests. But
> exit() would have cleaned up the resources that the rpc thread would
> have been using, leading to the thread segfaulting.
Okay, I will retrigger the test.
Pranith
>
>>> ~Atin
>>>
>>> On 02/10/2016 05:58 PM, Pranith Kumar Karampuri wrote:
>>>> hi Atin, Kaushal,
>>>> Is this a known issue?
>>>>
>>>> (gdb) #1 0xbb789fb7 in __synclock_unlock (lock=0xbb1d4ac0)
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1056
>>>>
>>>> #2 0xbb789ffd in synclock_unlock (lock=0xbb1d4ac0)
>>>> at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1071
>>>>
>>>> #3 0xb9b803ff in glusterd_big_locked_notify (rpc=0xb8bc2070,
>>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0,
>>>> notify_fn=0xb9b8ec28 <__glusterd_brick_rpc_notify>)
>>>> at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:72
>>>>
>>>> #4 0xb9b8f0d7 in glusterd_brick_rpc_notify (rpc=0xb8bc2070,
>>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0)
>>>> at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:4993
>>>>
>>>> #5 0xbb708c22 in rpc_clnt_notify (trans=0xb8bc8030, mydata=0xb8bc2090,
>>>> event=RPC_TRANSPORT_DISCONNECT, data=0xb8bc8030)
>>>> at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnt
>>>> at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnIllegal
>>>> process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1056.
>>>>
>>>> (gdb) #2 0xbb789ffd in synclock_unlock (lock=0xbb1d4ac0)
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1071
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1071.
>>>>
>>>> (gdb) #3 0xb9b803ff in glusterd_big_locked_notify (rpc=0xb8bc2070,
>>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0,
>>>> (gdb) notify_fn=0xb9b8ec28 <__glusterd_brick_rpc_notify>)
>>>> Undefined command: "notify_fn". Try "help".
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:72
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:72.
>>>>
>>>> (gdb) #4 0xb9b8f0d7 in glusterd_brick_rpc_notify (rpc=0xb8bc2070,
>>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0)
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:4993
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:4993.
>>>>
>>>> #5 0xbb708c22 in rpc_clnt_notify (trans=0xb8bc8030, mydata=0xb8bc2090,
>>>> event=RPC_TRANSPORT_DISCONNECT, data=0xb8bc8030)
>>>>
>>>> /jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnt
>>>> at 8030)
>>>> at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546
>>>>
>>>> #7 0xbb231847 in socket_event_poll_err (this=0xb8bc8030)
>>>> at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:1151
>>>>
>>>> #8 0xbb2359a2 in socket_event_handler (fd=15, idx=9, data=0xb8bc8030,
>>>> poll_in=1, poll_out=4, poll_err=0)
>>>> at /home/jenkins/root/workspace/rackspace-n(gdb) #5 0xbb708c22 in
>>>> rpc_clnt_notify (trans=0xb8bc8030, mydata=0xb8bc2090,
>>>> event=RPC_TRANSPORT_DISCONNECT, data=0xb8bc8030)
>>>> (gdb)
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnt
>>>> at 8030)
>>>> Undefined command: "". Try "help".
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546.
>>>>
>>>> (gdb) #7 0xbb231847 in socket_event_poll_err (this=0xb8bc8030)
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:1151
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:1151.
>>>>
>>>> (gdb) #8 0xbb2359a2 in socket_event_handler (fd=15, idx=9,
>>>> data=0xb8bc8030, poll_in=1, poll_out=4, poll_err=0)
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:2356
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:2356.
>>>>
>>>> (gdb) #9 0xbb7a572e in event_dispatch_poll_handler
>>>> (event_pool=0xbb143030, ufds=0xbb18f370, i=9)
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:393
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:393.
>>>>
>>>> (gdb) #10 0xbb7a5a75 in event_dispatch_poll (event_pool=0xbb143030)
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:489
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:489.
>>>>
>>>> (gdb) #11 0xbb774a35 in event_dispatch (event_pool=0xbb143030)
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event.c:128
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event.c:128.
>>>>
>>>> (gdb) #12 0x08050e20 in main (argc=1, argv=0xbf7febdc)
>>>> (gdb) at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/glusterfsd/src/glusterfsd.c:2350
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/glusterfsd/src/glusterfsd.c:2350.
>>>>
>>>>
>>>> Pranith
>>
More information about the Gluster-devel
mailing list