[Gluster-devel] regarding glusterd crash on NetBSD https://build.gluster.org/job/rackspace-netbsd7-regression-triggered/14100/consoleFull

Wed Feb 10 12:56:48 UTC 2016


On 02/10/2016 06:13 PM, Kaushal M wrote:
> On Wed, Feb 10, 2016 at 6:08 PM, Pranith Kumar Karampuri
> <pkarampu at redhat.com> wrote:
>>
>>
>> On 02/10/2016 06:01 PM, Atin Mukherjee wrote:
>>>
>>> Not that I am aware of. Do you have backtrace of all the threads?
>>
>>
>> it doesn't seem to give proper output for 'thread apply all bt':
>> (gdb) thread apply all bt
>>
>> Thread 6 (process 2):
>> #0  0xbb354977 in ?? ()
>> #1  0xbb682b67 in ?? ()
>> #2  0xba4fff98 in ?? ()
>> Cannot access memory at address 0xba4fffd4
>>
>> I tried info threads and went to each thread, all of them except the thread
>> that crashed has '??' in all frames. You can check the core on the netbsd
>> machine mentioned in the console output in subject.
>>
>> Pranith
> 
> We've observed these sorts of crashes (segfaults with stack having a
> rpc event happening), before when GlusterD is in the process of
> shutting down and exit() has been called. Since we don't correctly
> call fini() on xlators, rpcsvc continues accepting requests. But
> exit() would have cleaned up the resources that the rpc thread would
> have been using, leading to the thread segfaulting.

I've seen urcu unlock function crashes in this case. Haven't seen any
other functions crashing apart from that, but as Kaushal said it can
crash from any places where the resource is being read and cleanup
thread has already freed it up. So complete thread details might have
helped here.
> 
>>
>>>
>>> ~Atin
>>>
>>> On 02/10/2016 05:58 PM, Pranith Kumar Karampuri wrote:
>>>>
>>>> hi Atin, Kaushal,
>>>>            Is this a known issue?
>>>>
>>>> (gdb) #1  0xbb789fb7 in __synclock_unlock (lock=0xbb1d4ac0)
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1056
>>>>
>>>> #2  0xbb789ffd in synclock_unlock (lock=0xbb1d4ac0)
>>>>      at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1071
>>>>
>>>> #3  0xb9b803ff in glusterd_big_locked_notify (rpc=0xb8bc2070,
>>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0,
>>>>      notify_fn=0xb9b8ec28 <__glusterd_brick_rpc_notify>)
>>>>      at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:72
>>>>
>>>> #4  0xb9b8f0d7 in glusterd_brick_rpc_notify (rpc=0xb8bc2070,
>>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0)
>>>>      at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:4993
>>>>
>>>> #5  0xbb708c22 in rpc_clnt_notify (trans=0xb8bc8030, mydata=0xb8bc2090,
>>>> event=RPC_TRANSPORT_DISCONNECT, data=0xb8bc8030)
>>>>      at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnt
>>>> at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnIllegal
>>>> process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1056.
>>>>
>>>> (gdb) #2  0xbb789ffd in synclock_unlock (lock=0xbb1d4ac0)
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1071
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1071.
>>>>
>>>> (gdb) #3  0xb9b803ff in glusterd_big_locked_notify (rpc=0xb8bc2070,
>>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0,
>>>> (gdb)     notify_fn=0xb9b8ec28 <__glusterd_brick_rpc_notify>)
>>>> Undefined command: "notify_fn".  Try "help".
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:72
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:72.
>>>>
>>>> (gdb) #4  0xb9b8f0d7 in glusterd_brick_rpc_notify (rpc=0xb8bc2070,
>>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0)
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:4993
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:4993.
>>>>
>>>> #5  0xbb708c22 in rpc_clnt_notify (trans=0xb8bc8030, mydata=0xb8bc2090,
>>>> event=RPC_TRANSPORT_DISCONNECT, data=0xb8bc8030)
>>>>
>>>> /jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnt
>>>> at 8030)
>>>>      at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546
>>>>
>>>> #7  0xbb231847 in socket_event_poll_err (this=0xb8bc8030)
>>>>      at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:1151
>>>>
>>>> #8  0xbb2359a2 in socket_event_handler (fd=15, idx=9, data=0xb8bc8030,
>>>> poll_in=1, poll_out=4, poll_err=0)
>>>>      at /home/jenkins/root/workspace/rackspace-n(gdb) #5  0xbb708c22 in
>>>> rpc_clnt_notify (trans=0xb8bc8030, mydata=0xb8bc2090,
>>>> event=RPC_TRANSPORT_DISCONNECT, data=0xb8bc8030)
>>>> (gdb)
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnt
>>>> at 8030)
>>>> Undefined command: "".  Try "help".
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546.
>>>>
>>>> (gdb) #7  0xbb231847 in socket_event_poll_err (this=0xb8bc8030)
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:1151
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:1151.
>>>>
>>>> (gdb) #8  0xbb2359a2 in socket_event_handler (fd=15, idx=9,
>>>> data=0xb8bc8030, poll_in=1, poll_out=4, poll_err=0)
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:2356
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:2356.
>>>>
>>>> (gdb) #9  0xbb7a572e in event_dispatch_poll_handler
>>>> (event_pool=0xbb143030, ufds=0xbb18f370, i=9)
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:393
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:393.
>>>>
>>>> (gdb) #10 0xbb7a5a75 in event_dispatch_poll (event_pool=0xbb143030)
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:489
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:489.
>>>>
>>>> (gdb) #11 0xbb774a35 in event_dispatch (event_pool=0xbb143030)
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event.c:128
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event.c:128.
>>>>
>>>> (gdb) #12 0x08050e20 in main (argc=1, argv=0xbf7febdc)
>>>> (gdb)     at
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/glusterfsd/src/glusterfsd.c:2350
>>>>
>>>> Illegal process-id:
>>>>
>>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/glusterfsd/src/glusterfsd.c:2350.
>>>>
>>>>
>>>> Pranith
>>
>>