[Gluster-devel] regarding glusterd crash on NetBSD https://build.gluster.org/job/rackspace-netbsd7-regression-triggered/14100/consoleFull

[Kaushal M]

On Wed, Feb 10, 2016 at 6:08 PM, Pranith Kumar Karampuri
<pkarampu at redhat.com> wrote:
>
>
> On 02/10/2016 06:01 PM, Atin Mukherjee wrote:
>>
>> Not that I am aware of. Do you have backtrace of all the threads?
>
>
> it doesn't seem to give proper output for 'thread apply all bt':
> (gdb) thread apply all bt
>
> Thread 6 (process 2):
> #0  0xbb354977 in ?? ()
> #1  0xbb682b67 in ?? ()
> #2  0xba4fff98 in ?? ()
> Cannot access memory at address 0xba4fffd4
>
> I tried info threads and went to each thread, all of them except the thread
> that crashed has '??' in all frames. You can check the core on the netbsd
> machine mentioned in the console output in subject.
>
> Pranith

We've observed these sorts of crashes (segfaults with stack having a
rpc event happening), before when GlusterD is in the process of
shutting down and exit() has been called. Since we don't correctly
call fini() on xlators, rpcsvc continues accepting requests. But
exit() would have cleaned up the resources that the rpc thread would
have been using, leading to the thread segfaulting.

>
>>
>> ~Atin
>>
>> On 02/10/2016 05:58 PM, Pranith Kumar Karampuri wrote:
>>>
>>> hi Atin, Kaushal,
>>>            Is this a known issue?
>>>
>>> (gdb) #1  0xbb789fb7 in __synclock_unlock (lock=0xbb1d4ac0)
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1056
>>>
>>> #2  0xbb789ffd in synclock_unlock (lock=0xbb1d4ac0)
>>>      at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1071
>>>
>>> #3  0xb9b803ff in glusterd_big_locked_notify (rpc=0xb8bc2070,
>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0,
>>>      notify_fn=0xb9b8ec28 <__glusterd_brick_rpc_notify>)
>>>      at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:72
>>>
>>> #4  0xb9b8f0d7 in glusterd_brick_rpc_notify (rpc=0xb8bc2070,
>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0)
>>>      at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:4993
>>>
>>> #5  0xbb708c22 in rpc_clnt_notify (trans=0xb8bc8030, mydata=0xb8bc2090,
>>> event=RPC_TRANSPORT_DISCONNECT, data=0xb8bc8030)
>>>      at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnt
>>> at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnIllegal
>>> process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1056.
>>>
>>> (gdb) #2  0xbb789ffd in synclock_unlock (lock=0xbb1d4ac0)
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1071
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/syncop.c:1071.
>>>
>>> (gdb) #3  0xb9b803ff in glusterd_big_locked_notify (rpc=0xb8bc2070,
>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0,
>>> (gdb)     notify_fn=0xb9b8ec28 <__glusterd_brick_rpc_notify>)
>>> Undefined command: "notify_fn".  Try "help".
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:72
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:72.
>>>
>>> (gdb) #4  0xb9b8f0d7 in glusterd_brick_rpc_notify (rpc=0xb8bc2070,
>>> mydata=0xb950efa0, event=RPC_CLNT_DISCONNECT, data=0x0)
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:4993
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/xlators/mgmt/glusterd/src/glusterd-handler.c:4993.
>>>
>>> #5  0xbb708c22 in rpc_clnt_notify (trans=0xb8bc8030, mydata=0xb8bc2090,
>>> event=RPC_TRANSPORT_DISCONNECT, data=0xb8bc8030)
>>>
>>> /jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnt
>>> at 8030)
>>>      at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546
>>>
>>> #7  0xbb231847 in socket_event_poll_err (this=0xb8bc8030)
>>>      at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:1151
>>>
>>> #8  0xbb2359a2 in socket_event_handler (fd=15, idx=9, data=0xb8bc8030,
>>> poll_in=1, poll_out=4, poll_err=0)
>>>      at /home/jenkins/root/workspace/rackspace-n(gdb) #5  0xbb708c22 in
>>> rpc_clnt_notify (trans=0xb8bc8030, mydata=0xb8bc2090,
>>> event=RPC_TRANSPORT_DISCONNECT, data=0xb8bc8030)
>>> (gdb)
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-clnt
>>> at 8030)
>>> Undefined command: "".  Try "help".
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-lib/src/rpc-transport.c:546.
>>>
>>> (gdb) #7  0xbb231847 in socket_event_poll_err (this=0xb8bc8030)
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:1151
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:1151.
>>>
>>> (gdb) #8  0xbb2359a2 in socket_event_handler (fd=15, idx=9,
>>> data=0xb8bc8030, poll_in=1, poll_out=4, poll_err=0)
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:2356
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/rpc/rpc-transport/socket/src/socket.c:2356.
>>>
>>> (gdb) #9  0xbb7a572e in event_dispatch_poll_handler
>>> (event_pool=0xbb143030, ufds=0xbb18f370, i=9)
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:393
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:393.
>>>
>>> (gdb) #10 0xbb7a5a75 in event_dispatch_poll (event_pool=0xbb143030)
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:489
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event-poll.c:489.
>>>
>>> (gdb) #11 0xbb774a35 in event_dispatch (event_pool=0xbb143030)
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event.c:128
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/libglusterfs/src/event.c:128.
>>>
>>> (gdb) #12 0x08050e20 in main (argc=1, argv=0xbf7febdc)
>>> (gdb)     at
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/glusterfsd/src/glusterfsd.c:2350
>>>
>>> Illegal process-id:
>>>
>>> /home/jenkins/root/workspace/rackspace-netbsd7-regression-triggered/glusterfsd/src/glusterfsd.c:2350.
>>>
>>>
>>> Pranith
>
>