[Gluster-devel] crash in '_Unwind_Backtrace () from ./lib64/libgcc_s.so.1'

[Soumya Koduri]

Hi,

I have raised BZ#1293594 to get inputs from gcc team to further debug 
the crash we have been seeing (especially with the test 
-./tests/bugs/snapshot/bug-1140162-file-snapshot-features-encrypt-opts-validation.t 
).
If anyone happen to run into this issue again, kindly install 
gcc-debuginfo package on that machine to get full backtrace (as 
requested in [1]) and update the bug with details.

Thanks,
Soumya

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1293594#c4

On 11/26/2015 03:07 PM, Soumya Koduri wrote:
> Below are the findings from the core and the logs
>
> 1) [2015-11-25 19:06:41.592905] E
> [crypt.c:4298:master_set_master_vol_key] 0-patchy-crypt: FATAL: missing
> master key
>
> xlator_init() of crypt xlator fails, which I assume gets loaded when
> features.encryption is on (which the below mentioned .t test does)
>
> Request anyone familiar with crypt xlator to take a look at it.
>
> This results in the shutdown of NFS (glusterfsd) process calling
> cleanup_an_exit()
>
>
> 2) There is a crash in libgcc
>
> Thread 1 (LWP 11584):
> #0  0x00007f922ad34867 in ?? () from ./lib64/libgcc_s.so.1
> #1  0x00007f922ad35119 in _Unwind_Backtrace () from ./lib64/libgcc_s.so.1
> #2  0x00007f923660b936 in backtrace () from ./lib64/libc.so.6
> ---Type <return> to continue, or q <return> to quit---
> #3  0x00007f92379abf73 in _gf_msg_backtrace_nomem (level=GF_LOG_ALERT,
> stacksize=200)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/logging.c:1090
>
> #4  0x00007f92379b1d38 in gf_print_trace (signum=11, ctx=0x2331010)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/common-utils.c:740
>
> #5  0x00000000004098d6 in glusterfsd_print_trace (signum=11)
>      at
> /home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd.c:2033
>
> #6  <signal handler called>
> #7  0x00007f9229eb8cec in ?? ()
> #8  0x00007f9236c8ba51 in start_thread () from ./lib64/libpthread.so.0
> #9  0x00007f92365f593d in clone () from ./lib64/libc.so.6
> (gdb)
>
> This looks similar to the issue pointed earlier by Kotresh [1]
>
> As mentioned in the last mail of that thread, [2] doesn't seem to have
> fixed it completely.
>
> [1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10866
> [2] http://review.gluster.org/#/c/10417/
>
>
> Thanks,
> Soumya
>
> On 11/26/2015 10:43 AM, Nithya Balachandran wrote:
>> Hi,
>>
>>
>> The test
>> tests/bugs/snapshot/bug-1140162-file-snapshot-features-encrypt-opts-validation.t
>> has failed with a core.Can you please take a look?
>>
>>
>> The NFS log says:
>> gluster/02c803ff8630bd12cc8dc9dc043a6103.socket)
>> [2015-11-25 19:06:41.561937] I [MSGID: 101190]
>> [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started
>> thread with index 1
>> [2015-11-25 19:06:41.566469] I
>> [rpcsvc.c:2211:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
>> Configured rpc.outstanding-rpc-limit with value 16
>> [2015-11-25 19:06:41.568896] I [MSGID: 112153]
>> [mount3.c:3924:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled!
>> [2015-11-25 19:06:41.592393] I [rpc-drc.c:689:rpcsvc_drc_init]
>> 0-rpc-service: DRC is turned OFF
>> [2015-11-25 19:06:41.592441] I [MSGID: 112110] [nfs.c:1513:init]
>> 0-nfs: NFS service started
>> [2015-11-25 19:06:41.592905] E
>> [crypt.c:4298:master_set_master_vol_key] 0-patchy-crypt: FATAL:
>> missing master key
>> [2015-11-25 19:06:41.592928] E [MSGID: 101019]
>> [xlator.c:429:xlator_init] 0-patchy-crypt: Initialization of volume
>> 'patchy-crypt' failed, review your volfile again
>> [2015-11-25 19:06:41.592953] E [MSGID: 101066]
>> [graph.c:324:glusterfs_graph_init] 0-patchy-crypt: initializing
>> translator failed
>> [2015-11-25 19:06:41.592976] E [MSGID: 101176]
>> [graph.c:670:glusterfs_graph_activate] 0-graph: init failed
>> pending frames:
>> frame : type(0) op(0)
>> patchset: git://git.gluster.com/glusterfs.git
>> signal received: 11
>> time of crash:
>> 2015-11-25 19:06:41
>> configuration details:
>> argp 1
>> backtrace 1
>> dlfcn 1
>> libpthread 1
>> llistxattr 1
>> setfsid 1
>> pending frames:
>> frame : type(0) op(0)
>> patchset: git://git.gluster.com/glusterfs.git
>> signal received: 11
>> time of crash:
>> 2015-11-25 19:06:41
>> configuration details:
>> argp 1
>> backtrace 1
>> dlfcn 1
>> libpthread 1
>> llistxattr 1
>> setfsid 1
>> spinlock 1
>> epoll.h 1
>> xattr.h 1
>> st_atim.tv_nsec 1
>> package-string: glusterfs 3.8dev
>> [2015-11-25 19:06:41.594542] W [glusterfsd.c:1231:cleanup_and_exit]
>> (-->/build/install/sbin/glusterfs(mgmt_getspec_cbk+0x34d) [0x40e71d]
>> -->/b
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel