[Gluster-devel] Upstream regression crash : https://build.gluster.org/job/rackspace-regression-2GB-triggered/16191/consoleFull

[Soumya Koduri]

Below are the findings from the core and the logs

1) [2015-11-25 19:06:41.592905] E 
[crypt.c:4298:master_set_master_vol_key] 0-patchy-crypt: FATAL: missing 
master key

xlator_init() of crypt xlator fails, which I assume gets loaded when 
features.encryption is on (which the below mentioned .t test does)

Request anyone familiar with crypt xlator to take a look at it.

This results in the shutdown of NFS (glusterfsd) process calling 
cleanup_an_exit()


2) There is a crash in libgcc

Thread 1 (LWP 11584):
#0  0x00007f922ad34867 in ?? () from ./lib64/libgcc_s.so.1
#1  0x00007f922ad35119 in _Unwind_Backtrace () from ./lib64/libgcc_s.so.1
#2  0x00007f923660b936 in backtrace () from ./lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
#3  0x00007f92379abf73 in _gf_msg_backtrace_nomem (level=GF_LOG_ALERT, 
stacksize=200)
     at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/logging.c:1090
#4  0x00007f92379b1d38 in gf_print_trace (signum=11, ctx=0x2331010)
     at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/libglusterfs/src/common-utils.c:740
#5  0x00000000004098d6 in glusterfsd_print_trace (signum=11)
     at 
/home/jenkins/root/workspace/rackspace-regression-2GB-triggered/glusterfsd/src/glusterfsd.c:2033
#6  <signal handler called>
#7  0x00007f9229eb8cec in ?? ()
#8  0x00007f9236c8ba51 in start_thread () from ./lib64/libpthread.so.0
#9  0x00007f92365f593d in clone () from ./lib64/libc.so.6
(gdb)

This looks similar to the issue pointed earlier by Kotresh [1]

As mentioned in the last mail of that thread, [2] doesn't seem to have 
fixed it completely.

[1] http://thread.gmane.org/gmane.comp.file-systems.gluster.devel/10866
[2] http://review.gluster.org/#/c/10417/


Thanks,
Soumya

On 11/26/2015 10:43 AM, Nithya Balachandran wrote:
> Hi,
>
>
> The test tests/bugs/snapshot/bug-1140162-file-snapshot-features-encrypt-opts-validation.t has failed with a core.Can you please take a look?
>
>
> The NFS log says:
> gluster/02c803ff8630bd12cc8dc9dc043a6103.socket)
> [2015-11-25 19:06:41.561937] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
> [2015-11-25 19:06:41.566469] I [rpcsvc.c:2211:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 16
> [2015-11-25 19:06:41.568896] I [MSGID: 112153] [mount3.c:3924:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled!
> [2015-11-25 19:06:41.592393] I [rpc-drc.c:689:rpcsvc_drc_init] 0-rpc-service: DRC is turned OFF
> [2015-11-25 19:06:41.592441] I [MSGID: 112110] [nfs.c:1513:init] 0-nfs: NFS service started
> [2015-11-25 19:06:41.592905] E [crypt.c:4298:master_set_master_vol_key] 0-patchy-crypt: FATAL: missing master key
> [2015-11-25 19:06:41.592928] E [MSGID: 101019] [xlator.c:429:xlator_init] 0-patchy-crypt: Initialization of volume 'patchy-crypt' failed, review your volfile again
> [2015-11-25 19:06:41.592953] E [MSGID: 101066] [graph.c:324:glusterfs_graph_init] 0-patchy-crypt: initializing translator failed
> [2015-11-25 19:06:41.592976] E [MSGID: 101176] [graph.c:670:glusterfs_graph_activate] 0-graph: init failed
> pending frames:
> frame : type(0) op(0)
> patchset: git://git.gluster.com/glusterfs.git
> signal received: 11
> time of crash:
> 2015-11-25 19:06:41
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> libpthread 1
> llistxattr 1
> setfsid 1
> pending frames:
> frame : type(0) op(0)
> patchset: git://git.gluster.com/glusterfs.git
> signal received: 11
> time of crash:
> 2015-11-25 19:06:41
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 3.8dev
> [2015-11-25 19:06:41.594542] W [glusterfsd.c:1231:cleanup_and_exit] (-->/build/install/sbin/glusterfs(mgmt_getspec_cbk+0x34d) [0x40e71d] -->/b
>