[Gluster-devel] core while running tests/bugs/snapshot/bug-1112559.t

[Avra Sengupta]

Hi,

Was analysing this failure in 
http://build.gluster.org/job/rackspace-regression-2GB-triggered/8571/consoleFull 
. Analyzing the core it looks like the mount process was unable to 
process the volfile and ended up calling cleanup_and_exit() from a 
thread. Looking at the logs we can see that the process had a socket 
failure which resulted in failure to initialize the volume and 
ultimately it crashed. Can someone shed some more light into this if 
they have context.

Below are the mount-logs:

[2015-05-06 21:08:40.488468] E [socket.c:823:__socket_server_bind] 
0-socket.patchy-changelog: binding to  failed: Address already in use
[2015-05-06 21:08:40.488499] E [socket.c:826:__socket_server_bind] 
0-socket.patchy-changelog: Port is already in use
[2015-05-06 21:08:40.488526] W [rpcsvc.c:1602:rpcsvc_transport_create] 
0-rpc-service: listening on transport failed
[2015-05-06 21:08:40.488637] I [mem-pool.c:587:mem_pool_destroy] 
0-patchy-changelog: size=116 max=0 total=0
[2015-05-06 21:08:40.488662] E [xlator.c:426:xlator_init] 
0-patchy-changelog: Initialization of volume 'patchy-changelog' failed, 
review your volfile again
[2015-05-06 21:08:40.488681] E [graph.c:322:glusterfs_graph_init] 
0-patchy-changelog: initializing translator failed
[2015-05-06 21:08:40.488697] E [graph.c:661:glusterfs_graph_activate] 
0-graph: init failed
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash:
2015-05-06 21:08:40
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.8dev
[2015-05-06 21:08:41.392207] I [login.c:81:gf_auth] 0-auth/login: 
allowed user names: 9114f852-2c38-4dea-a3a5-69cafc2f85d1
[2015-05-06 21:08:41.392246] I [MSGID: 115029] 
[server-handshake.c:593:server_setvolume] 0-patchy-server: accepted 
client from 
slave22.cloud.gluster.org-10927-2015/05/06-21:08:40:486401-patchy-client-1-0-0 
(version: 3.8dev)
[2015-05-06 21:08:41.425871] I [MSGID: 115036] 
[server.c:537:server_rpc_notify] 0-patchy-server: disconnecting 
connection from 
slave22.cloud.gluster.org-10927-2015/05/06-21:08:40:486401-patchy-client-1-0-0
[2015-05-06 21:08:41.425927] I [client_t.c:418:gf_client_unref] 
0-patchy-server: Shutting down connection 
slave22.cloud.gluster.org-10927-2015/05/06-21:08:40:486401-patchy-client-1-0-0
[2015-05-06 21:08:41.435193] I [changelog.c:1950:notify] 
0-patchy-changelog: Barrier off notification
[2015-05-06 21:08:41.435215] W [changelog.c:1952:notify] 
0-patchy-changelog: Changelog is not active, return success
[2015-05-06 21:08:42.587326] I [login.c:81:gf_auth] 0-auth/login: 
allowed user names: 9114f852-2c38-4dea-a3a5-69cafc2f85d1
[2015-05-06 21:08:42.587364] I [MSGID: 115029] 
[server-handshake.c:593:server_setvolume] 0-patchy-server: accepted 
client from 
slave22.cloud.gluster.org-10823-2015/05/06-21:08:38:521224-patchy-client-1-0-0 
(version: 3.8dev)
[2015-05-06 21:08:42.662481] I [login.c:81:gf_auth] 0-auth/login: 
allowed user names: 9114f852-2c38-4dea-a3a5-69cafc2f85d1
[2015-05-06 21:08:42.662516] I [MSGID: 115029] 
[server-handshake.c:593:server_setvolume] 0-patchy-server: accepted 
client from 
slave22.cloud.gluster.org-11019-2015/05/06-21:08:42:434160-patchy-client-1-0-0 
(version: 3.8dev)
[2015-05-06 21:08:43.674553] I [MSGID: 115036] 
[server.c:537:server_rpc_notify] 0-patchy-server: disconnecting 
connection from 
slave22.cloud.gluster.org-10823-2015/05/06-21:08:38:521224-patchy-client-1-0-0
[2015-05-06 21:08:43.674611] I [client_t.c:418:gf_client_unref] 
0-patchy-server: Shutting down connection 
slave22.cloud.gluster.org-10823-2015/05/06-21:08:38:521224-patchy-client-1-0-0
[2015-05-06 21:08:43.675217] W [glusterfsd.c:1219:cleanup_and_exit] (--> 
0-: received signum (15), shutting down

Regards,
Avra

On 05/06/2015 09:36 AM, Joseph Fernandes wrote:
> CCing Venky and Kotresh
>
> ----- Original Message -----
> From: "Jeff Darcy" <jdarcy at redhat.com>
> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> Cc: "Joseph Fernandes" <josferna at redhat.com>, "Avra Sengupta" <asengupt at redhat.com>, "Rajesh Joseph" <rjoseph at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Wednesday, May 6, 2015 8:39:23 AM
> Subject: Re: [Gluster-devel] core while running tests/bugs/snapshot/bug-1112559.t
>
>>           Could you please look at this issue:
>> http://build.gluster.org/job/rackspace-regression-2GB-triggered/8456/consoleFull
> I looked at this one for a while.  It looks like a brick failed to
> start because changelog failed to initialize, but neither the core
> nor the logs shed much light on why.