[Bugs] [Bug 1211640] New: [SNAPSHOT]: glusterd crash when snapshot create was in progress on different volumes at the same time - job edited to create snapshots at the given time
bugzilla at redhat.com
bugzilla at redhat.com
Tue Apr 14 13:32:04 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1211640
Bug ID: 1211640
Summary: [SNAPSHOT]: glusterd crash when snapshot create was in
progress on different volumes at the same time - job
edited to create snapshots at the given time
Product: GlusterFS
Version: mainline
Component: snapshot
Severity: high
Assignee: bugs at gluster.org
Reporter: senaik at redhat.com
CC: bugs at gluster.org, gluster-bugs at redhat.com
Description of problem:
=======================
Edited 2 jobs to create snapshots on 2 different volumes at the same time, snap
create failed as glusterd crashed
Version-Release number of selected component (if applicable):
=============================================================
gluster --version
glusterfs 3.7dev built on Apr 13 2015 07:14:27
How reproducible:
=================
1/1
Steps to Reproduce:
===================
1.Create 2 volumes (vol0 and vol1)and start it.
Fuse and NFS mount the volume and create some IO
2.Create another shared storage and fuse mount the shared storage on all nodes
3.Initialise the snapshot scheduler on all nodes using snap_scheduler.py init
4.Enable the snap scheduler on one of the nodes using snap_scheduler.py enable
5.Added a job which created snapshot on vol0 at 18:20
Added another job which created snapshot on vol1 at 18:20
snap_scheduler.py add "J1_vol0" "20 18 * * * " "vol0"
snap_scheduler.py add "J1_vol1" "20 18 * * * " "vol1"
Snapshots were created successfully on the volume
gluster snapshot list
Scheduled-J1_vol0-vol0_GMT-2015.04.14-12.50.01
Scheduled-J1_vol1-vol1_GMT-2015.04.14-12.50.01
6.Edit both the jobs to create snapshots at 18:36
snap_scheduler.py edit "J1_vol1" "36 18 * * * " "vol1"
snap_scheduler.py edit "J1_vol0" "36 18 * * * " "vol0"
Snapshot creation failed
-------------Part of var/log/glusterfs/gcron.log----------
[2015-04-14 18:36:01,350 gcron.py:67 takeSnap] DEBUG Running command 'gluster
snapshot create Scheduled-J1_vol0-vol0 vol0'
[2015-04-14 18:36:01,351 gcron.py:95 doJob] DEBUG
/var/run/gluster/shared_storage/snaps/lock_files/J1_vol1 last modified at Tue
Apr 14 18:20:26 2015
[2015-04-14 18:36:01,351 gcron.py:97 doJob] DEBUG Processing job
Scheduled-J1_vol1-vol1
[2015-04-14 18:36:01,352 gcron.py:67 takeSnap] DEBUG Running command 'gluster
snapshot create Scheduled-J1_vol1-vol1 vol1'
[2015-04-14 18:36:20,009 gcron.py:74 takeSnap] DEBUG Command 'gluster snapshot
create Scheduled-J1_vol0-vol0 vol0' returned '1'
[2015-04-14 18:36:20,009 gcron.py:74 takeSnap] DEBUG Command 'gluster snapshot
create Scheduled-J1_vol1-vol1 vol1' returned '1'
[2015-04-14 18:36:20,010 gcron.py:77 takeSnap] ERROR Snapshot of vol0 failed
[2015-04-14 18:36:20,014 gcron.py:78 takeSnap] ERROR Command output:
[2015-04-14 18:36:20,014 gcron.py:79 takeSnap] ERROR
[2015-04-14 18:36:20,014 gcron.py:101 doJob] ERROR Job Scheduled-J1_vol0-vol0
failed
[2015-04-14 18:36:20,010 gcron.py:77 takeSnap] ERROR Snapshot of vol1 failed
[2015-04-14 18:36:20,019 gcron.py:78 takeSnap] ERROR Command output:
[2015-04-14 18:36:20,020 gcron.py:79 takeSnap] ERROR
[2015-04-14 18:36:20,020 gcron.py:101 doJob] ERROR Job Scheduled-J1_vol1-vol1
failed
------------------------------------------------------------
Actual results:
==============
glusterd crashed
Expected results:
================
There should be no crash observed
Additional info:
================
[2015-04-14 12:38:44.527287] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/7089eb2213ea459a8a12ba
56023bd163.socket failed (No data available)
[2015-04-14 12:38:47.918396] W [socket.c:642:__socket_rwv] 0-quotad: readv on
/var/run/gluster/7089eb2213ea459a8a12ba
56023bd163.socket failed (No data available)
The message "I [MSGID: 106006]
[glusterd-snapd-svc.c:379:glusterd_snapdsvc_rpc_notify] 0-management: snapd has
discon
nected from glusterd." repeated 2 times between [2015-04-14 12:38:34.733354]
and [2015-04-14 12:38:40.636734]
The message "I [MSGID: 106006]
[glusterd-svc-mgmt.c:327:glusterd_svc_common_rpc_notify] 0-management: nfs has
disconn
ected from glusterd." repeated 3 times between [2015-04-14 12:37:56.254712] and
[2015-04-14 12:38:40.641386]
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash:
2015-04-14 13:06:18
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7dev
(gdb) bt
#0 0x00007fe862f85d16 in rcu_read_unlock_bp () from /usr/lib64/liburcu-bp.so.1
#1 0x00007fe863283252 in glusterd_mgmt_v3_commit (op=GD_OP_SNAP,
op_ctx=0x7fe800000003, req_dict=0x7fe85c344c4c, op_errstr=0x7fe85879ace8,
txn_generation=3)
at glusterd-mgmt.c:1232
#2 0x00007fe863287078 in glusterd_mgmt_v3_initiate_snap_phases (req=0x1345108,
op=GD_OP_SNAP, dict=0x7fe85c45d87c) at glusterd-mgmt.c:1998
#3 0x00007fe863272a10 in glusterd_handle_snapshot_create (req=0x1345108,
op=GD_OP_SNAP, dict=0x7fe85c45d87c, err_str=<value optimized out>,
len=140635893514400)
at glusterd-snapshot.c:3763
#4 0x00007fe86327e7c1 in glusterd_handle_snapshot_fn (req=0x1345108) at
glusterd-snapshot.c:8305
#5 0x00007fe8631c9d7f in glusterd_big_locked_handler (req=0x1345108,
actor_fn=0x7fe86327dfa0 <glusterd_handle_snapshot_fn>) at glusterd-handler.c:83
#6 0x0000003abac61c72 in synctask_wrap (old_task=<value optimized out>) at
syncop.c:375
#7 0x0000003a964438f0 in ?? () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list