[Bugs] [Bug 1363613] New: Crash of glusterd when force-creating geo-replication

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1363613

            Bug ID: 1363613
           Summary: Crash of glusterd when force-creating geo-replication
           Product: GlusterFS
           Version: 3.7.14
         Component: geo-replication
          Severity: medium
          Assignee: bugs at gluster.org
          Reporter: bugzilla at ii.nl
                CC: bugs at gluster.org



Description of problem:

I have some existing data on the slave that I'm going to use for geo-rep, this
in the hope that I don't have to transfer 400G of data over geo-rep (the data
is already available at the location of the slave, just not in gluster)

Following this manual:

https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3.1/html/Administration_Guide/sect-Preparing_to_Deploy_Geo-replication.html#Setting_Up_the_Environment_for_a_Secure_Geo-replication_Slave

the crash happens at step 9.

This is (probably) expected:

root at gluster-3:/home/mrten# gluster volume geo-replication gl0
georeplication at gluster-4.glstr::glbackup create push-pem
gluster-4.glstr::glbackup is not empty. Please delete existing files in
gluster-4.glstr::glbackup and retry, or use force to continue without deleting
the existing files.
geo-replication command failed

So force it:

root at gluster-3:/home/mrten# gluster volume geo-replication gl0
georeplication at gluster-4.glstr::glbackup create push-pem force
Connection failed. Please check if gluster daemon is operational.
geo-replication command failed

At this stage, there is a crash log in
/var/log/glusterfs/etc-glusterfs-glusterd.vol.log:

pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash:
2016-08-03 08:00:49
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.14
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f6db19d5a32]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f6db19facdd]
/lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f6db0dd3cb0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7f6db0dd3c37]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f6db0dd7028]
/lib/x86_64-linux-gnu/libc.so.6(+0x732a4)[0x7f6db0e102a4]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f6db0ea7bbc]
/lib/x86_64-linux-gnu/libc.so.6(+0x109a90)[0x7f6db0ea6a90]
/lib/x86_64-linux-gnu/libc.so.6(__stpncpy_chk+0x0)[0x7f6db0ea5ef0]
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.14/xlator/mgmt/glusterd.so(+0xc5d6b)[0x7f6dacf83d6b]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach_match+0x77)[0x7f6db19cf8a7]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_foreach+0x18)[0x7f6db19cfa18]
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.14/xlator/mgmt/glusterd.so(glusterd_op_stage_gsync_create+0x1cea)[0x7f6dacf92faa]
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.14/xlator/mgmt/glusterd.so(glusterd_op_stage_validate+0xdb)[0x7f6dacf2184b]
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.14/xlator/mgmt/glusterd.so(gd_stage_op_phase+0x16a)[0x7f6dacfb20ea]
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.14/xlator/mgmt/glusterd.so(gd_sync_task_begin+0x6de)[0x7f6dacfb3bbe]
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.14/xlator/mgmt/glusterd.so(glusterd_op_begin_synctask+0x30)[0x7f6dacfb3ef0]
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.14/xlator/mgmt/glusterd.so(__glusterd_handle_gsync_set+0x628)[0x7f6dacf871b8]
/usr/lib/x86_64-linux-gnu/glusterfs/3.7.14/xlator/mgmt/glusterd.so(glusterd_big_locked_handler+0x30)[0x7f6dacf0c240]
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_wrap+0x12)[0x7f6db1a232d2]
/lib/x86_64-linux-gnu/libc.so.6(+0x49800)[0x7f6db0de6800]

and glusterd is gone.

These are the log messages just before the crash, perhaps related:

[2016-08-03 08:00:49.674995] I [MSGID: 106316]
[glusterd-geo-rep.c:3096:glusterd_op_stage_gsync_create] 0-management:
georeplication at gluster-4.glstr::glbackup is not a valid slave volume. Error:
gluster-4.glstr::glbackup is not empty. Please delete existing files in
gluster-4.glstr::glbackup and retry, or use force to continue without deleting
the existing files.. Force creating geo-rep session.
[2016-08-03 08:00:49.675032] W [MSGID: 106029]
[glusterd-geo-rep.c:2522:glusterd_get_statefile_name] 0-management: Config file
(/var/lib/glusterd/geo-replication/gl0_gluster-4.glstr_glbackup/gsyncd.conf)
missing. Looking for template config file
(/var/lib/glusterd/geo-replication/gsyncd_template.conf) [No such file or
directory]
[2016-08-03 08:00:49.675048] I [MSGID: 106294]
[glusterd-geo-rep.c:2531:glusterd_get_statefile_name] 0-management: Using
default config
template(/var/lib/glusterd/geo-replication/gsyncd_template.conf).



Version-Release number of selected component (if applicable):
3.7.14 but saw it in 3.7.13 as well.

How reproducible:
Every time

Additional info:

This is on Ubuntu 14.04, using the gluster PPA, kernel 3.13.0-92-generic.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.