[Gluster-users] GlusterFS 4.1.3, Geo replication unable to setup

Thu Sep 6 09:24:14 UTC 2018

Hello, 

On our dev environment we want to test GeoReplication with GlusterFS 4.1 and every attempt so far fails. 
For now, we don't care (yet) about running it as a non-root user (not using the mountbroker etc). 

Installed packages, both on master and slaves: 
[root at clrv0000110367 geo-replication]# rpm -qa | grep gluster 
glusterfs-client-xlators-4.1.3-1.el7.x86_64 
glusterfs-events-4.1.3-1.el7.x86_64 
glusterfs-geo-replication-4.1.3-1.el7.x86_64 
glusterfs-4.1.3-1.el7.x86_64 
glusterfs-api-4.1.3-1.el7.x86_64 
glusterfs-fuse-4.1.3-1.el7.x86_64 
glusterfs-server-4.1.3-1.el7.x86_64 
glusterfs-rdma-4.1.3-1.el7.x86_64 
glusterfs-extra-xlators-4.1.3-1.el7.x86_64 
glusterfs-libs-4.1.3-1.el7.x86_64 
glusterfs-cli-4.1.3-1.el7.x86_64 
python2-gluster-4.1.3-1.el7.x86_64 
glusterfs-coreutils-0.2.0-1.el7.x86_64 

Master volume setup: 
# gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110367:/gluster/VOLUME2/export clrv0000110371:/gluster/VOLUME2/export clrv0000110389:/gluster/VOLUME2/export 
# gluster v start VOLUME2 
# gluster volume set all cluster.enable-shared-storage enable 

Slave volume setup 
# gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110605:/gluster/VOLUME2/export clrv0000110608:/gluster/VOLUME2/export clrv0000110606:/gluster/VOLUME2/export 
# gluster v start VOLUME2 
# gluster volume set all cluster.enable-shared-storage enable 

On master server: 
# ssh-keygen (accepting all defaults) 
# ssh-copy-id clrv0000110605 (one of the slave servers) 
# gluster-georep-sshkey generate 
# gluster volume geo-replication VOLUME2 clrv0000110605.ic.ing.net::VOLUME2 create push-pem 

Several seconds later, all of the glusterd instances on the master side crash, with /var/log/glusterfs/glusterd.log such as: 

[2018-09-06 08:50:20.663584] W [MSGID: 106028] [glusterd-geo-rep.c:2568:glusterd_get_statefile_name] 0-management: Config file (/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf) missing. Looking for template config file (/var/lib/glusterd/geo-replication/gsyncd_template.conf) [No such file or directory] 
[2018-09-06 08:50:20.663724] I [MSGID: 106294] [glusterd-geo-rep.c:2577:glusterd_get_statefile_name] 0-management: Using default config template(/var/lib/glusterd/geo-replication/gsyncd_template.conf). 
[2018-09-06 08:50:24.072321] I [MSGID: 106494] [glusterd-handler.c:3024:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume VOLUME1 
[2018-09-06 08:50:24.074876] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req 
[2018-09-06 08:50:24.744276] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped 
[2018-09-06 08:50:24.744443] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is stopped 
[2018-09-06 08:50:24.744497] I [MSGID: 106599] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed 
[2018-09-06 08:50:24.749139] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 40886 
[2018-09-06 08:50:25.749748] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd service is stopped 
[2018-09-06 08:50:25.750047] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting glustershd service 
[2018-09-06 08:50:25.757036] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped 
[2018-09-06 08:50:25.757100] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: bitd service is stopped 
[2018-09-06 08:50:25.757288] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped 
[2018-09-06 08:50:25.757330] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: scrub service is stopped 
[2018-09-06 08:50:28.391332] I [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a) [0x7fae33053b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5) [0x7fae330535e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fae3e55f0c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2 is_push_pem=1,pub_file=/var/lib/glusterd/geo-replication/common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol=VOLUME2,ssh_port=22 
[2018-09-06 08:50:31.901854] I [MSGID: 106327] [glusterd-geo-rep.c:2562:glusterd_get_statefile_name] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf). 
[2018-09-06 08:50:32.252426] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on clrv0000110389.ic.ing.net. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist. 
[2018-09-06 08:50:32.271707] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on clrv0000110371.ic.ing.net. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist. 
[2018-09-06 08:50:32.332909] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on clrv0000110372.ic.ing.net. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist. 
[2018-09-06 08:50:35.690440] E [mem-pool.c:335:__gf_free] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e) [0x7fae32f9536e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10) [0x7fae32f91e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104) [0x7fae3e5344f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size) 
The message "I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state] 0-management: Received request to get state for glusterd" 
repeated 6 times between [2018-09-06 08:49:02.500390] and [2018-09-06 08:50:35.690241] 
[2018-09-06 08:50:35.846286] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf). 
[2018-09-06 08:50:36.144575] E [mem-pool.c:326:__gf_free] (-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fae3e4fe2c0] -->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fae3e4fd92d] -->/lib64/libglusterfs.so.0(__gf_free+0xa4) [0x7fae3e534494] ) 0-: Assertion failed: GF_MEM_HEADER_MAGIC == 
header->magic 
[2018-09-06 08:50:11.549220] E [MSGID: 106332] [glusterd-utils.c:12886:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key 
pending frames: 
frame : type(0) op(0) 
patchset: git://git.gluster.org/glusterfs.git 
signal received: 6 
time of crash: 
2018-09-06 08:50:36 
configuration details: 
argp 1 
backtrace 1 
dlfcn 1 
libpthread 1 
llistxattr 1 
setfsid 1 
spinlock 1 
epoll.h 1 
xattr.h 1 
st_atim.tv_nsec 1 
package-string: glusterfs 4.1.3 
/lib64/libglusterfs.so.0(+0x25920)[0x7fae3e509920] 
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fae3e513874] 
/lib64/libc.so.6(+0x36280)[0x7fae3cb6e280] 
/lib64/libc.so.6(gsignal+0x37)[0x7fae3cb6e207] 
/lib64/libc.so.6(abort+0x148)[0x7fae3cb6f8f8] 
/lib64/libc.so.6(+0x78cc7)[0x7fae3cbb0cc7] 
/lib64/libc.so.6(+0x81429)[0x7fae3cbb9429] 
/lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fae3e4fd92d] 
/lib64/libglusterfs.so.0(+0x1a2c0)[0x7fae3e4fe2c0] 
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x22254)[0x7fae32f93254] 
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e)[0x7fae32f9536e] 
/lib64/libglusterfs.so.0(+0x622b0)[0x7fae3e5462b0] 
/lib64/libc.so.6(+0x47fc0)[0x7fae3cb7ffc0] 
--------- 

Note that the gsyncd_template.conf file (as found in /var/lib/glusterd/geo-replication) is empty, and was (re)created when the gluster volume geo-replication command was issued. (is it supposed to be empty ?) 

Anyone any idea's ? 

Regards, Nico van Roijen 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180906/f3d1159b/attachment.html>