[Gluster-users] GlusterFS 4.1.3, Geo replication unable to setup
Kotresh Hiremath Ravishankar
khiremat at redhat.com
Thu Sep 6 10:58:37 UTC 2018
Hi Nico,
The glusterd has crashed on this node. Please raise a bug with core file?
Please use the following tool [1] to setup geo-rep by bringing back the
glusterd
if you are finding it difficult with geo-rep setup steps and let us know if
if it still crashes?
[1] http://aravindavk.in/blog/introducing-georepsetup/
On Thu, Sep 6, 2018 at 2:54 PM, Nico van Royen <nico at van-royen.nl> wrote:
> Hello,
>
> On our dev environment we want to test GeoReplication with GlusterFS 4.1
> and every attempt so far fails.
> For now, we don't care (yet) about running it as a non-root user (not
> using the mountbroker etc).
>
> Installed packages, both on master and slaves:
> [root at clrv0000110367 geo-replication]# rpm -qa | grep gluster
> glusterfs-client-xlators-4.1.3-1.el7.x86_64
> glusterfs-events-4.1.3-1.el7.x86_64
> glusterfs-geo-replication-4.1.3-1.el7.x86_64
> glusterfs-4.1.3-1.el7.x86_64
> glusterfs-api-4.1.3-1.el7.x86_64
> glusterfs-fuse-4.1.3-1.el7.x86_64
> glusterfs-server-4.1.3-1.el7.x86_64
> glusterfs-rdma-4.1.3-1.el7.x86_64
> glusterfs-extra-xlators-4.1.3-1.el7.x86_64
> glusterfs-libs-4.1.3-1.el7.x86_64
> glusterfs-cli-4.1.3-1.el7.x86_64
> python2-gluster-4.1.3-1.el7.x86_64
> glusterfs-coreutils-0.2.0-1.el7.x86_64
>
> Master volume setup:
> # gluster v create VOLUME2 replica 3 arbiter 1 transport tcp
> clrv0000110367:/gluster/VOLUME2/export clrv0000110371:/gluster/VOLUME2/export
> clrv0000110389:/gluster/VOLUME2/export
> # gluster v start VOLUME2
> # gluster volume set all cluster.enable-shared-storage enable
>
> Slave volume setup
> # gluster v create VOLUME2 replica 3 arbiter 1 transport tcp
> clrv0000110605:/gluster/VOLUME2/export clrv0000110608:/gluster/VOLUME2/export
> clrv0000110606:/gluster/VOLUME2/export
> # gluster v start VOLUME2
> # gluster volume set all cluster.enable-shared-storage enable
>
> On master server:
> # ssh-keygen (accepting all defaults)
> # ssh-copy-id clrv0000110605 (one of the slave servers)
> # gluster-georep-sshkey generate
> # gluster volume geo-replication VOLUME2 clrv0000110605.ic.ing.net::VOLUME2
> create push-pem
>
> Several seconds later, all of the glusterd instances on the master side
> crash, with /var/log/glusterfs/glusterd.log such as:
>
> [2018-09-06 08:50:20.663584] W [MSGID: 106028] [glusterd-geo-rep.c:2568:glusterd_get_statefile_name]
> 0-management: Config file (/var/lib/glusterd/geo-replication/VOLUME2_
> clrv0000110605_VOLUME2/gsyncd.conf) missing. Looking for template config
> file (/var/lib/glusterd/geo-replication/gsyncd_template.conf) [No such
> file or directory]
> [2018-09-06 08:50:20.663724] I [MSGID: 106294] [glusterd-geo-rep.c:2577:glusterd_get_statefile_name]
> 0-management: Using default config template(/var/lib/glusterd/
> geo-replication/gsyncd_template.conf).
> [2018-09-06 08:50:24.072321] I [MSGID: 106494] [glusterd-handler.c:3024:__
> glusterd_handle_cli_profile_volume] 0-management: Received volume profile
> req for volume VOLUME1
> [2018-09-06 08:50:24.074876] I [MSGID: 106487] [glusterd-handler.c:1486:__
> glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
> [2018-09-06 08:50:24.744276] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop]
> 0-management: nfs already stopped
> [2018-09-06 08:50:24.744443] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop]
> 0-management: nfs service is stopped
> [2018-09-06 08:50:24.744497] I [MSGID: 106599] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager]
> 0-management: nfs/server.so xlator is not installed
> [2018-09-06 08:50:24.749139] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop]
> 0-management: Stopping glustershd daemon running in pid: 40886
> [2018-09-06 08:50:25.749748] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop]
> 0-management: glustershd service is stopped
> [2018-09-06 08:50:25.750047] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start]
> 0-management: Starting glustershd service
> [2018-09-06 08:50:25.757036] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop]
> 0-management: bitd already stopped
> [2018-09-06 08:50:25.757100] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop]
> 0-management: bitd service is stopped
> [2018-09-06 08:50:25.757288] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop]
> 0-management: scrub already stopped
> [2018-09-06 08:50:25.757330] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop]
> 0-management: scrub service is stopped
> [2018-09-06 08:50:28.391332] I [run.c:241:runner_log]
> (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a)
> [0x7fae33053b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5)
> [0x7fae330535e5] -->/lib64/libglusterfs.so.0(runner_log+0x115)
> [0x7fae3e55f0c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/
> gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2
> is_push_pem=1,pub_file=/var/lib/glusterd/geo-replication/
> common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol=
> VOLUME2,ssh_port=22
> [2018-09-06 08:50:31.901854] I [MSGID: 106327] [glusterd-geo-rep.c:2562:glusterd_get_statefile_name]
> 0-management: Using passed config template(/var/lib/glusterd/
> geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).
> [2018-09-06 08:50:32.252426] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors]
> 0-glusterd: Staging failed on clrv0000110389.ic.ing.net. Error:
> Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does
> not exist.
> [2018-09-06 08:50:32.271707] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors]
> 0-glusterd: Staging failed on clrv0000110371.ic.ing.net. Error:
> Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does
> not exist.
> [2018-09-06 08:50:32.332909] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors]
> 0-glusterd: Staging failed on clrv0000110372.ic.ing.net. Error:
> Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does
> not exist.
> [2018-09-06 08:50:35.690440] E [mem-pool.c:335:__gf_free]
> (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e)
> [0x7fae32f9536e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10)
> [0x7fae32f91e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104)
> [0x7fae3e5344f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t
> *)((char *)free_ptr + header->size)
> The message "I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state]
> 0-management: Received request to get state for glusterd"
> repeated 6 times between [2018-09-06 08:49:02.500390] and [2018-09-06
> 08:50:35.690241]
> [2018-09-06 08:50:35.846286] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file]
> 0-management: Using passed config template(/var/lib/glusterd/
> geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).
> [2018-09-06 08:50:36.144575] E [mem-pool.c:326:__gf_free]
> (-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fae3e4fe2c0]
> -->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fae3e4fd92d]
> -->/lib64/libglusterfs.so.0(__gf_free+0xa4) [0x7fae3e534494] ) 0-:
> Assertion failed: GF_MEM_HEADER_MAGIC ==
> header->magic
> [2018-09-06 08:50:11.549220] E [MSGID: 106332] [glusterd-utils.c:12886:
> glusterd_get_value_for_vme_entry] 0-management: Failed to get option for
> xattr-user-namespace-mode key
> pending frames:
> frame : type(0) op(0)
> patchset: git://git.gluster.org/glusterfs.git
> signal received: 6
> time of crash:
> 2018-09-06 08:50:36
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 4.1.3
> /lib64/libglusterfs.so.0(+0x25920)[0x7fae3e509920]
> /lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fae3e513874]
> /lib64/libc.so.6(+0x36280)[0x7fae3cb6e280]
> /lib64/libc.so.6(gsignal+0x37)[0x7fae3cb6e207]
> /lib64/libc.so.6(abort+0x148)[0x7fae3cb6f8f8]
> /lib64/libc.so.6(+0x78cc7)[0x7fae3cbb0cc7]
> /lib64/libc.so.6(+0x81429)[0x7fae3cbb9429]
> /lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fae3e4fd92d]
> /lib64/libglusterfs.so.0(+0x1a2c0)[0x7fae3e4fe2c0]
> /usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+
> 0x22254)[0x7fae32f93254]
> /usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+
> 0x2436e)[0x7fae32f9536e]
> /lib64/libglusterfs.so.0(+0x622b0)[0x7fae3e5462b0]
> /lib64/libc.so.6(+0x47fc0)[0x7fae3cb7ffc0]
> ---------
>
> Note that the gsyncd_template.conf file (as found in /var/lib/glusterd/geo-replication)
> is empty, and was (re)created when the gluster volume geo-replication
> command was issued. (is it supposed to be empty ?)
>
> Anyone any idea's ?
>
> Regards, Nico van Roijen
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
--
Thanks and Regards,
Kotresh H R
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180906/f5473a7f/attachment.html>
More information about the Gluster-users
mailing list