<div dir="ltr"><div><div><div>Hi Nico,<br><br></div>The glusterd has crashed on this node. Please raise a bug with core file?<br><br></div>Please use the following tool [1] to setup geo-rep by bringing back the glusterd</div><div>if you are finding it difficult with geo-rep setup steps and let us know if if it still crashes?</div><div><br></div><div>[1] <a href="http://aravindavk.in/blog/introducing-georepsetup/" rel="bookmark" title="Permalink to Introducing georepsetup - Gluster Geo-replication Setup Tool">http://aravindavk.in/blog/introducing-georepsetup/</a><br></div><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 6, 2018 at 2:54 PM, Nico van Royen <span dir="ltr"><<a href="mailto:nico@van-royen.nl" target="_blank">nico@van-royen.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-family:arial,helvetica,sans-serif;font-size:12pt;color:#000000"><div>Hello,</div><div><br></div><div>On our dev environment we want to test GeoReplication with GlusterFS 4.1 and every attempt so far fails.</div><div>For now, we don't care (yet) about running it as a non-root user (not using the mountbroker etc).</div><div><br></div><div>Installed packages, both on master and slaves:</div><div><div>[root@clrv0000110367 geo-replication]# rpm -qa | grep gluster</div><div>glusterfs-client-xlators-4.1.<wbr>3-1.el7.x86_64</div><div>glusterfs-events-4.1.3-1.el7.<wbr>x86_64</div><div>glusterfs-geo-replication-4.1.<wbr>3-1.el7.x86_64</div><div>glusterfs-4.1.3-1.el7.x86_64</div><div>glusterfs-api-4.1.3-1.el7.x86_<wbr>64</div><div>glusterfs-fuse-4.1.3-1.el7.<wbr>x86_64</div><div>glusterfs-server-4.1.3-1.el7.<wbr>x86_64</div><div>glusterfs-rdma-4.1.3-1.el7.<wbr>x86_64</div><div>glusterfs-extra-xlators-4.1.3-<wbr>1.el7.x86_64</div><div>glusterfs-libs-4.1.3-1.el7.<wbr>x86_64</div><div>glusterfs-cli-4.1.3-1.el7.x86_<wbr>64</div><div>python2-gluster-4.1.3-1.el7.<wbr>x86_64</div><div>glusterfs-coreutils-0.2.0-1.<wbr>el7.x86_64</div><div><br></div><div>Master volume setup:</div><div># gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110367:/gluster/<wbr>VOLUME2/export clrv0000110371:/gluster/<wbr>VOLUME2/export clrv0000110389:/gluster/<wbr>VOLUME2/export</div><div># gluster v start VOLUME2</div><div># gluster volume set all cluster.enable-shared-storage enable</div><div></div><div><br></div><div>Slave volume setup</div><div># gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110605:/gluster/<wbr>VOLUME2/export clrv0000110608:/gluster/<wbr>VOLUME2/export clrv0000110606:/gluster/<wbr>VOLUME2/export<br></div><div># gluster v start VOLUME2</div><div># gluster volume set all cluster.enable-shared-storage enable</div><div><br></div><div>On master server:</div><div> <div style="color:#000000;font-family:arial,helvetica,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:#ffffff;text-decoration-style:initial;text-decoration-color:initial"># ssh-keygen (accepting all defaults)</div><div style="color:#000000;font-family:arial,helvetica,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:#ffffff;text-decoration-style:initial;text-decoration-color:initial"># ssh-copy-id <span> </span><span style="color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline!important">clrv0000110605 (one of the slave servers)</span></div><div style="color:#000000;font-family:arial,helvetica,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:#ffffff;text-decoration-style:initial;text-decoration-color:initial"><span style="color:rgb(0,0,0);font-family:arial,helvetica,sans-serif;font-size:16px;font-style:normal;font-variant-ligatures:normal;font-variant-caps:normal;font-weight:400;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline!important"># gluster-georep-sshkey generate</span></div><div style="clear:both"># gluster volume geo-replication VOLUME2 <a href="http://clrv0000110605.ic.ing.net">clrv0000110605.ic.ing.net</a>::<wbr>VOLUME2 create push-pem<br></div><div style="clear:both"><br></div><div style="clear:both">Several seconds later, all of the glusterd instances on the master side crash, with /var/log/glusterfs/glusterd.<wbr>log such as:</div><div style="clear:both"><br></div><div style="clear:both"><div style="clear:both">[2018-09-06 08:50:20.663584] W [MSGID: 106028] [glusterd-geo-rep.c:2568:<wbr>glusterd_get_statefile_name] 0-management: Config file (/var/lib/glusterd/geo-<wbr>replication/VOLUME2_<wbr>clrv0000110605_VOLUME2/gsyncd.<wbr>conf) missing. Looking for template config file (/var/lib/glusterd/geo-<wbr>replication/gsyncd_template.<wbr>conf) [No such file or directory]</div><div style="clear:both">[2018-09-06 08:50:20.663724] I [MSGID: 106294] [glusterd-geo-rep.c:2577:<wbr>glusterd_get_statefile_name] 0-management: Using default config template(/var/lib/glusterd/<wbr>geo-replication/gsyncd_<wbr>template.conf).</div><div style="clear:both">[2018-09-06 08:50:24.072321] I [MSGID: 106494] [glusterd-handler.c:3024:__<wbr>glusterd_handle_cli_profile_<wbr>volume] 0-management: Received volume profile req for volume VOLUME1</div><div style="clear:both">[2018-09-06 08:50:24.074876] I [MSGID: 106487] [glusterd-handler.c:1486:__<wbr>glusterd_handle_cli_list_<wbr>friends] 0-glusterd: Received cli list req</div><div style="clear:both">[2018-09-06 08:50:24.744276] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:<wbr>glusterd_proc_stop] 0-management: nfs already stopped</div><div style="clear:both">[2018-09-06 08:50:24.744443] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:<wbr>glusterd_svc_stop] 0-management: nfs service is stopped</div><div style="clear:both">[2018-09-06 08:50:24.744497] I [MSGID: 106599] [glusterd-nfs-svc.c:82:<wbr>glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed</div><div style="clear:both">[2018-09-06 08:50:24.749139] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:<wbr>glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 40886</div><div style="clear:both">[2018-09-06 08:50:25.749748] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:<wbr>glusterd_svc_stop] 0-management: glustershd service is stopped</div><div style="clear:both">[2018-09-06 08:50:25.750047] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:<wbr>glusterd_svc_start] 0-management: Starting glustershd service</div><div style="clear:both">[2018-09-06 08:50:25.757036] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:<wbr>glusterd_proc_stop] 0-management: bitd already stopped</div><div style="clear:both">[2018-09-06 08:50:25.757100] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:<wbr>glusterd_svc_stop] 0-management: bitd service is stopped</div><div style="clear:both">[2018-09-06 08:50:25.757288] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:<wbr>glusterd_proc_stop] 0-management: scrub already stopped</div><div style="clear:both">[2018-09-06 08:50:25.757330] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:<wbr>glusterd_svc_stop] 0-management: scrub service is stopped</div><div style="clear:both">[2018-09-06 08:50:28.391332] I [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.<wbr>3/xlator/mgmt/glusterd.so(+<wbr>0xe2b1a) [0x7fae33053b1a] -->/usr/lib64/glusterfs/4.1.3/<wbr>xlator/mgmt/glusterd.so(+<wbr>0xe25e5) [0x7fae330535e5] -->/lib64/libglusterfs.so.0(<wbr>runner_log+0x115) [0x7fae3e55f0c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/<wbr>gsync-create/post/S56glusterd-<wbr>geo-rep-create-post.sh --volname=VOLUME2 is_push_pem=1,pub_file=/var/<wbr>lib/glusterd/geo-replication/<wbr>common_secret.pem.pub,slave_<wbr>user=root,slave_ip=<wbr>clrv0000110605,slave_vol=<wbr>VOLUME2,ssh_port=22</div><div style="clear:both">[2018-09-06 08:50:31.901854] I [MSGID: 106327] [glusterd-geo-rep.c:2562:<wbr>glusterd_get_statefile_name] 0-management: Using passed config template(/var/lib/glusterd/<wbr>geo-replication/VOLUME2_<wbr>clrv0000110605_VOLUME2/gsyncd.<wbr>conf).</div><div style="clear:both">[2018-09-06 08:50:32.252426] E [MSGID: 106152] [glusterd-syncop.c:113:gd_<wbr>collate_errors] 0-glusterd: Staging failed on <a href="http://clrv0000110389.ic.ing.net" target="_blank">clrv0000110389.ic.ing.net</a>. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.</div><div style="clear:both">[2018-09-06 08:50:32.271707] E [MSGID: 106152] [glusterd-syncop.c:113:gd_<wbr>collate_errors] 0-glusterd: Staging failed on <a href="http://clrv0000110371.ic.ing.net" target="_blank">clrv0000110371.ic.ing.net</a>. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.</div><div style="clear:both">[2018-09-06 08:50:32.332909] E [MSGID: 106152] [glusterd-syncop.c:113:gd_<wbr>collate_errors] 0-glusterd: Staging failed on <a href="http://clrv0000110372.ic.ing.net" target="_blank">clrv0000110372.ic.ing.net</a>. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.</div><div style="clear:both">[2018-09-06 08:50:35.690440] E [mem-pool.c:335:__gf_free] (-->/usr/lib64/glusterfs/4.1.<wbr>3/xlator/mgmt/glusterd.so(+<wbr>0x2436e) [0x7fae32f9536e] -->/usr/lib64/glusterfs/4.1.3/<wbr>xlator/mgmt/glusterd.so(+<wbr>0x20e10) [0x7fae32f91e10] -->/lib64/libglusterfs.so.0(__<wbr>gf_free+0x104) [0x7fae3e5344f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size)</div><div style="clear:both">The message "I [MSGID: 106584] [glusterd-handler.c:5904:__<wbr>glusterd_handle_get_state] 0-management: Received request to get state for glusterd"</div><div style="clear:both">repeated 6 times between [2018-09-06 08:49:02.500390] and [2018-09-06 08:50:35.690241]</div><div style="clear:both">[2018-09-06 08:50:35.846286] I [MSGID: 106327] [glusterd-geo-rep.c:4482:<wbr>glusterd_read_status_file] 0-management: Using passed config template(/var/lib/glusterd/<wbr>geo-replication/VOLUME2_<wbr>clrv0000110605_VOLUME2/gsyncd.<wbr>conf).</div><div style="clear:both">[2018-09-06 08:50:36.144575] E [mem-pool.c:326:__gf_free] (-->/lib64/libglusterfs.so.0(+<wbr>0x1a2c0) [0x7fae3e4fe2c0] -->/lib64/libglusterfs.so.0(<wbr>data_destroy+0x5d) [0x7fae3e4fd92d] -->/lib64/libglusterfs.so.0(__<wbr>gf_free+0xa4) [0x7fae3e534494] ) 0-: Assertion failed: GF_MEM_HEADER_MAGIC ==</div><div style="clear:both">header->magic</div><div style="clear:both">[2018-09-06 08:50:11.549220] E [MSGID: 106332] [glusterd-utils.c:12886:<wbr>glusterd_get_value_for_vme_<wbr>entry] 0-management: Failed to get option for xattr-user-namespace-mode key</div><div style="clear:both">pending frames:</div><div style="clear:both">frame : type(0) op(0)</div><div style="clear:both">patchset: git://<a href="http://git.gluster.org/glusterfs.git" target="_blank">git.gluster.org/<wbr>glusterfs.git</a></div><div style="clear:both">signal received: 6</div><div style="clear:both">time of crash:</div><div style="clear:both">2018-09-06 08:50:36</div><div style="clear:both">configuration details:</div><div style="clear:both">argp 1</div><div style="clear:both">backtrace 1</div><div style="clear:both">dlfcn 1</div><div style="clear:both">libpthread 1</div><div style="clear:both">llistxattr 1</div><div style="clear:both">setfsid 1</div><div style="clear:both">spinlock 1</div><div style="clear:both">epoll.h 1</div><div style="clear:both">xattr.h 1</div><div style="clear:both">st_atim.tv_nsec 1</div><div style="clear:both">package-string: glusterfs 4.1.3</div><div style="clear:both">/lib64/libglusterfs.so.0(+<wbr>0x25920)[0x7fae3e509920]</div><div style="clear:both">/lib64/libglusterfs.so.0(gf_<wbr>print_trace+0x334)[<wbr>0x7fae3e513874]</div><div style="clear:both">/lib64/libc.so.6(+0x36280)[<wbr>0x7fae3cb6e280]</div><div style="clear:both">/lib64/libc.so.6(gsignal+0x37)<wbr>[0x7fae3cb6e207]</div><div style="clear:both">/lib64/libc.so.6(abort+0x148)[<wbr>0x7fae3cb6f8f8]</div><div style="clear:both">/lib64/libc.so.6(+0x78cc7)[<wbr>0x7fae3cbb0cc7]</div><div style="clear:both">/lib64/libc.so.6(+0x81429)[<wbr>0x7fae3cbb9429]</div><div style="clear:both">/lib64/libglusterfs.so.0(data_<wbr>destroy+0x5d)[0x7fae3e4fd92d]</div><div style="clear:both">/lib64/libglusterfs.so.0(+<wbr>0x1a2c0)[0x7fae3e4fe2c0]</div><div style="clear:both">/usr/lib64/glusterfs/4.1.3/<wbr>xlator/mgmt/glusterd.so(+<wbr>0x22254)[0x7fae32f93254]</div><div style="clear:both">/usr/lib64/glusterfs/4.1.3/<wbr>xlator/mgmt/glusterd.so(+<wbr>0x2436e)[0x7fae32f9536e]</div><div style="clear:both">/lib64/libglusterfs.so.0(+<wbr>0x622b0)[0x7fae3e5462b0]</div><div style="clear:both">/lib64/libc.so.6(+0x47fc0)[<wbr>0x7fae3cb7ffc0]</div><div style="clear:both">---------</div><div style="clear:both"><br></div><div style="clear:both">Note that the gsyncd_template.conf file (as found in /var/lib/glusterd/geo-<wbr>replication) is empty, and was (re)created when the gluster volume geo-replication command was issued. (is it supposed to be empty ?)</div><div style="clear:both"><br></div><div style="clear:both">Anyone any idea's ?</div><div style="clear:both"><br></div><div style="clear:both">Regards, Nico van Roijen</div></div><div style="clear:both"><br></div></div></div></div></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</div>