<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hi <!--StartFragment--><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Kotresh,</span></div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Same result using the georepsetup tool.</span></div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">We removed the volume (on the master side) and recreated it. (its empty anyway).</span></div><div><span style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></div><div><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><div># gluster-georep-setup VOLUME2 clrv0000110605.ic.ing.net::VOLUME2</div><div>Geo-replication session will be established between VOLUME2 and clrv0000110605::VOLUME2</div><div>Root password of clrv0000110605 is required to complete the setup. NOTE: Password will not be stored.</div><div><br></div><div>root@clrv0000110605.ic.ing.net's password: </div><div>[ OK] clrv0000110605.ic.ing.net is Reachable(Port 22)</div><div>[ OK] SSH Connection established root@clrv0000110605.ic.ing.net</div><div>[ OK] Master Volume and Slave Volume are compatible (Version: 4.1.3)</div><div>[ OK] Common secret pub file present at /var/lib/glusterd/geo-replication/common_secret.pem.pub</div><div>[ OK] common_secret.pem.pub file copied to clrv0000110605</div><div>[ OK] Master SSH Keys copied to all Up Slave nodes</div><div>[ OK] Updated Master SSH Keys to all Up Slave nodes authorized_keys file</div><div>[NOT OK] Failed to Establish Geo-replication Session</div><div>Connection failed. Please check if gluster daemon is operational.</div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br></div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Gluster deamon on all master servers crashed/stopped with similair entries in the logfile:</span></div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></span></div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">[2018-09-06 11:33:09.552529] I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state] 0-management: Received request to get state for glusterd</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">[2018-09-06 11:33:09.552667] E [mem-pool.c:335:__gf_free] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e) [0x7fb27220636e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10) [0x7fb272202e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104) [0x7fb27d7a54f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size)</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">[2018-09-06 11:33:09.583766] I [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a) [0x7fb2722c4b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5) [0x7fb2722c45e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fb27d7d00c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2 is_push_pem=0,pub_file=/var/lib/glusterd/geo-replication/common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol=VOLUME2,ssh_port=22</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">[2018-09-06 11:33:09.725093] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">[2018-09-06 11:33:10.020642] E [mem-pool.c:326:__gf_free] (-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fb27d76f2c0] -->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fb27d76e92d] -->/lib64/libglusterfs.so.0(__gf_free+0xa4) [0x7fb27d7a5494] ) 0-: Assertion failed: GF_MEM_HEADER_MAGIC ==</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">header->magic</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">pending frames:</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">frame : type(0) op(0)</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">frame : type(0) op(0)</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">frame : type(0) op(0)</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">frame : type(0) op(0)</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">frame : type(0) op(0)</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">patchset: git://git.gluster.org/glusterfs.git</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">signal received: 6</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">time of crash:</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">2018-09-06 11:33:10</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">configuration details:</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">argp 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">backtrace 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">dlfcn 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">libpthread 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">llistxattr 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">setfsid 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">spinlock 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">epoll.h 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">xattr.h 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">st_atim.tv_nsec 1</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">package-string: glusterfs 4.1.3</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libglusterfs.so.0(+0x25920)[0x7fb27d77a920]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fb27d784874]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libc.so.6(+0x36280)[0x7fb27bddf280]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libc.so.6(gsignal+0x37)[0x7fb27bddf207]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libc.so.6(abort+0x148)[0x7fb27bde08f8]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libc.so.6(+0x78cc7)[0x7fb27be21cc7]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libc.so.6(+0x81429)[0x7fb27be2a429]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fb27d76e92d]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libglusterfs.so.0(+0x1a2c0)[0x7fb27d76f2c0]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x22254)[0x7fb272204254]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e)[0x7fb27220636e]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libglusterfs.so.0(+0x622b0)[0x7fb27d7b72b0]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">/lib64/libc.so.6(+0x47fc0)[0x7fb27bdf0fc0]</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;">---------</div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></span></span></span></div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Will try to see if we can enable coredumps (none have been created so far).</span></span></span></span></span></span></span></span></div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br></div><div data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;">Thanks already, Nico van Roijen</span></span></span></span></div></span></span></span></div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></span></div><div style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><span style="orphans: 2; text-align: start; text-indent: 0px; widows: 2; background-color: rgb(255, 255, 255); text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline !important;" data-mce-style="color: #000000; font-family: arial, helvetica, sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; display: inline !important; float: none;"><br data-mce-bogus="1"></span></span></span></div></span></div><div><br></div><hr id="zwchr" data-marker="__DIVIDER__"><div data-marker="__HEADERS__"><b>Van: </b>"Kotresh Hiremath Ravishankar" <khiremat@redhat.com><br><b>Aan: </b>"Nico van Royen" <nico@van-royen.nl><br><b>Cc: </b>"gluster-users" <gluster-users@gluster.org><br><b>Verzonden: </b>Donderdag 6 september 2018 12:58:37<br><b>Onderwerp: </b>Re: [Gluster-users] GlusterFS 4.1.3, Geo replication unable to setup<br></div><br><div data-marker="__QUOTED_TEXT__"><div dir="ltr"><div><div><div>Hi Nico,<br><br></div>The glusterd has crashed on this node. Please raise a bug with core file?<br><br></div>Please use the following tool [1] to setup geo-rep by bringing back the glusterd</div><div>if you are finding it difficult with geo-rep setup steps and let us know if if it still crashes?</div><br><div>[1] <a href="http://aravindavk.in/blog/introducing-georepsetup/" rel="bookmark" title="Permalink to Introducing georepsetup - Gluster Geo-replication Setup Tool" target="_blank">http://aravindavk.in/blog/introducing-georepsetup/</a><br></div><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Sep 6, 2018 at 2:54 PM, Nico van Royen <span dir="ltr"><<a href="mailto:nico@van-royen.nl" target="_blank">nico@van-royen.nl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;" data-mce-style="margin: 0 0 0 .8ex; border-left: 1px #ccc solid; padding-left: 1ex;"><div><div style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000;" data-mce-style="font-family: arial,helvetica,sans-serif; font-size: 12pt; color: #000000;"><div>Hello,</div><br><div>On our dev environment we want to test GeoReplication with GlusterFS 4.1 and every attempt so far fails.</div><div>For now, we don't care (yet) about running it as a non-root user (not using the mountbroker etc).</div><br><div>Installed packages, both on master and slaves:</div><div><div>[root@clrv0000110367 geo-replication]# rpm -qa | grep gluster</div><div>glusterfs-client-xlators-4.1.3-1.el7.x86_64</div><div>glusterfs-events-4.1.3-1.el7.x86_64</div><div>glusterfs-geo-replication-4.1.3-1.el7.x86_64</div><div>glusterfs-4.1.3-1.el7.x86_64</div><div>glusterfs-api-4.1.3-1.el7.x86_64</div><div>glusterfs-fuse-4.1.3-1.el7.x86_64</div><div>glusterfs-server-4.1.3-1.el7.x86_64</div><div>glusterfs-rdma-4.1.3-1.el7.x86_64</div><div>glusterfs-extra-xlators-4.1.3-1.el7.x86_64</div><div>glusterfs-libs-4.1.3-1.el7.x86_64</div><div>glusterfs-cli-4.1.3-1.el7.x86_64</div><div>python2-gluster-4.1.3-1.el7.x86_64</div><div>glusterfs-coreutils-0.2.0-1.el7.x86_64</div><br><div>Master volume setup:</div><div># gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110367:/gluster/VOLUME2/export clrv0000110371:/gluster/VOLUME2/export clrv0000110389:/gluster/VOLUME2/export</div><div># gluster v start VOLUME2</div><div># gluster volume set all cluster.enable-shared-storage enable</div><br><div>Slave volume setup</div><div># gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110605:/gluster/VOLUME2/export clrv0000110608:/gluster/VOLUME2/export clrv0000110606:/gluster/VOLUME2/export<br></div><div># gluster v start VOLUME2</div><div># gluster volume set all cluster.enable-shared-storage enable</div><br><div>On master server:</div><div> <div style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial;"># ssh-keygen (accepting all defaults)</div><div style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial;"># ssh-copy-id <span> </span><span style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline!important;" data-mce-style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline!important;">clrv0000110605 (one of the slave servers)</span></div><div style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial;" data-mce-style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial;"><span style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline!important;" data-mce-style="color: #000000; font-family: arial,helvetica,sans-serif; font-size: 16px; font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; background-color: #ffffff; text-decoration-style: initial; text-decoration-color: initial; float: none; display: inline!important;"># gluster-georep-sshkey generate</span></div><div style="clear: both;" data-mce-style="clear: both;"># gluster volume geo-replication VOLUME2 <a href="http://clrv0000110605.ic.ing.net" target="_blank">clrv0000110605.ic.ing.net</a>::VOLUME2 create push-pem<br></div><div style="clear: both;" data-mce-style="clear: both;"><br></div><div style="clear: both;" data-mce-style="clear: both;">Several seconds later, all of the glusterd instances on the master side crash, with /var/log/glusterfs/glusterd.log such as:</div><div style="clear: both;" data-mce-style="clear: both;"><br></div><div style="clear: both;" data-mce-style="clear: both;"><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:20.663584] W [MSGID: 106028] [glusterd-geo-rep.c:2568:glusterd_get_statefile_name] 0-management: Config file (/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf) missing. Looking for template config file (/var/lib/glusterd/geo-replication/gsyncd_template.conf) [No such file or directory]</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:20.663724] I [MSGID: 106294] [glusterd-geo-rep.c:2577:glusterd_get_statefile_name] 0-management: Using default config template(/var/lib/glusterd/geo-replication/gsyncd_template.conf).</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:24.072321] I [MSGID: 106494] [glusterd-handler.c:3024:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume VOLUME1</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:24.074876] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:24.744276] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:24.744443] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is stopped</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:24.744497] I [MSGID: 106599] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:24.749139] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 40886</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:25.749748] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd service is stopped</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:25.750047] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting glustershd service</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:25.757036] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:25.757100] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: bitd service is stopped</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:25.757288] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:25.757330] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: scrub service is stopped</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:28.391332] I [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a) [0x7fae33053b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5) [0x7fae330535e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fae3e55f0c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2 is_push_pem=1,pub_file=/var/lib/glusterd/geo-replication/common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol=VOLUME2,ssh_port=22</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:31.901854] I [MSGID: 106327] [glusterd-geo-rep.c:2562:glusterd_get_statefile_name] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:32.252426] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on <a href="http://clrv0000110389.ic.ing.net" target="_blank">clrv0000110389.ic.ing.net</a>. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:32.271707] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on <a href="http://clrv0000110371.ic.ing.net" target="_blank">clrv0000110371.ic.ing.net</a>. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:32.332909] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on <a href="http://clrv0000110372.ic.ing.net" target="_blank">clrv0000110372.ic.ing.net</a>. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:35.690440] E [mem-pool.c:335:__gf_free] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e) [0x7fae32f9536e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10) [0x7fae32f91e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104) [0x7fae3e5344f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size)</div><div style="clear: both;" data-mce-style="clear: both;">The message "I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state] 0-management: Received request to get state for glusterd"</div><div style="clear: both;" data-mce-style="clear: both;">repeated 6 times between [2018-09-06 08:49:02.500390] and [2018-09-06 08:50:35.690241]</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:35.846286] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:36.144575] E [mem-pool.c:326:__gf_free] (-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fae3e4fe2c0] -->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fae3e4fd92d] -->/lib64/libglusterfs.so.0(__gf_free+0xa4) [0x7fae3e534494] ) 0-: Assertion failed: GF_MEM_HEADER_MAGIC ==</div><div style="clear: both;" data-mce-style="clear: both;">header->magic</div><div style="clear: both;" data-mce-style="clear: both;">[2018-09-06 08:50:11.549220] E [MSGID: 106332] [glusterd-utils.c:12886:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key</div><div style="clear: both;" data-mce-style="clear: both;">pending frames:</div><div style="clear: both;" data-mce-style="clear: both;">frame : type(0) op(0)</div><div style="clear: both;" data-mce-style="clear: both;">patchset: git://<a href="http://git.gluster.org/glusterfs.git" target="_blank">git.gluster.org/glusterfs.git</a></div><div style="clear: both;" data-mce-style="clear: both;">signal received: 6</div><div style="clear: both;" data-mce-style="clear: both;">time of crash:</div><div style="clear: both;" data-mce-style="clear: both;">2018-09-06 08:50:36</div><div style="clear: both;" data-mce-style="clear: both;">configuration details:</div><div style="clear: both;" data-mce-style="clear: both;">argp 1</div><div style="clear: both;" data-mce-style="clear: both;">backtrace 1</div><div style="clear: both;" data-mce-style="clear: both;">dlfcn 1</div><div style="clear: both;" data-mce-style="clear: both;">libpthread 1</div><div style="clear: both;" data-mce-style="clear: both;">llistxattr 1</div><div style="clear: both;" data-mce-style="clear: both;">setfsid 1</div><div style="clear: both;" data-mce-style="clear: both;">spinlock 1</div><div style="clear: both;" data-mce-style="clear: both;">epoll.h 1</div><div style="clear: both;" data-mce-style="clear: both;">xattr.h 1</div><div style="clear: both;" data-mce-style="clear: both;">st_atim.tv_nsec 1</div><div style="clear: both;" data-mce-style="clear: both;">package-string: glusterfs 4.1.3</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libglusterfs.so.0(+0x25920)[0x7fae3e509920]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fae3e513874]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libc.so.6(+0x36280)[0x7fae3cb6e280]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libc.so.6(gsignal+0x37)[0x7fae3cb6e207]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libc.so.6(abort+0x148)[0x7fae3cb6f8f8]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libc.so.6(+0x78cc7)[0x7fae3cbb0cc7]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libc.so.6(+0x81429)[0x7fae3cbb9429]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fae3e4fd92d]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libglusterfs.so.0(+0x1a2c0)[0x7fae3e4fe2c0]</div><div style="clear: both;" data-mce-style="clear: both;">/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x22254)[0x7fae32f93254]</div><div style="clear: both;" data-mce-style="clear: both;">/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e)[0x7fae32f9536e]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libglusterfs.so.0(+0x622b0)[0x7fae3e5462b0]</div><div style="clear: both;" data-mce-style="clear: both;">/lib64/libc.so.6(+0x47fc0)[0x7fae3cb7ffc0]</div><div style="clear: both;" data-mce-style="clear: both;">---------</div><div style="clear: both;" data-mce-style="clear: both;"><br></div><div style="clear: both;" data-mce-style="clear: both;">Note that the gsyncd_template.conf file (as found in /var/lib/glusterd/geo-replication) is empty, and was (re)created when the gluster volume geo-replication command was issued. (is it supposed to be empty ?)</div><div style="clear: both;" data-mce-style="clear: both;"><br></div><div style="clear: both;" data-mce-style="clear: both;">Anyone any idea's ?</div><div style="clear: both;" data-mce-style="clear: both;"><br></div><div style="clear: both;" data-mce-style="clear: both;">Regards, Nico van Roijen</div></div><div style="clear: both;" data-mce-style="clear: both;"><br></div></div></div></div></div><br>_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</div><br></div></div></body></html>