<div dir="ltr"><div><div>Hi Marcus,<br><br></div>I think the fix [1] is needed in 4.1<br></div>Could you please this out and let us know if that works for you?<br><div><br>[1] <a href="https://review.gluster.org/#/c/20207/">https://review.gluster.org/#/c/20207/</a></div><div><br></div><div>Thanks,<br></div><div>Kotresh HR<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 12, 2018 at 1:49 AM, Marcus Pedersén <span dir="ltr"><<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr" style="font-size:12pt;color:#000000;background-color:#ffffff;font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi all,</p>
<p>I have upgraded from 3.12.9 to 4.1.1 and been following upgrade instructions for offline upgrade.</p>
<p>I upgraded geo-replication side first 1 x (2+1) and the master side after that 2 x (2+1).</p>
<p>Both clusters works the way they should on their own.</p>
<p>After upgrade on master side status <wbr>for all geo-replication nodes <wbr>is Stopped.</p>
<p>I tried to start the geo-replication from master node and response back was started successfully.</p>
<p>Status again .... Stopped</p>
<p>Tried to start again and get response started successfully, after that all glusterd crashed on all master nodes.</p>
<p>After a restart of all glusterd the master cluster was up again.</p>
<p>Status for geo-replication is still Stopped and every try to start it after this gives the response successful but still status Stopped.</p>
<p><br>
</p>
<p>Please help me get the geo-replication up and running again.</p>
<p><br>
</p>
<p>Best regards</p>
<p>Marcus Pedersén<br>
</p>
<p><br>
</p>
<p>Part of geo-replication log from master node:</p>
<p>[2018-07-11 18:42:48.941760] I [changelogagent(/urd-gds/<wbr>gluster):73:__init__] ChangelogAgent: Agent listining...<br>
[2018-07-11 18:42:48.947567] I [resource(/urd-gds/gluster):<wbr>1780:connect_remote] SSH: Initializing SSH connection between master and slave...<br>
[2018-07-11 18:42:49.363514] E [syncdutils(/urd-gds/gluster):<wbr>304:log_raise_exception] <top>: connection to peer is broken<br>
[2018-07-11 18:42:49.364279] E [resource(/urd-gds/gluster):<wbr>210:errlog] Popen: command returned error cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-<wbr>replication/secret\<br>
.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-hjRhBo/<wbr>7e5534547f3675a710a10772231748<wbr>4f.sock geouser@urd-gds-geo-000 /nonexistent/gsyncd --session-owner 5e94eb7d-219f-4741-a179-<wbr>d4ae6b50c7ee --local-id .%\<br>
2Furd-gds%2Fgluster --local-node urd-gds-001 -N --listen --timeout 120 gluster://localhost:urd-gds-<wbr>volume error=2<br>
[2018-07-11 18:42:49.364586] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh> usage: gsyncd.py [-h]<br>
[2018-07-11 18:42:49.364799] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh><br>
[2018-07-11 18:42:49.364989] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh> {monitor-status,monitor,<wbr>worker,agent,slave,status,<wbr>config-check,config-get,<wbr>config-set,config-reset,<wbr>voluuidget,d\<br>
elete}<br>
[2018-07-11 18:42:49.365210] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh> ...<br>
[2018-07-11 18:42:49.365408] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh> gsyncd.py: error: argument subcmd: invalid choice: '5e94eb7d-219f-4741-a179-<wbr>d4ae6b50c7ee' (choose from 'monitor-status', 'monit\<br>
or', 'worker', 'agent', 'slave', 'status', 'config-check', 'config-get', 'config-set', 'config-reset', 'voluuidget', 'delete')<br>
[2018-07-11 18:42:49.365919] I [syncdutils(/urd-gds/gluster):<wbr>271:finalize] <top>: exiting.<br>
[2018-07-11 18:42:49.369316] I [repce(/urd-gds/gluster):92:<wbr>service_loop] RepceServer: terminating on reaching EOF.<br>
[2018-07-11 18:42:49.369921] I [syncdutils(/urd-gds/gluster):<wbr>271:finalize] <top>: exiting.<br>
[2018-07-11 18:42:49.369694] I [monitor(monitor):353:monitor] Monitor: worker died before establishing connection brick=/urd-gds/gluster<br>
[2018-07-11 18:42:59.492762] I [monitor(monitor):280:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=ssh://geouser@urd-<wbr>gds-geo-000:gluster://<wbr>localhost:urd-gds-volume<br>
[2018-07-11 18:42:59.558491] I [resource(/urd-gds/gluster):<wbr>1780:connect_remote] SSH: Initializing SSH connection between master and slave...<br>
[2018-07-11 18:42:59.559056] I [changelogagent(/urd-gds/<wbr>gluster):73:__init__] ChangelogAgent: Agent listining...<br>
[2018-07-11 18:42:59.945693] E [syncdutils(/urd-gds/gluster):<wbr>304:log_raise_exception] <top>: connection to peer is broken<br>
[2018-07-11 18:42:59.946439] E [resource(/urd-gds/gluster):<wbr>210:errlog] Popen: command returned error cmd=ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i /var/lib/glusterd/geo-<wbr>replication/secret\<br>
.pem -p 22 -oControlMaster=auto -S /tmp/gsyncd-aux-ssh-992bk7/<wbr>7e5534547f3675a710a10772231748<wbr>4f.sock geouser@urd-gds-geo-000 /nonexistent/gsyncd --session-owner 5e94eb7d-219f-4741-a179-<wbr>d4ae6b50c7ee --local-id .%\<br>
2Furd-gds%2Fgluster --local-node urd-gds-001 -N --listen --timeout 120 gluster://localhost:urd-gds-<wbr>volume error=2<br>
[2018-07-11 18:42:59.946748] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh> usage: gsyncd.py [-h]<br>
[2018-07-11 18:42:59.946962] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh><br>
[2018-07-11 18:42:59.947150] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh> {monitor-status,monitor,<wbr>worker,agent,slave,status,<wbr>config-check,config-get,<wbr>config-set,config-reset,<wbr>voluuidget,d\<br>
elete}<br>
[2018-07-11 18:42:59.947369] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh> ...<br>
[2018-07-11 18:42:59.947552] E [resource(/urd-gds/gluster):<wbr>214:logerr] Popen: ssh> gsyncd.py: error: argument subcmd: invalid choice: '5e94eb7d-219f-4741-a179-<wbr>d4ae6b50c7ee' (choose from 'monitor-status', 'monit\<br>
or', 'worker', 'agent', 'slave', 'status', 'config-check', 'config-get', 'config-set', 'config-reset', 'voluuidget', 'delete')<br>
[2018-07-11 18:42:59.948046] I [syncdutils(/urd-gds/gluster):<wbr>271:finalize] <top>: exiting.<br>
[2018-07-11 18:42:59.951392] I [repce(/urd-gds/gluster):92:<wbr>service_loop] RepceServer: terminating on reaching EOF.<br>
[2018-07-11 18:42:59.951760] I [syncdutils(/urd-gds/gluster):<wbr>271:finalize] <top>: exiting.<br>
[2018-07-11 18:42:59.951817] I [monitor(monitor):353:monitor] Monitor: worker died before establishing connection brick=/urd-gds/gluster<br>
[2018-07-11 18:43:10.54580] I [monitor(monitor):280:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=ssh://geouser@urd-<wbr>gds-geo-000:gluster://<wbr>localhost:urd-gds-volume<br>
[2018-07-11 18:43:10.88356] I [monitor(monitor):345:monitor] Monitor: Changelog Agent died, Aborting Worker brick=/urd-gds/gluster<br>
[2018-07-11 18:43:10.88613] I [monitor(monitor):353:monitor] Monitor: worker died before establishing connection brick=/urd-gds/gluster<br>
[2018-07-11 18:43:20.112435] I [gsyncdstatus(monitor):242:<wbr>set_worker_status] GeorepStatus: Worker Status Change status=inconsistent<br>
[2018-07-11 18:43:20.112885] E [syncdutils(monitor):331:log_<wbr>raise_exception] <top>: FAIL:<br>
Traceback (most recent call last):<br>
File "/usr/libexec/glusterfs/<wbr>python/syncdaemon/syncdutils.<wbr>py", line 361, in twrap<br>
except:<br>
File "/usr/libexec/glusterfs/<wbr>python/syncdaemon/monitor.py", line 428, in wmon<br>
sys.exit()<br>
TypeError: 'int' object is not iterable<br>
[2018-07-11 18:43:20.114610] I [syncdutils(monitor):271:<wbr>finalize] <top>: exiting.<br>
</p>
<p>---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">här </a><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">here </a></p>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</div>