<div dir="ltr"><div><div><div><div><div>Hi,<br><br></div>It is failing to get the virtual xattr value of "trusted.glusterfs.volume-mark" at master volume root. <br></div>Could you share the geo-replication logs under /var/log/glusterfs/geo-replication/*.gluster.log ?<br></div>I think if there are any transient errors, stopping geo-rep and restarting master volume should fix it.<br><br></div>Thanks,<br></div>Kotresh HR<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 6, 2018 at 2:28 PM, Curt Lestrup <span dir="ltr"><<a href="mailto:curt@lestrup.se" target="_blank">curt@lestrup.se</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="white" link="#0563C1" vlink="#954F72" lang="EN-GB">
<div class="m_-8783196927232487504WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt" lang="SV">Hi,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt" lang="SV"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Have problems with geo replication on glusterfs 3.12.6 / Ubuntu 16.04.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">I can see a “master volinfo unavailable” in master logfile.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Any ideas?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Master:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Status of volume: testtomcat<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Gluster process TCP Port RDMA Port Online Pid<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">------------------------------<wbr>------------------------------<wbr>------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Brick gfstest07:/gfs/testtomcat/<wbr>mount 49153 0 Y 326<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Brick gfstest05:/gfs/testtomcat/<wbr>mount 49153 0 Y 326<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Brick gfstest01:/gfs/testtomcat/<wbr>mount 49153 0 Y 335<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Self-heal Daemon on localhost N/A N/A Y 1134<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Self-heal Daemon on gfstest07 N/A N/A Y 564<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Self-heal Daemon on gfstest05 N/A N/A Y 1038<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Slave:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Status of volume: testtomcat<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Gluster process <wbr> TCP Port RDMA Port Online Pid<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">------------------------------<wbr>------------------------------<wbr>------------------<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Courier",serif">Brick stogfstest11:/gfs/testtomcat/<wbr>mount 49152 0 Y 294<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Created & started the session with:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">gluster volume geo-replication testtomcat stogfstest11::testtomcat create no-verify<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">gluster volume geo-replication testtomcat stogfstest11::testtomcat start<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">getting the following logs:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">master:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:46.767544] I [gsyncdstatus(monitor):242:<wbr>set_worker_status] GeorepStatus: Worker Status Change status=Initializing...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:46.872857] I [monitor(monitor):280:monitor] Monitor: starting gsyncd worker brick=/gfs/testtomcat/mount <wbr> slave_node=ssh://root@<wbr>stogfstest11:gluster://<wbr>localhost:testtomcat<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:46.961122] I [changelogagent(/gfs/<wbr>testtomcat/mount):73:__init__] ChangelogAgent: Agent listining...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:46.962470] I [resource(/gfs/testtomcat/<wbr>mount):1771:connect_remote] SSH: Initializing SSH connection between master and slave...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:48.515974] I [resource(/gfs/testtomcat/<wbr>mount):1778:connect_remote] SSH: SSH connection between master and slave established. duration=1.5530<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:48.516247] I [resource(/gfs/testtomcat/<wbr>mount):1493:connect] GLUSTER: Mounting gluster volume locally...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:49.739631] I [resource(/gfs/testtomcat/<wbr>mount):1506:connect] GLUSTER: Mounted gluster volume duration=1.2232<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:49.739870] I [gsyncd(/gfs/testtomcat/mount)<wbr>:799:main_i] <top>: Closing feedback fd, waking up the monitor<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:51.872872] I [master(/gfs/testtomcat/mount)<wbr>:1518:register] _GMaster: Working dir path=/var/lib/misc/glusterfsd/<wbr>testtomcat/ssh%3A%2F%2Froot%<wbr>40172.16.81.101%3Agluster%3A%<wbr>2F%2F127.0.0.1%3Atesttomcat/<wbr>b6a7905143e15d9b079b804c0a8ebf<wbr>42<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:51.873176] I [resource(/gfs/testtomcat/<wbr>mount):1653:service_loop] GLUSTER: Register time time=1520325171<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:51.926801] E [syncdutils(/gfs/testtomcat/<wbr>mount):299:log_raise_<wbr>exception] <top>: master volinfo unavailable<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:51.936203] I [syncdutils(/gfs/testtomcat/<wbr>mount):271:finalize] <top>: exiting.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:51.938469] I [repce(/gfs/testtomcat/mount):<wbr>92:service_loop] RepceServer: terminating on reaching EOF.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:51.938776] I [syncdutils(/gfs/testtomcat/<wbr>mount):271:finalize] <top>: exiting.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:52.743696] I [monitor(monitor):363:monitor] Monitor: worker died in startup phase brick=/gfs/testtomcat/mount<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:52.763276] I [gsyncdstatus(monitor):242:<wbr>set_worker_status] GeorepStatus: Worker Status Change status=Faulty<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">slave:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:47.434591] I [resource(slave):1502:connect] GLUSTER: Mounting gluster volume locally...<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:48.490775] I [resource(slave):1515:connect] GLUSTER: Mounted gluster volume duration=1.0557<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:48.493134] I [resource(slave):1012:service_<wbr>loop] GLUSTER: slave listening<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:51.942531] I [repce(slave):92:service_loop] RepceServer: terminating on reaching EOF.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">[2018-03-06 08:32:51.955379] I [syncdutils(slave):271:<wbr>finalize] <top>: exiting.<span class="HOEnZb"><font color="#888888"><u></u><u></u></font></span></span></p><span class="HOEnZb"><font color="#888888">
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">/Curt<u></u><u></u></span></p>
</font></span></div>
</div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>
</div>