<div dir="ltr">This issue has been recently fixed with the following patch and should be available in latest gluster-6.x<div><br></div><div><a href="https://review.gluster.org/#/c/glusterfs/+/23570/">https://review.gluster.org/#/c/glusterfs/+/23570/</a><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 19, 2019 at 10:26 AM deepu srinivasan &lt;<a href="mailto:sdeepugd@gmail.com">sdeepugd@gmail.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><br><span style="font-family:sans-serif;font-size:12.8px">Hi Aravinda</span><br style="font-family:sans-serif;font-size:12.8px"><b style="font-family:sans-serif;font-size:12.8px">The below logs are from master end:</b><br style="font-family:sans-serif;font-size:12.8px"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;font-family:sans-serif;font-size:12.8px">[2019-11-16 17:29:43.536881] I [gsyncdstatus(worker /home/sas/gluster/data/code-misc6):281:set_active] GeorepStatus: Worker Status Change       status=Active<br>[2019-11-16 17:29:43.629620] I [gsyncdstatus(worker /home/sas/gluster/data/code-misc6):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change   status=History Crawl<br>[2019-11-16 17:29:43.630328] I [master(worker /home/sas/gluster/data/code-misc6):1517:crawl] _GMaster: starting history crawl   turns=1 stime=(1573924576, 0)   entry_stime=(1573924576, 0)     etime=1573925383<br>[2019-11-16 17:29:44.636725] I [master(worker /home/sas/gluster/data/code-misc6):1546:crawl] _GMaster: slave&#39;s time     stime=(1573924576, 0)<br>[2019-11-16 17:29:44.778966] I [master(worker /home/sas/gluster/data/code-misc6):898:fix_possible_entry_failures] _GMaster: Fixing ENOENT error in slave. Parent does not exist on master. Safe to ignore, take out entry       retry_count=1   entry=({&#39;uid&#39;: 0, &#39;gfid&#39;: &#39;c02519e0-0ead-4fe8-902b-dcae72ef83a3&#39;, &#39;gid&#39;: 0, &#39;mode&#39;: 33188, &#39;entry&#39;: &#39;.gfid/d60aa0d5-4fdf-4721-97dc-9e3e50995dab/368307802&#39;, &#39;op&#39;: &#39;CREATE&#39;}, 2, {&#39;slave_isdir&#39;: False, &#39;gfid_mismatch&#39;: False, &#39;slave_name&#39;: None, &#39;slave_gfid&#39;: None, &#39;name_mismatch&#39;: False, &#39;dst&#39;: False})<br>[2019-11-16 17:29:44.779306] I [master(worker /home/sas/gluster/data/code-misc6):942:handle_entry_failures] _GMaster: Sucessfully fixed entry ops with gfid mismatch    retry_count=1<br>[2019-11-16 17:29:44.779516] I [master(worker /home/sas/gluster/data/code-misc6):1194:process_change] _GMaster: Retry original entries. count = 1<br>[2019-11-16 17:29:44.879321] E [repce(worker /home/sas/gluster/data/code-misc6):214:__call__] RepceClient: call failed  call=151945:140353273153344:1573925384.78       method=entry_ops        error=OSError<br>[2019-11-16 17:29:44.879750] E [syncdutils(worker /home/sas/gluster/data/code-misc6):338:log_raise_exception] &lt;top&gt;: FAIL:<br>Traceback (most recent call last):<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py&quot;, line 322, in main<br>    func(args)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/subcmds.py&quot;, line 82, in subcmd_worker<br>    local.service_loop(remote)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/resource.py&quot;, line 1277, in service_loop<br>    g3.crawlwrap(oneshot=True)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 599, in crawlwrap<br>    self.crawl()<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 1555, in crawl<br>    self.changelogs_batch_process(changes)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 1455, in changelogs_batch_process<br>    self.process(batch)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 1290, in process<br>    self.process_change(change, done, retry)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 1195, in process_change<br>    failures = self.slave.server.entry_ops(entries)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/repce.py&quot;, line 233, in __call__<br>    return self.ins(self.meth, *a)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/repce.py&quot;, line 215, in __call__<br>    raise res<br>OSError: [Errno 13] Permission denied: &#39;/home/sas/gluster/data/code-misc6/.glusterfs/6a/90/6a9008b1-a4aa-4c30-9ae7-92a33e05d0bb&#39;<br>[2019-11-16 17:29:44.911767] I [repce(agent /home/sas/gluster/data/code-misc6):97:service_loop] RepceServer: terminating on reaching EOF.<br>[2019-11-16 17:29:45.509344] I [monitor(monitor):278:monitor] Monitor: worker died in startup phase     brick=/home/sas/gluster/data/code-misc6<br>[2019-11-16 17:29:45.511806] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty<br></blockquote><br style="font-family:sans-serif;font-size:12.8px"><br style="font-family:sans-serif;font-size:12.8px"><b style="font-family:sans-serif;font-size:12.8px">The below logs are from the slave end.</b><br style="font-family:sans-serif;font-size:12.8px"><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex;font-family:sans-serif;font-size:12.8px">[2019-11-16 17:24:42.281599] I [resource(slave <a href="http://192.168.185.106/home/sas/gluster/data/code-misc6%29:580:entry_ops" style="text-decoration-line:none;color:rgb(66,133,244)" target="_blank">192.168.185.106/home/sas/gluster/data/code-misc6):580:entry_ops</a>] &lt;top&gt;: Special case: rename on mkdir    gfid=6a9008b1-a4aa-4c30-9ae7-92a33e05d0bb       entry=&#39;.gfid/a8921d78-a078-46d3-aca5-8b078eb62cac/8878061b-d5b3-47a6-b01c-8310fee39b20&#39;<br>[2019-11-16 17:24:42.370582] E [repce(slave <a href="http://192.168.185.106/home/sas/gluster/data/code-misc6%29:122:worker" style="text-decoration-line:none;color:rgb(66,133,244)" target="_blank">192.168.185.106/home/sas/gluster/data/code-misc6):122:worker</a>] &lt;top&gt;: call failed:<br>Traceback (most recent call last):<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/repce.py&quot;, line 118, in worker<br>    res = getattr(self.obj, rmeth)(*in_data[2:])<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/resource.py&quot;, line 581, in entry_ops<br>    src_entry = get_slv_dir_path(slv_host, slv_volume, gfid)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py&quot;, line 690, in get_slv_dir_path<br>    [ENOENT], [ESTALE])<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py&quot;, line 546, in errno_wrap<br>    return call(*arg)<br>OSError: [Errno 13] Permission denied: &#39;/home/sas/gluster/data/code-misc6/.glusterfs/6a/90/6a9008b1-a4aa-4c30-9ae7-92a33e05d0bb&#39;<br>[2019-11-16 17:24:42.400402] I [repce(slave <a href="http://192.168.185.106/home/sas/gluster/data/code-misc6%29:97:service_loop" style="text-decoration-line:none;color:rgb(66,133,244)" target="_blank">192.168.185.106/home/sas/gluster/data/code-misc6):97:service_loop</a>] RepceServer: terminating on reaching EOF.<br>[2019-11-16 17:24:53.403165] W [gsyncd(slave <a href="http://192.168.185.106/home/sas/gluster/data/code-misc6%29:304:main" style="text-decoration-line:none;color:rgb(66,133,244)" target="_blank">192.168.185.106/home/sas/gluster/data/code-misc6):304:main</a>] &lt;top&gt;: Session config file not exists, using the default config        path=/var/lib/glusterd/geo-replication/code-misc_192.168.185.107_code-misc/gsyncd.con</blockquote></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Nov 16, 2019, 9:26 PM Aravinda Vishwanathapura Krishna Murthy &lt;<a href="mailto:avishwan@redhat.com" target="_blank">avishwan@redhat.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Deepu,</div><div><br></div><div>Please share the reason for Faulty from Geo-rep logs of respective <br>master node.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Nov 16, 2019 at 1:01 AM deepu srinivasan &lt;<a href="mailto:sdeepugd@gmail.com" rel="noreferrer" target="_blank">sdeepugd@gmail.com</a>&gt; wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Users/Development Team<div>We have set up a Geo-replication session with non-root in slave setup in our DC.</div><div>It was working well with Active Status and Changelogcrawl.</div><div><br></div><div>We were mounting the master node and the file is being written in it.</div><div>We were running some process as the root user so the process wrote some file and folder with root permission.<br>After stopping the geo-replication and starting the process the session went to the faulty state.</div><div>How to recover?</div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>regards<br></div>Aravinda VK<br></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>