<div dir="ltr"><div class="gmail_quote"><div dir="ltr"><div>Dear Developers/Users,</div><div><br></div><div>A geo-rep session of a sub-volume is in &#39;faulty&#39; status. Please find the setup and log details below.</div><div><br></div><div>Setup Details:</div><blockquote style="margin:0 0 0 40px;border:none;padding:0px">Gluster version - 7.0<br>Volume configuration - 2x3 (DxR)<br>gysncd permission(master)  - root<br>gysncd permission(slave)  - sas (non-root)<br>glusterd, glusterfsd permissions(master) - root<br>glusterd, glusterfsd permissions(slave) - root<br></blockquote><div><br></div>Log details:<br>In the <u>master gyncd</u> log, this traceback is printed <u>repeatedly</u>.<div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[2020-05-22 12:09:43.838727] I [master(worker /home/sas/gluster/data/code-ide):1991:syncjob] Syncer: Sync Time Taken        duration=0.4240        num_files=1        job=1        return_code=0<br>[2020-05-22 12:09:43.944392] E [repce(worker /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed        call=261471:140535761106752:1590149383.8        method=entry_ops        error=OSError<br>[2020-05-22 12:09:43.944746] E [syncdutils(worker /home/sas/gluster/data/code-ide):338:log_raise_exception] &lt;top&gt;: FAIL: <br>Traceback (most recent call last):<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py&quot;, line 325, in main<br>    func(args)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/subcmds.py&quot;, line 86, in subcmd_worker<br>    local.service_loop(remote)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/resource.py&quot;, line 1305, in service_loop<br>    g3.crawlwrap(oneshot=True)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 602, in crawlwrap<br>    self.crawl()<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 1592, in crawl<br>    self.changelogs_batch_process(changes)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 1492, in changelogs_batch_process<br>    self.process(batch)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 1327, in process<br>    self.process_change(change, done, retry)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/master.py&quot;, line 1221, in process_change<br>    failures = self.slave.server.entry_ops(entries)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/repce.py&quot;, line 233, in __call__<br>    return self.ins(self.meth, *a)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/repce.py&quot;, line 215, in __call__<br>    raise res<br>OSError: [Errno 13] Permission denied: &#39;/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e&#39;<br>[2020-05-22 12:09:43.968710] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.<br>[2020-05-22 12:09:44.912470] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase        brick=/home/sas/gluster/data/code-ide<br>[2020-05-22 12:09:44.913692] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change        status=Faulty<br></blockquote><div><br></div><div>In salve end, these are printed <u>repeatedly</u>. <br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>[2020-05-22 11:23:26.65115] W [gsyncd(slave <a href="http://10.47.8.153/home/sas/gluster/data/code-ide):307:main" target="_blank">10.47.8.153/home/sas/gluster/data/code-ide):307:main</a>] &lt;top&gt;: Session config file not exists, using the default config        path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf<br>[2020-05-22 11:23:26.77414] I [resource(slave <a href="http://10.47.8.153/home/sas/gluster/data/code-ide):1105:connect" target="_blank">10.47.8.153/home/sas/gluster/data/code-ide):1105:connect</a>] GLUSTER: Mounting gluster volume locally...<br>[2020-05-22 11:23:27.297466] I [resource(slave <a href="http://10.47.8.153/home/sas/gluster/data/code-ide):1128:connect" target="_blank">10.47.8.153/home/sas/gluster/data/code-ide):1128:connect</a>] GLUSTER: Mounted gluster volume        duration=1.2199<br>[2020-05-22 11:23:27.298125] I [resource(slave <a href="http://10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop" target="_blank">10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop</a>] GLUSTER: slave listening<br>[2020-05-22 11:23:32.654939] E [repce(slave <a href="http://10.47.8.153/home/sas/gluster/data/code-ide):122:worker" target="_blank">10.47.8.153/home/sas/gluster/data/code-ide):122:worker</a>] &lt;top&gt;: call failed: <br>Traceback (most recent call last):<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/repce.py&quot;, line 118, in worker<br>    res = getattr(self.obj, rmeth)(*in_data[2:])<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/resource.py&quot;, line 706, in entry_ops<br>    collect_failure(e, cmd_ret, uid, gid)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/resource.py&quot;, line 444, in collect_failure<br>    disk_gfid)<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py&quot;, line 687, in get_slv_dir_path<br>    [ENOENT], [ESTALE])<br>  File &quot;/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py&quot;, line 546, in errno_wrap<br>    return call(*arg)<br>OSError: [Errno 13] Permission denied: &#39;/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e&#39;<br>[2020-05-22 11:23:32.741317] I [repce(slave <a href="http://10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop" target="_blank">10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop</a>] RepceServer: terminating on reaching EOF.<br></blockquote><div><br></div><div><br></div><div>Additional info:<br>Parallel to this GFID(/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e) mentioned in master gyscnd log, there are some files with ---------T permission &amp; <b>trusted.glusterfs.dht.linkto</b> extended attribute in the master subvolume for which geo-rep session is in faulty status. Not sure whether this is related to this geo-rep issue or not.<br><br>I have attached a few screenshots and log stmts for further info. Please let us know how we should solve this.<br>Thanks in advance.</div></div><div><br></div><div>Regards,</div><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr"><div>Naranderan R</div></div></div></div></div></div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><br></div></div></div></div>