<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi Milind,</p>
<p><br>
</p>
<p> I do not know if this will help, but using ausearch on one of the master nodes gives this:</p>
<p><br>
</p>
<p>----<br>
time->Tue Sep 11 03:28:56 2018<br>
type=PROCTITLE msg=audit(1536629336.548:1202535): proctitle=2F7573722F7362696E2F676C7573746572667364002D73007572642D6764732D303031002D2D766F6C66696C652D6964007572642D6764732D766F6C756D652E7572642D6764732D3030312E7572642D6764732D676C7573746572002D70002F7661722F72756E2F676C75737465722F766F6C732F7572642D6764732D766F6C<br>
type=SYSCALL msg=audit(1536629336.548:1202535): arch=c000003e syscall=87 success=yes exit=0 a0=7f99acce4d70 a1=7f99acce4ed0 a2=1 a3=6533393436373635 items=0 ppid=1 pid=3582 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295
comm="glusteriotwr12" exe="/usr/sbin/glusterfsd" subj=system_u:system_r:glusterd_t:s0 key=(null)<br>
type=AVC msg=audit(1536629336.548:1202535): avc: denied { unlink } for pid=3582 comm="glusteriotwr12" name="67b1d169-14d0-413c-8f76-5676493ea7b8" dev="dm-7" ino=36507351099 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
tclass=sock_file<br>
----<br>
time->Tue Sep 11 04:49:29 2018<br>
type=PROCTITLE msg=audit(1536634169.350:1207973): proctitle=2F7573722F7362696E2F676C7573746572667364002D73007572642D6764732D303031002D2D766F6C66696C652D6964007572642D6764732D766F6C756D652E7572642D6764732D3030312E7572642D6764732D676C7573746572002D70002F7661722F72756E2F676C75737465722F766F6C732F7572642D6764732D766F6C<br>
type=SYSCALL msg=audit(1536634169.350:1207973): arch=c000003e syscall=280 success=yes exit=0 a0=ffffffffffffff9c a1=7f99ad6f5400 a2=7f99ad6f54a0 a3=100 items=0 ppid=1 pid=3582 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none)
ses=4294967295 comm="glusteriotwr12" exe="/usr/sbin/glusterfsd" subj=system_u:system_r:glusterd_t:s0 key=(null)<br>
type=AVC msg=audit(1536634169.350:1207973): avc: denied { setattr } for pid=3582 comm="glusteriotwr12" name="fe949f56-bc6e-411a-9a40-ad6209fbbd89" dev="dm-7" ino=51546135597 scontext=system_u:system_r:glusterd_t:s0 tcontext=system_u:object_r:unlabeled_t:s0
tclass=sock_file</p>
<p><br>
</p>
<p><br>
</p>
<p>Best Regards</p>
<p>Marcus Pedersén</p>
<p><br>
</p>
<p><br>
</p>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Från:</b> Milind Changire <mchangir@redhat.com><br>
<b>Skickat:</b> den 11 september 2018 08:56<br>
<b>Till:</b> Marcus Pedersén<br>
<b>Kopia:</b> Kotresh Hiremath Ravishankar; gluster-users@gluster.org<br>
<b>Ämne:</b> Re: [Gluster-users] Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Marcus,</div>
<div>Is it possible to send over the SELinux errors that you encountered before turning it off ?</div>
<div>We could inspect and get the SELinux issues fixed as an aside.</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 10, 2018 at 4:43 PM, Marcus Pedersén <span dir="ltr">
<<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr" style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi Kotresh,</p>
<p>I have been running 4.1.3 from the end of August.</p>
<p>Since then data has been synced to geo side with a couple of hundred GB per 24 hour, even with the errors I have reported in this thread.</p>
<p><br>
</p>
<p>Four days ago all data transfer to geo side stopped, and the logs repeats the same error over and over again (see below).</p>
<p>Both nodes toggle status Active/Faulty.<br>
</p>
<p><br>
</p>
<p>Thanks alot!</p>
<p><br>
</p>
<p>Best regards</p>
<p>Marcus <br>
</p>
<p><br>
</p>
<p>One master node, gsyncd.log:</p>
<p>[2018-09-10 10:53:38.409709] I [gsyncdstatus(monitor):244:<wbr>set_worker_status] GeorepStatus: Worker Status Change status=Faulty<br>
[2018-09-10 10:53:47.783914] I [gsyncd(config-get):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf<br>
[2018-09-10 10:53:47.852792] I [gsyncd(status):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf<br>
[2018-09-10 10:53:48.421061] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000<br>
[2018-09-10 10:53:48.462655] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf<br>
[2018-09-10 10:53:48.463366] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...<br>
[2018-09-10 10:53:48.465905] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf<br>
[2018-09-10 10:53:48.474558] I [resource(worker /urd-gds/gluster):1377:<wbr>connect_remote] SSH: Initializing SSH connection between master and slave...<br>
[2018-09-10 10:53:50.70219] I [resource(worker /urd-gds/gluster):1424:<wbr>connect_remote] SSH: SSH connection between master and slave established. duration=1.5954<br>
[2018-09-10 10:53:50.70777] I [resource(worker /urd-gds/gluster):1096:<wbr>connect] GLUSTER: Mounting gluster volume locally...<br>
[2018-09-10 10:53:51.170597] I [resource(worker /urd-gds/gluster):1119:<wbr>connect] GLUSTER: Mounted gluster volume duration=1.0994<br>
[2018-09-10 10:53:51.171158] I [subcmds(worker /urd-gds/gluster):70:subcmd_<wbr>worker] <top>: Worker spawn successful. Acknowledging back to monitor<br>
[2018-09-10 10:53:51.696057] I [gsyncd(config-get):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf<br>
[2018-09-10 10:53:51.764605] I [gsyncd(status):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf<br>
[2018-09-10 10:53:53.210553] I [master(worker /urd-gds/gluster):1593:<wbr>register] _GMaster: Working dir path=/var/lib/misc/gluster/<wbr>gsyncd/urd-gds-volume_urd-gds-<wbr>geo-001_urd-gds-volume/urd-<wbr>gds-gluster<br>
[2018-09-10 10:53:53.211148] I [resource(worker /urd-gds/gluster):1282:<wbr>service_loop] GLUSTER: Register time time=1536576833<br>
[2018-09-10 10:53:53.230945] I [gsyncdstatus(worker /urd-gds/gluster):277:set_<wbr>active] GeorepStatus: Worker Status Change status=Active<br>
[2018-09-10 10:53:53.233444] I [gsyncdstatus(worker /urd-gds/gluster):249:set_<wbr>worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl<br>
[2018-09-10 10:53:53.233632] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl turns=1 stime=(1524272046, 0) entry_stime=(1524271940, 0) etime=1536576833<br>
[2018-09-10 10:53:53.234951] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1524272046, 0)<br>
[2018-09-10 10:53:53.762105] I [master(worker /urd-gds/gluster):1944:<wbr>syncjob] Syncer: Sync Time Taken duration=0.0856 num_files=1 job=1 return_code=0<br>
[2018-09-10 10:53:54.437858] I [master(worker /urd-gds/gluster):1374:<wbr>process] _GMaster: Entry Time Taken MKD=0 MKN=0 LIN=0 SYM=0 REN=0 RMD=0 CRE=0 duration=0.0000 UNL=0<br>
[2018-09-10 10:53:54.437973] I [master(worker /urd-gds/gluster):1384:<wbr>process] _GMaster: Data/Metadata Time Taken SETA=0 SETX=0 meta_duration=0.0000 data_duration=1.1979 DATA=1 XATT=0<br>
[2018-09-10 10:53:54.438153] I [master(worker /urd-gds/gluster):1394:<wbr>process] _GMaster: Batch Completed changelog_end=1524272047 <wbr> entry_stime=(1524271940, 0) changelog_start=1524272047 <wbr> stime=(152\<br>
4272046, 0) duration=1.2029 num_changelogs=1 mode=history_changelog<br>
[2018-09-10 10:53:54.482408] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1524272046, 0)<br>
[2018-09-10 10:53:54.583467] E [repce(worker /urd-gds/gluster):197:__call__<wbr>] RepceClient: call failed call=1844:139973681694528:<wbr>1536576834.54 method=entry_ops error=GsyncdError<br>
[2018-09-10 10:53:54.583585] E [syncdutils(worker /urd-gds/gluster):300:log_<wbr>raise_exception] <top>: execution of "gluster" failed with ENOENT (No such file or directory)<br>
[2018-09-10 10:53:54.600353] I [repce(agent /urd-gds/gluster):80:service_<wbr>loop] RepceServer: terminating on reaching EOF.<br>
[2018-09-10 10:53:55.175978] I [monitor(monitor):279:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster<br>
[2018-09-10 10:53:55.182988] I [gsyncdstatus(monitor):244:<wbr>set_worker_status] GeorepStatus: Worker Status Change status=Faulty<br>
[2018-09-10 10:53:56.24414] I [gsyncd(config-get):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>Other master node, gsyncd.log:</p>
<p>[2018-09-10 11:10:43.10458] I [gsyncdstatus(monitor):244:<wbr>set_worker_status] GeorepStatus: Worker Status Change status=Faulty<br>
[2018-09-10 11:10:53.28702] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000<br>
[2018-09-10 11:10:53.69638] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf<br>
[2018-09-10 11:10:53.70264] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...<br>
[2018-09-10 11:10:53.71902] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf<br>
[2018-09-10 11:10:53.80737] I [resource(worker /urd-gds/gluster):1377:<wbr>connect_remote] SSH: Initializing SSH connection between master and slave...<br>
[2018-09-10 11:10:54.621948] I [resource(worker /urd-gds/gluster):1424:<wbr>connect_remote] SSH: SSH connection between master and slave established. duration=1.5410<br>
[2018-09-10 11:10:54.622504] I [resource(worker /urd-gds/gluster):1096:<wbr>connect] GLUSTER: Mounting gluster volume locally...<br>
[2018-09-10 11:10:55.721349] I [resource(worker /urd-gds/gluster):1119:<wbr>connect] GLUSTER: Mounted gluster volume duration=1.0984<br>
[2018-09-10 11:10:55.721913] I [subcmds(worker /urd-gds/gluster):70:subcmd_<wbr>worker] <top>: Worker spawn successful. Acknowledging back to monitor<br>
[2018-09-10 11:10:58.543606] I [master(worker /urd-gds/gluster):1593:<wbr>register] _GMaster: Working dir path=/var/lib/misc/gluster/<wbr>gsyncd/urd-gds-volume_urd-gds-<wbr>geo-001_urd-gds-volume/urd-<wbr>gds-gluster<br>
[2018-09-10 11:10:58.545701] I [resource(worker /urd-gds/gluster):1282:<wbr>service_loop] GLUSTER: Register time time=1536577858<br>
[2018-09-10 11:10:58.564208] I [gsyncdstatus(worker /urd-gds/gluster):277:set_<wbr>active] GeorepStatus: Worker Status Change status=Active<br>
[2018-09-10 11:10:58.565689] I [gsyncdstatus(worker /urd-gds/gluster):249:set_<wbr>worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl<br>
[2018-09-10 11:10:58.565876] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl turns=1 stime=(1527128725, 0) entry_stime=(1527128815, 0) etime=1536577858<br>
[2018-09-10 11:10:59.593652] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1527128725, 0)<br>
[2018-09-10 11:11:01.755116] I [master(worker /urd-gds/gluster):1944:<wbr>syncjob] Syncer: Sync Time Taken duration=0.5233 num_files=103 job=1 return_code=0<br>
[2018-09-10 11:11:02.897665] I [master(worker /urd-gds/gluster):1944:<wbr>syncjob] Syncer: Sync Time Taken duration=0.6648 num_files=116 job=2 return_code=0<br>
[2018-09-10 11:11:03.98150] I [master(worker /urd-gds/gluster):1944:<wbr>syncjob] Syncer: Sync Time Taken duration=0.2003 num_files=59 job=2 return_code=23<br>
[2018-09-10 11:11:03.219059] I [master(worker /urd-gds/gluster):1944:<wbr>syncjob] Syncer: Sync Time Taken duration=0.1207 num_files=16 job=2 return_code=0<br>
[2018-09-10 11:11:03.841105] I [master(worker /urd-gds/gluster):1944:<wbr>syncjob] Syncer: Sync Time Taken duration=0.1212 num_files=32 job=2 return_code=23<br>
[2018-09-10 11:11:04.951658] I [master(worker /urd-gds/gluster):1944:<wbr>syncjob] Syncer: Sync Time Taken duration=0.2160 num_files=24 job=3 return_code=0<br>
[2018-09-10 11:11:05.2938] E [repce(worker /urd-gds/gluster):197:__call__<wbr>] RepceClient: call failed call=2935:140696531339072:<wbr>1536577864.67 method=entry_ops error=GsyncdError<br>
[2018-09-10 11:11:05.3125] E [syncdutils(worker /urd-gds/gluster):300:log_<wbr>raise_exception] <top>: execution of "gluster" failed with ENOENT (No such file or directory)<br>
[2018-09-10 11:11:05.17061] I [repce(agent /urd-gds/gluster):80:service_<wbr>loop] RepceServer: terminating on reaching EOF.<br>
[2018-09-10 11:11:05.733716] I [monitor(monitor):279:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster<br>
[2018-09-10 11:11:05.768186] I [gsyncdstatus(monitor):244:<wbr>set_worker_status] GeorepStatus: Worker Status Change status=Faulty<br>
[2018-09-10 11:11:15.788830] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000<br>
[2018-09-10 11:11:15.829871] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-<wbr>replication/urd-gds-volume_<wbr>urd-gds-geo-001_urd-gds-<wbr>volume/gsyncd.conf</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div style="color:rgb(33,33,33)">
<hr style="display:inline-block; width:98%">
<div id="m_1340459009448613356divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><span class=""><b>Från:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>><br>
</span><b>Skickat:</b> den 3 september 2018 07:58
<div>
<div class="h5"><br>
<b>Till:</b> Marcus Pedersén<br>
<b>Kopia:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Ämne:</b> Re: [Gluster-users] Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="h5">
<div>
<div dir="ltr">
<div>
<div>
<div>Hi Marcus,<br>
<br>
</div>
Geo-rep had few important fixes in 4.1.3. Is it possible to upgrade and check whether the issue is still seen?<br>
<br>
</div>
Thanks,<br>
</div>
Kotresh HR<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Sep 1, 2018 at 5:08 PM, Marcus Pedersén <span dir="ltr">
<<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr" style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi again,</p>
<p>I found another problem on the other master node.</p>
<p>The node toggles Active/Faulty and it is the same error over and over again.</p>
<p><br>
</p>
<p>[2018-09-01 11:23:02.94080] E [repce(worker /urd-gds/gluster):197:__call__<wbr>] RepceClient: call failed call=1226:139955262510912:1535<wbr>800981.24 method=entry_ops error=GsyncdError<br>
[2018-09-01 11:23:02.94214] E [syncdutils(worker /urd-gds/gluster):300:log_rais<wbr>e_exception] <top>: execution of "gluster" failed with ENOENT (No such file or directory)<br>
[2018-09-01 11:23:02.106194] I [repce(agent /urd-gds/gluster):80:service_l<wbr>oop] RepceServer: terminating on reaching EOF.<br>
[2018-09-01 11:23:02.124444] I [gsyncdstatus(monitor):244:set<wbr>_worker_status] GeorepStatus: Worker Status Change status=Faulty</p>
<p><br>
</p>
<p>I have also found a python error as well, I have only seen this once though.</p>
<p><br>
</p>
<p>[2018-09-01 11:16:45.907660] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1524101534, 0)<br>
[2018-09-01 11:16:47.364109] E [syncdutils(worker /urd-gds/gluster):332:log_rais<wbr>e_exception] <top>: FAIL:<span><br>
Traceback (most recent call last):<br>
</span> File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/syncdutils.py", line 362, in twrap<br>
tf(*aargs)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/master.py", line 1939, in syncjob<br>
po = self.sync_engine(pb, self.log_err)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/resource.py", line 1442, in rsync<br>
rconf.ssh_ctl_args + \<br>
AttributeError: 'NoneType' object has no attribute 'split'<br>
[2018-09-01 11:16:47.384531] I [repce(agent /urd-gds/gluster):80:service_l<wbr>oop] RepceServer: terminating on reaching EOF.<br>
[2018-09-01 11:16:48.362987] I [monitor(monitor):279:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster<br>
[2018-09-01 11:16:48.370701] I [gsyncdstatus(monitor):244:set<wbr>_worker_status] GeorepStatus: Worker Status Change status=Faulty<br>
[2018-09-01 11:16:58.390548] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000</p>
<p><br>
</p>
<p>I attach the logs as well.</p>
<p><br>
</p>
<p>Many thanks!</p>
<p><br>
</p>
<p>Best regards</p>
<p>Marcus Pedersén</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<div dir="ltr" style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<hr style="display:inline-block; width:98%">
<div id="m_1340459009448613356m_1594857026570226030divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Från:</b>
<a href="mailto:gluster-users-bounces@gluster.org" target="_blank">gluster-users-bounces@gluster.<wbr>org</a> <<a href="mailto:gluster-users-bounces@gluster.org" target="_blank">gluster-users-bounces@gluster<wbr>.org</a>> för Marcus Pedersén <<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>><br>
<b>Skickat:</b> den 31 augusti 2018 16:09<br>
<b>Till:</b> <a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>
<div>
<div class="m_1340459009448613356h5"><br>
<b>Kopia:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Ämne:</b> Re: [Gluster-users] Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_1340459009448613356h5">
<div>
<p>I realy appologize, third try to make mail smaller.</p>
<p><br>
</p>
<p>/Marcus </p>
<p><br>
</p>
<div dir="ltr" style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<hr style="display:inline-block; width:98%">
<div id="m_1340459009448613356m_1594857026570226030divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Från:</b> Marcus Pedersén<br>
<b>Skickat:</b> den 31 augusti 2018 16:03<br>
<b>Till:</b> Kotresh Hiremath Ravishankar<br>
<b>Kopia:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Ämne:</b> SV: [Gluster-users] Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty</font>
<div> </div>
</div>
<div>
<p>Sorry, resend due to too large mail.</p>
<p><br>
</p>
<p>/Marcus<br>
</p>
<div dir="ltr" style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<hr style="display:inline-block; width:98%">
<div id="m_1340459009448613356m_1594857026570226030divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Från:</b> Marcus Pedersén<br>
<b>Skickat:</b> den 31 augusti 2018 15:19<br>
<b>Till:</b> Kotresh Hiremath Ravishankar<br>
<b>Kopia:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Ämne:</b> SV: [Gluster-users] Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty</font>
<div> </div>
</div>
<div>
<p>Hi Kotresh,</p>
<p>Please find attached logs, only logs from today.</p>
<p>The python error was repeated over and over again until I disabled selinux.</p>
<p>After that the node bacame active again.</p>
<p>The return code 23 seems to be repeated over and over again.<br>
</p>
<p><br>
</p>
<p>rsync version 3.1.2</p>
<p><br>
</p>
<p>Thanks a lot!</p>
<p><br>
</p>
<p>Best regards</p>
<p>Marcus<br>
</p>
<p><br>
</p>
<div style="color:rgb(33,33,33)">
<hr style="display:inline-block; width:98%">
<div id="m_1340459009448613356m_1594857026570226030divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>Från:</b> Kotresh Hiremath Ravishankar <<a href="mailto:khiremat@redhat.com" target="_blank">khiremat@redhat.com</a>><br>
<b>Skickat:</b> den 31 augusti 2018 11:09<br>
<b>Till:</b> Marcus Pedersén<br>
<b>Kopia:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Ämne:</b> Re: [Gluster-users] Was: Upgrade to 4.1.2 geo-replication does not work Now: Upgraded to 4.1.3 geo node Faulty</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>Hi Marcus,<br>
<br>
</div>
Could you attach full logs? Is the same trace back happening repeatedly? It will be helpful you attach the corresponding mount log as well.<br>
</div>
What's the rsync version, you are using?<br>
<br>
</div>
Thanks,<br>
</div>
Kotresh HR<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Aug 31, 2018 at 12:16 PM, Marcus Pedersén <span dir="ltr">
<<a href="mailto:marcus.pedersen@slu.se" target="_blank">marcus.pedersen@slu.se</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr" style="font-size:12pt; color:#000000; background-color:#ffffff; font-family:Calibri,Arial,Helvetica,sans-serif">
<p>Hi all,</p>
<p>I had problems with stopping sync after upgrade to 4.1.2.</p>
<p>I upgraded to 4.1.3 and it ran fine for one day, but now one of the master nodes shows faulty.</p>
<p>Most of the sync jobs have return code 23, how do I resolve this?</p>
<p>I see messages like: </p>
<p>_GMaster: Sucessfully fixed all entry ops with gfid mismatch</p>
<p>Will this resolve error code 23?<br>
</p>
<p>There is also a python error.</p>
<p>The python error was a selinux problem, turning off selinux made node go to active again.<br>
</p>
<p>See log below.<br>
</p>
<p><br>
</p>
<p>CentOS 7, installed through SIG Gluster (OS updated to latest at the same time)<br>
</p>
<p>Master cluster: 2 x (2 + 1) distributed, replicated</p>
<p>Client cluster: 1 x (2 + 1) replicated</p>
<p><br>
</p>
<p>Many thanks in advance!</p>
<p><br>
</p>
<p>Best regards</p>
<p>Marcus Pedersén<br>
</p>
<p><br>
</p>
<p><br>
</p>
<p>gsyncd.log from Faulty node:</p>
<p>[2018-08-31 06:25:51.375267] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.8099 num_files=57 job=3 return_code=23<br>
[2018-08-31 06:25:51.465895] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0904 num_files=3 job=3 return_code=23<br>
[2018-08-31 06:25:52.562107] E [repce(worker /urd-gds/gluster):197:__call__<wbr>] RepceClient: call failed call=30069:139655665837888:153<wbr>5696752.35 method=entry_ops error=OSError<br>
[2018-08-31 06:25:52.562346] E [syncdutils(worker /urd-gds/gluster):332:log_rais<wbr>e_exception] <top>: FAIL:<br>
Traceback (most recent call last):<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/gsyncd.py", line 311, in main<br>
func(args)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/subcmds.py", line 72, in subcmd_worker<br>
local.service_loop(remote)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/resource.py", line 1288, in service_loop<br>
g3.crawlwrap(oneshot=True)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/master.py", line 615, in crawlwrap<br>
self.crawl()<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/master.py", line 1545, in crawl<br>
self.changelogs_batch_process(<wbr>changes)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/master.py", line 1445, in changelogs_batch_process<br>
self.process(batch)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/master.py", line 1280, in process<br>
self.process_change(change, done, retry)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/master.py", line 1179, in process_change<br>
failures = self.slave.server.entry_ops(en<wbr>tries)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/repce.py", line 216, in __call__<br>
return self.ins(self.meth, *a)<br>
File "/usr/libexec/glusterfs/python<wbr>/syncdaemon/repce.py", line 198, in __call__<br>
raise res<br>
OSError: [Errno 13] Permission denied<br>
[2018-08-31 06:25:52.578367] I [repce(agent /urd-gds/gluster):80:service_l<wbr>oop] RepceServer: terminating on reaching EOF.<br>
[2018-08-31 06:25:53.558765] I [monitor(monitor):279:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster<br>
[2018-08-31 06:25:53.569777] I [gsyncdstatus(monitor):244:set<wbr>_worker_status] GeorepStatus: Worker Status Change status=Faulty<br>
[2018-08-31 06:26:03.593161] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000<br>
[2018-08-31 06:26:03.636452] I [gsyncd(agent /urd-gds/gluster):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-rep<wbr>lication/urd-gds-volume_urd-gd<wbr>s-geo-001_urd-gds-volume/gsync<wbr>d.conf<br>
[2018-08-31 06:26:03.636810] I [gsyncd(worker /urd-gds/gluster):297:main] <top>: Using session config file path=/var/lib/glusterd/geo-rep<wbr>lication/urd-gds-volume_urd-gd<wbr>s-geo-001_urd-gds-volume/gsync<wbr>d.conf<br>
[2018-08-31 06:26:03.637486] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining...<br>
[2018-08-31 06:26:03.650330] I [resource(worker /urd-gds/gluster):1377:connect<wbr>_remote] SSH: Initializing SSH connection between master and slave...<br>
[2018-08-31 06:26:05.296473] I [resource(worker /urd-gds/gluster):1424:connect<wbr>_remote] SSH: SSH connection between master and slave established. duration=1.6457<br>
[2018-08-31 06:26:05.297904] I [resource(worker /urd-gds/gluster):1096:connect<wbr>] GLUSTER: Mounting gluster volume locally...<br>
[2018-08-31 06:26:06.396939] I [resource(worker /urd-gds/gluster):1119:connect<wbr>] GLUSTER: Mounted gluster volume duration=1.0985<br>
[2018-08-31 06:26:06.397691] I [subcmds(worker /urd-gds/gluster):70:subcmd_wo<wbr>rker] <top>: Worker spawn successful. Acknowledging back to monitor<br>
[2018-08-31 06:26:16.815566] I [master(worker /urd-gds/gluster):1593:registe<wbr>r] _GMaster: Working dir path=/var/lib/misc/gluster/gsy<wbr>ncd/urd-gds-volume_urd-gds-geo<wbr>-001_urd-gds-volume/urd-gds-gl<wbr>uster<br>
[2018-08-31 06:26:16.816423] I [resource(worker /urd-gds/gluster):1282:service<wbr>_loop] GLUSTER: Register time time=1535696776<br>
[2018-08-31 06:26:16.888772] I [gsyncdstatus(worker /urd-gds/gluster):277:set_acti<wbr>ve] GeorepStatus: Worker Status Change status=Active<br>
[2018-08-31 06:26:16.892049] I [gsyncdstatus(worker /urd-gds/gluster):249:set_work<wbr>er_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl<br>
[2018-08-31 06:26:16.892703] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl turns=1 stime=(1525739167, 0) entry_stime=(1525740143, 0) etime=1535696776<br>
[2018-08-31 06:26:17.914803] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1525739167, 0)<br>
[2018-08-31 06:26:18.521718] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.1063 num_files=17 job=3 return_code=23<br>
[2018-08-31 06:26:19.260137] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.3441 num_files=34 job=1 return_code=23<br>
[2018-08-31 06:26:19.615191] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0923 num_files=7 job=3 return_code=23<br>
[2018-08-31 06:26:19.891227] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.1302 num_files=12 job=1 return_code=23<br>
[2018-08-31 06:26:19.922700] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.5024 num_files=50 job=2 return_code=23</p>
<p>[2018-08-31 06:26:21.639342] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=1.5233 num_files=5 job=3 return_code=23<br>
[2018-08-31 06:26:22.12726] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.1191 num_files=7 job=1 return_code=23<br>
[2018-08-31 06:26:22.86136] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0731 num_files=4 job=1 return_code=23<br>
[2018-08-31 06:26:22.503290] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0779 num_files=15 job=2 return_code=23<br>
[2018-08-31 06:26:23.214704] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0738 num_files=9 job=3 return_code=23<br>
[2018-08-31 06:26:23.251876] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.2478 num_files=33 job=2 return_code=23<br>
[2018-08-31 06:26:23.802699] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0873 num_files=9 job=3 return_code=23<br>
[2018-08-31 06:26:23.828176] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0758 num_files=3 job=2 return_code=23<br>
[2018-08-31 06:26:23.854063] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.2662 num_files=34 job=1 return_code=23<br>
[2018-08-31 06:26:24.403228] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0997 num_files=30 job=3 return_code=23<br>
[2018-08-31 06:26:25.526] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0965 num_files=8 job=3 return_code=23<br>
[2018-08-31 06:26:25.438527] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0832 num_files=9 job=1 return_code=23<br>
[2018-08-31 06:26:25.447256] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.6180 num_files=86 job=2 return_code=23<br>
[2018-08-31 06:26:25.571913] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0706 num_files=2 job=3 return_code=0<br>
[2018-08-31 06:26:27.21325] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0814 num_files=1 job=1 return_code=23<br>
[2018-08-31 06:26:27.615520] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0933 num_files=13 job=1 return_code=23<br>
[2018-08-31 06:26:27.668323] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.2190 num_files=95 job=2 return_code=23<br>
[2018-08-31 06:26:27.740139] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0716 num_files=11 job=2 return_code=23<br>
[2018-08-31 06:26:28.191068] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.1167 num_files=38 job=3 return_code=23<br>
[2018-08-31 06:26:28.268213] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0768 num_files=7 job=3 return_code=23<br>
[2018-08-31 06:26:28.317909] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0770 num_files=4 job=2 return_code=23<br>
[2018-08-31 06:26:28.710064] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0932 num_files=23 job=1 return_code=23<br>
[2018-08-31 06:26:28.907250] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0886 num_files=26 job=2 return_code=23<br>
[2018-08-31 06:26:28.976679] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0692 num_files=4 job=2 return_code=23<br>
[2018-08-31 06:26:29.55774] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0788 num_files=9 job=2 return_code=23<br>
[2018-08-31 06:26:29.295576] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0847 num_files=16 job=1 return_code=23<br>
[2018-08-31 06:26:29.665076] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.1087 num_files=25 job=2 return_code=23<br>
[2018-08-31 06:26:30.277998] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.1122 num_files=40 job=2 return_code=23<br>
[2018-08-31 06:26:31.153105] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.3822 num_files=74 job=3 return_code=23<br>
[2018-08-31 06:26:31.227639] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0743 num_files=18 job=3 return_code=23<br>
[2018-08-31 06:26:31.302660] I [master(worker /urd-gds/gluster):1944:syncjob<wbr>] Syncer: Sync Time Taken duration=0.0748 num_files=18 job=3 return_code=23</p>
<p><br>
</p>
<p>---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">
här </a><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">
here </a></p>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="m_1340459009448613356m_1594857026570226030gmail_signature">
<div dir="ltr">
<div>Thanks and Regards,<br>
</div>
Kotresh H R<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">
här </a><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">
here </a></p>
</div>
</div>
</div>
</div>
<div>
<div class="m_1340459009448613356h5">
<p>---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">
här </a><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">
here </a></p>
</div>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="m_1340459009448613356gmail_signature">
<div dir="ltr">
<div>Thanks and Regards,<br>
</div>
Kotresh H R<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div class="h5">
<p>---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/" target="_blank">
här </a><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/" target="_blank">
here </a></p>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
<div class="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">Milind<br>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p>---<br>
När du skickar e-post till SLU så innebär detta att SLU behandlar dina personuppgifter. För att läsa mer om hur detta går till, klicka
<a href="https://www.slu.se/om-slu/kontakta-slu/personuppgifter/">här </a><br>
E-mailing SLU will result in SLU processing your personal data. For more information on how this is done, click
<a href="https://www.slu.se/en/about-slu/contact-slu/personal-data/">here </a></p>
</body>
</html>