[2018-09-01 11:16:10.909337] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.8130 num_files=92 job=1 return_code=23 [2018-09-01 11:16:16.659548] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7450 num_files=83 job=1 return_code=0 [2018-09-01 11:16:19.382323] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.3692 num_files=85 job=3 return_code=23 [2018-09-01 11:16:19.402731] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed call=3934:139973243336512:1535800578.96 method=entry_ops error=GsyncdError [2018-09-01 11:16:19.402901] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] : execution of "gluster" failed with ENOENT (No such file or directory) [2018-09-01 11:16:19.418284] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF. [2018-09-01 11:16:19.439542] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2018-09-01 11:16:29.456732] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-09-01 11:16:29.526350] I [gsyncd(agent /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:16:29.527015] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-09-01 11:16:29.529350] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:16:29.538482] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-09-01 11:16:31.212218] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established. duration=1.6735 [2018-09-01 11:16:31.212785] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally... [2018-09-01 11:16:32.344389] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume duration=1.1312 [2018-09-01 11:16:32.344978] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2018-09-01 11:16:44.829792] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster [2018-09-01 11:16:44.830971] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time time=1535800604 [2018-09-01 11:16:44.884566] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change status=Active [2018-09-01 11:16:44.886054] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl [2018-09-01 11:16:44.886224] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl turns=1 stime=(1524101534, 0) entry_stime=(1524101804, 0) etime=1535800604 [2018-09-01 11:16:45.907660] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1524101534, 0) [2018-09-01 11:16:47.364109] E [syncdutils(worker /urd-gds/gluster):332:log_raise_exception] : FAIL: Traceback (most recent call last): File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 362, in twrap tf(*aargs) File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1939, in syncjob po = self.sync_engine(pb, self.log_err) File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1442, in rsync rconf.ssh_ctl_args + \ AttributeError: 'NoneType' object has no attribute 'split' [2018-09-01 11:16:47.384531] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF. [2018-09-01 11:16:48.362987] I [monitor(monitor):279:monitor] Monitor: worker died in startup phase brick=/urd-gds/gluster [2018-09-01 11:16:48.370701] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2018-09-01 11:16:58.390548] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-09-01 11:16:58.462002] I [gsyncd(agent /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:16:58.462469] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:16:58.462757] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-09-01 11:16:58.471395] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-09-01 11:17:00.62880] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established. duration=1.5912 [2018-09-01 11:17:00.63438] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally... [2018-09-01 11:17:01.164509] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume duration=1.1006 [2018-09-01 11:17:01.165066] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2018-09-01 11:17:03.448198] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster [2018-09-01 11:17:03.448826] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time time=1535800623 [2018-09-01 11:17:03.491442] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change status=Active [2018-09-01 11:17:03.493484] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl [2018-09-01 11:17:03.493901] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl turns=1 stime=(1524101534, 0) entry_stime=(1524101804, 0) etime=1535800623 [2018-09-01 11:17:04.513282] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1524101534, 0) [2018-09-01 11:17:08.725709] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5496 num_files=113 job=1 return_code=23 [2018-09-01 11:17:13.759978] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=3.5809 num_files=107 job=2 return_code=23 [2018-09-01 11:17:17.215708] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0307 num_files=116 job=3 return_code=23 [2018-09-01 11:17:22.900979] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.6799 num_files=116 job=3 return_code=23 [2018-09-01 11:17:23.799818] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.3979 num_files=100 job=3 return_code=23 [2018-09-01 11:17:25.914112] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.1130 num_files=97 job=3 return_code=23 [2018-09-01 11:17:30.17796] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7712 num_files=122 job=1 return_code=23 [2018-09-01 11:17:31.464807] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5453 num_files=59 job=3 return_code=23 [2018-09-01 11:17:34.449332] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.6688 num_files=124 job=2 return_code=23 [2018-09-01 11:17:35.528789] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5787 num_files=82 job=2 return_code=23 [2018-09-01 11:17:43.626379] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=2.6518 num_files=131 job=3 return_code=23 [2018-09-01 11:17:45.60198] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.9330 num_files=68 job=3 return_code=0 [2018-09-01 11:17:52.924736] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=2.3856 num_files=123 job=1 return_code=0 [2018-09-01 11:17:55.471108] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5440 num_files=131 job=1 return_code=23 [2018-09-01 11:17:56.567857] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5962 num_files=64 job=1 return_code=0 [2018-09-01 11:18:05.184760] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.1271 num_files=125 job=2 return_code=0 [2018-09-01 11:18:06.305181] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7270 num_files=92 job=1 return_code=23 [2018-09-01 11:18:10.277966] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.6934 num_files=83 job=3 return_code=0 [2018-09-01 11:18:15.348804] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.1544 num_files=85 job=2 return_code=23 [2018-09-01 11:18:17.695431] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed call=7352:140551148898112:1535800696.72 method=entry_ops error=GsyncdError [2018-09-01 11:18:17.695614] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] : execution of "gluster" failed with ENOENT (No such file or directory) [2018-09-01 11:18:17.711199] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF. [2018-09-01 11:18:17.738365] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2018-09-01 11:18:27.760206] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-09-01 11:18:27.831969] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:18:27.841169] I [gsyncd(agent /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:18:27.842202] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-09-01 11:18:27.872488] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-09-01 11:18:29.449333] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established. duration=1.5766 [2018-09-01 11:18:29.449908] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally... [2018-09-01 11:18:30.571775] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume duration=1.1215 [2018-09-01 11:18:30.572541] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2018-09-01 11:18:41.219701] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster [2018-09-01 11:18:41.220571] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time time=1535800721 [2018-09-01 11:18:41.882281] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change status=Active [2018-09-01 11:18:41.884654] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl [2018-09-01 11:18:41.884835] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl turns=1 stime=(1524101534, 0) entry_stime=(1524101804, 0) etime=1535800721 [2018-09-01 11:18:42.904170] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1524101534, 0) [2018-09-01 11:18:47.568198] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.4760 num_files=113 job=3 return_code=23 [2018-09-01 11:18:51.249831] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=2.6803 num_files=107 job=3 return_code=23 [2018-09-01 11:18:54.151258] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0522 num_files=116 job=2 return_code=23 [2018-09-01 11:18:59.42026] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.4375 num_files=116 job=1 return_code=23 [2018-09-01 11:19:01.320400] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0612 num_files=100 job=3 return_code=23 [2018-09-01 11:19:03.631659] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.8092 num_files=97 job=3 return_code=23 [2018-09-01 11:19:07.380134] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.8306 num_files=122 job=1 return_code=23 [2018-09-01 11:19:08.305370] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.4248 num_files=59 job=1 return_code=23 [2018-09-01 11:19:14.602133] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=3.9631 num_files=124 job=3 return_code=23 [2018-09-01 11:19:22.315196] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=3.2089 num_files=82 job=3 return_code=23 [2018-09-01 11:19:28.428549] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7422 num_files=131 job=2 return_code=23 [2018-09-01 11:19:31.183011] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.8598 num_files=68 job=3 return_code=0 [2018-09-01 11:19:34.593025] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7619 num_files=123 job=1 return_code=0 [2018-09-01 11:19:40.10251] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0700 num_files=131 job=2 return_code=23 [2018-09-01 11:19:41.427376] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.3273 num_files=64 job=1 return_code=0 [2018-09-01 11:19:48.773497] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.2509 num_files=125 job=2 return_code=0 [2018-09-01 11:19:55.991848] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7845 num_files=92 job=3 return_code=23 [2018-09-01 11:19:57.328516] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5470 num_files=83 job=2 return_code=0 [2018-09-01 11:20:00.677280] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.1818 num_files=85 job=3 return_code=23 [2018-09-01 11:20:01.225960] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed call=31640:140394415150912:1535800800.37 method=entry_ops error=GsyncdError [2018-09-01 11:20:01.226191] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] : execution of "gluster" failed with ENOENT (No such file or directory) [2018-09-01 11:20:01.239264] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF. [2018-09-01 11:20:01.246619] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2018-09-01 11:20:11.265338] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-09-01 11:20:11.336337] I [gsyncd(agent /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:20:11.336961] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:20:11.337119] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-09-01 11:20:11.346508] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-09-01 11:20:12.882986] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established. duration=1.5362 [2018-09-01 11:20:12.883548] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally... [2018-09-01 11:20:14.585222] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume duration=1.7013 [2018-09-01 11:20:14.585820] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2018-09-01 11:20:27.485044] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster [2018-09-01 11:20:27.486057] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time time=1535800827 [2018-09-01 11:20:27.522258] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change status=Active [2018-09-01 11:20:27.523973] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl [2018-09-01 11:20:27.524256] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl turns=1 stime=(1524101534, 0) entry_stime=(1524101804, 0) etime=1535800827 [2018-09-01 11:20:28.543723] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1524101534, 0) [2018-09-01 11:20:31.499804] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.8938 num_files=113 job=2 return_code=23 [2018-09-01 11:20:34.94470] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.5934 num_files=107 job=2 return_code=23 [2018-09-01 11:20:37.100281] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5032 num_files=116 job=2 return_code=23 [2018-09-01 11:20:39.27117] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.9256 num_files=116 job=2 return_code=23 [2018-09-01 11:20:40.534573] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5062 num_files=100 job=2 return_code=23 [2018-09-01 11:20:45.553112] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0141 num_files=97 job=2 return_code=23 [2018-09-01 11:20:46.769187] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7157 num_files=122 job=2 return_code=23 [2018-09-01 11:20:47.585594] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.3161 num_files=59 job=2 return_code=23 [2018-09-01 11:20:50.111612] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5236 num_files=124 job=2 return_code=23 [2018-09-01 11:20:51.388463] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7754 num_files=82 job=2 return_code=23 [2018-09-01 11:20:56.437892] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.3078 num_files=131 job=3 return_code=23 [2018-09-01 11:20:58.168567] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.2740 num_files=68 job=2 return_code=0 [2018-09-01 11:21:00.717818] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5828 num_files=123 job=1 return_code=0 [2018-09-01 11:21:07.217396] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7691 num_files=131 job=3 return_code=23 [2018-09-01 11:21:08.989913] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.3103 num_files=64 job=2 return_code=0 [2018-09-01 11:21:13.825584] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=3.3339 num_files=125 job=2 return_code=0 [2018-09-01 11:21:18.233908] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5051 num_files=92 job=3 return_code=23 [2018-09-01 11:21:22.143229] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=3.3122 num_files=83 job=2 return_code=0 [2018-09-01 11:21:29.814635] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.6416 num_files=85 job=2 return_code=23 [2018-09-01 11:21:30.107693] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed call=22967:140511351723840:1535800889.35 method=entry_ops error=GsyncdError [2018-09-01 11:21:30.107831] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] : execution of "gluster" failed with ENOENT (No such file or directory) [2018-09-01 11:21:30.123255] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF. [2018-09-01 11:21:30.151117] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2018-09-01 11:21:40.173165] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-09-01 11:21:40.221231] I [gsyncd(agent /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:21:40.223439] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:21:40.223797] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-09-01 11:21:40.248348] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-09-01 11:21:41.869673] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established. duration=1.6210 [2018-09-01 11:21:41.870523] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally... [2018-09-01 11:21:42.999773] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume duration=1.1288 [2018-09-01 11:21:43.335] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2018-09-01 11:21:51.755662] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster [2018-09-01 11:21:51.756623] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time time=1535800911 [2018-09-01 11:21:51.774579] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change status=Active [2018-09-01 11:21:51.776083] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl [2018-09-01 11:21:51.776257] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl turns=1 stime=(1524101534, 0) entry_stime=(1524101804, 0) etime=1535800911 [2018-09-01 11:21:52.796566] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1524101534, 0) [2018-09-01 11:21:55.587325] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5713 num_files=113 job=2 return_code=23 [2018-09-01 11:21:57.472486] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.4549 num_files=107 job=1 return_code=23 [2018-09-01 11:21:59.762489] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.2890 num_files=116 job=1 return_code=23 [2018-09-01 11:22:02.462059] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.1976 num_files=116 job=1 return_code=23 [2018-09-01 11:22:04.363795] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7686 num_files=100 job=2 return_code=23 [2018-09-01 11:22:06.338656] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.9736 num_files=97 job=2 return_code=23 [2018-09-01 11:22:08.780875] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.9404 num_files=122 job=2 return_code=23 [2018-09-01 11:22:11.106581] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.1371 num_files=59 job=1 return_code=23 [2018-09-01 11:22:16.526092] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.4895 num_files=124 job=3 return_code=23 [2018-09-01 11:22:18.71861] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0447 num_files=82 job=3 return_code=23 [2018-09-01 11:22:22.375686] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.7998 num_files=131 job=3 return_code=23 [2018-09-01 11:22:25.157756] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=2.5396 num_files=68 job=1 return_code=0 [2018-09-01 11:22:33.463901] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0774 num_files=123 job=3 return_code=0 [2018-09-01 11:22:37.682714] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0129 num_files=131 job=1 return_code=23 [2018-09-01 11:22:39.574531] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.3898 num_files=64 job=1 return_code=0 [2018-09-01 11:22:48.546103] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.9546 num_files=125 job=1 return_code=0 [2018-09-01 11:22:55.26038] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0415 num_files=92 job=3 return_code=23 [2018-09-01 11:22:58.212902] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.3818 num_files=83 job=2 return_code=0 [2018-09-01 11:23:00.388211] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.8306 num_files=85 job=1 return_code=23 [2018-09-01 11:23:02.94080] E [repce(worker /urd-gds/gluster):197:__call__] RepceClient: call failed call=1226:139955262510912:1535800981.24 method=entry_ops error=GsyncdError [2018-09-01 11:23:02.94214] E [syncdutils(worker /urd-gds/gluster):300:log_raise_exception] : execution of "gluster" failed with ENOENT (No such file or directory) [2018-09-01 11:23:02.106194] I [repce(agent /urd-gds/gluster):80:service_loop] RepceServer: terminating on reaching EOF. [2018-09-01 11:23:02.124444] I [gsyncdstatus(monitor):244:set_worker_status] GeorepStatus: Worker Status Change status=Faulty [2018-09-01 11:23:12.133107] I [monitor(monitor):158:monitor] Monitor: starting gsyncd worker brick=/urd-gds/gluster slave_node=urd-gds-geo-000 [2018-09-01 11:23:12.180077] I [gsyncd(worker /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:23:12.183568] I [gsyncd(agent /urd-gds/gluster):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:23:12.184608] I [changelogagent(agent /urd-gds/gluster):72:__init__] ChangelogAgent: Agent listining... [2018-09-01 11:23:13.353393] I [resource(worker /urd-gds/gluster):1377:connect_remote] SSH: Initializing SSH connection between master and slave... [2018-09-01 11:23:14.927974] I [resource(worker /urd-gds/gluster):1424:connect_remote] SSH: SSH connection between master and slave established. duration=1.5740 [2018-09-01 11:23:14.928534] I [resource(worker /urd-gds/gluster):1096:connect] GLUSTER: Mounting gluster volume locally... [2018-09-01 11:23:16.29083] I [resource(worker /urd-gds/gluster):1119:connect] GLUSTER: Mounted gluster volume duration=1.1001 [2018-09-01 11:23:16.29679] I [subcmds(worker /urd-gds/gluster):70:subcmd_worker] : Worker spawn successful. Acknowledging back to monitor [2018-09-01 11:23:24.394258] I [gsyncd(config-get):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:23:24.463059] I [gsyncd(status):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:23:29.892555] I [master(worker /urd-gds/gluster):1593:register] _GMaster: Working dir path=/var/lib/misc/gluster/gsyncd/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/urd-gds-gluster [2018-09-01 11:23:29.893162] I [resource(worker /urd-gds/gluster):1282:service_loop] GLUSTER: Register time time=1535801009 [2018-09-01 11:23:29.927710] I [gsyncdstatus(worker /urd-gds/gluster):277:set_active] GeorepStatus: Worker Status Change status=Active [2018-09-01 11:23:29.930145] I [gsyncdstatus(worker /urd-gds/gluster):249:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl [2018-09-01 11:23:29.930346] I [master(worker /urd-gds/gluster):1507:crawl] _GMaster: starting history crawl turns=1 stime=(1524101534, 0) entry_stime=(1524101804, 0) etime=1535801009 [2018-09-01 11:23:30.951765] I [master(worker /urd-gds/gluster):1536:crawl] _GMaster: slave's time stime=(1524101534, 0) [2018-09-01 11:23:33.237665] I [gsyncd(config-get):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:23:33.314560] I [gsyncd(status):297:main] : Using session config file path=/var/lib/glusterd/geo-replication/urd-gds-volume_urd-gds-geo-001_urd-gds-volume/gsyncd.conf [2018-09-01 11:23:33.901304] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.8518 num_files=113 job=1 return_code=23 [2018-09-01 11:23:36.586030] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.0335 num_files=107 job=3 return_code=23 [2018-09-01 11:23:38.482296] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=0.5768 num_files=116 job=1 return_code=23 [2018-09-01 11:23:41.598927] I [master(worker /urd-gds/gluster):1944:syncjob] Syncer: Sync Time Taken duration=1.1144 num_files=116 job=1 return_code=23