<div dir="ltr">Could you try disabling syncing xattrs and check ?<div><br></div><div>gluster vol geo-rep <mastervol> <slavehost>::<savevol> config sync-xattrs false</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 13, 2020 at 1:42 AM Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On March 12, 2020 9:41:45 AM GMT+02:00, "Etem Bayoğlu" <<a href="mailto:etembayoglu@gmail.com" target="_blank">etembayoglu@gmail.com</a>> wrote:<br>
>Hello again,<br>
><br>
>These are gsyncd.log from master on DEBUG level. It tells entering<br>
>directory, synced files , and gfid information<br>
><br>
>[2020-03-12 07:18:16.702286] D [master(worker<br>
>/srv/media-storage):324:regjob] _GMaster: synced<br>
>file=.gfid/358fe62c-c7e8-449a-90dd-1cc1a3b7a346<br>
>[2020-03-12 07:18:16.702420] D [master(worker<br>
>/srv/media-storage):324:regjob] _GMaster: synced<br>
>file=.gfid/04eb63e3-7fcb-45d2-9f29-6292a5072adb<br>
>[2020-03-12 07:18:16.702574] D [master(worker<br>
>/srv/media-storage):324:regjob] _GMaster: synced<br>
>file=.gfid/4363e521-d81a-4a0f-bfa4-5ee6b92da2b4<br>
>[2020-03-12 07:18:16.702704] D [master(worker<br>
>/srv/media-storage):324:regjob] _GMaster: synced<br>
>file=.gfid/bed30509-2c5f-4c77-b2f9-81916a99abd9<br>
>[2020-03-12 07:18:16.702828] D [master(worker<br>
>/srv/media-storage):324:regjob] _GMaster: synced<br>
>file=.gfid/d86f44cc-3001-4bdf-8bae-6bed2a9c8381<br>
>[2020-03-12 07:18:16.702950] D [master(worker<br>
>/srv/media-storage):324:regjob] _GMaster: synced<br>
>file=.gfid/da40d429-d89e-4dc9-9dda-07922d87b3c8<br>
>[2020-03-12 07:18:16.703075] D [master(worker<br>
>/srv/media-storage):324:regjob] _GMaster: synced<br>
>file=.gfid/befc5e03-b7a1-43dc-b6c2-0a186019b6d5<br>
>[2020-03-12 07:18:16.703198] D [master(worker<br>
>/srv/media-storage):324:regjob] _GMaster: synced<br>
>file=.gfid/4e66035f-99f9-4802-b876-2e01686d18f2<br>
>[2020-03-12 07:18:16.703378] D [master(worker<br>
>/srv/media-storage):324:regjob] _GMaster: synced<br>
>file=.gfid/d1295b51-e461-4766-b504-8e9a941a056f<br>
>[2020-03-12 07:18:16.719875] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/06-02/1557813<br>
>[2020-03-12 07:18:17.72679] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/06-02/1557205<br>
>[2020-03-12 07:18:17.297362] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/06-02/1556880<br>
>[2020-03-12 07:18:17.488224] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/06-02/1557769<br>
>[2020-03-12 07:18:17.730181] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/06-02/1557028<br>
>[2020-03-12 07:18:17.869410] I [gsyncd(config-get):318:main] <top>:<br>
>Using<br>
>session config file<br>
>path=/var/lib/glusterd/geo-replication/media-storage_slave-node_dr-media/gsyncd.conf<br>
>[2020-03-12 07:18:18.65431] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/06-02/1558442<br>
>[2020-03-12 07:18:18.352381] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/06-02/1557391<br>
>[2020-03-12 07:18:18.374876] I [gsyncd(config-get):318:main] <top>:<br>
>Using<br>
>session config file<br>
>path=/var/lib/glusterd/geo-replication/media-storage_slave-node_dr-media/gsyncd.conf<br>
>[2020-03-12 07:18:18.482299] I [gsyncd(config-set):318:main] <top>:<br>
>Using<br>
>session config file<br>
>path=/var/lib/glusterd/geo-replication/media-storage_slave-nodem_dr-media/gsyncd.conf<br>
>[2020-03-12 07:18:18.507585] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/06-02/1558577<br>
>[2020-03-12 07:18:18.576061] I [gsyncd(config-get):318:main] <top>:<br>
>Using<br>
>session config file<br>
>path=/var/lib/glusterd/geo-replication/media-storage_slave-node_dr-media/gsyncd.conf<br>
>[2020-03-12 07:18:18.582772] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/06-02/1556831<br>
>[2020-03-12 07:18:18.684170] I [gsyncd(config-get):318:main] <top>:<br>
>Using<br>
>session config file<br>
>path=/var/lib/glusterd/geo-replication/media-storage_slave-node_dr-media/gsyncd.conf<br>
>[2020-03-12 07:18:18.691845] E [syncdutils(worker<br>
>/srv/media-storage):312:log_raise_exception] <top>: connection to peer<br>
>is<br>
>broken<br>
>[2020-03-12 07:18:18.692106] E [syncdutils(worker<br>
>/srv/media-storage):312:log_raise_exception] <top>: connection to peer<br>
>is<br>
>broken<br>
>[2020-03-12 07:18:18.694910] E [syncdutils(worker<br>
>/srv/media-storage):822:errlog] Popen: command returned error cmd=ssh<br>
>-oPasswordAuthentication=no -oStrictHostKeyChecking=no -i<br>
>/var/lib/glusterd/geo-replication/secret.pem -p 22 -oControlMaster=auto<br>
>-S<br>
>/tmp/gsyncd-aux-ssh-WaMqpG/241afba5343394352fc3f9c251909232.sock<br>
>slave-node<br>
>/nonexistent/gsyncd slave media-storage slave-node::dr-media<br>
>--master-node<br>
>master-node --master-node-id 023cdb20-2737-4278-93c2-0927917ee314<br>
>--master-brick /srv/media-storage --local-node slave-node<br>
>--local-node-id<br>
>cf34fc96-a08a-49c2-b8eb-a3df5a05f757 --slave-timeout 120<br>
>--slave-log-level<br>
>DEBUG --slave-gluster-log-level INFO --slave-gluster-command-dir<br>
>/usr/sbin<br>
>--master-dist-count 1 error=255<br>
>[2020-03-12 07:18:18.701545] E [syncdutils(worker<br>
>/srv/media-storage):826:logerr] Popen: ssh> Killed by signal 15.<br>
>[2020-03-12 07:18:18.721456] I [repce(agent<br>
>/srv/media-storage):96:service_loop] RepceServer: terminating on<br>
>reaching<br>
>EOF.<br>
>[2020-03-12 07:18:18.778527] I<br>
>[gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker<br>
>Status<br>
>Change status=Faulty<br>
>[2020-03-12 07:18:19.791198] I [gsyncd(config-get):318:main] <top>:<br>
>Using<br>
>session config file<br>
>path=/var/lib/glusterd/geo-replication/media-storage_slave-node_dr-media/gsyncd.conf<br>
>[2020-03-12 07:18:19.931610] I [gsyncd(monitor):318:main] <top>: Using<br>
>session config file<br>
>path=/var/lib/glusterd/geo-replication/media-storage_slave-node_dr-media/gsyncd.conf<br>
>[2020-03-12 07:18:20.97094] D [monitor(monitor):375:distribute] <top>:<br>
>master bricks: [{'host': 'master-node', 'uuid':<br>
>'023cdb20-2737-4278-93c2-0927917ee314', 'dir': '/srv/media-storage'}]<br>
>[2020-03-12 07:18:20.97544] D [monitor(monitor):385:distribute] <top>:<br>
>slave SSH gateway: slave-node<br>
>[2020-03-12 07:18:20.597170] D [monitor(monitor):405:distribute] <top>:<br>
>slave bricks: [{'host': 'slave-node', 'uuid':<br>
>'cf34fc96-a08a-49c2-b8eb-a3df5a05f757', 'dir': '/data/dr-media'}]<br>
>[2020-03-12 07:18:20.598018] D [syncdutils(monitor):907:is_hot]<br>
>Volinfo:<br>
>brickpath: 'master-node:/srv/media-storage'<br>
>[2020-03-12 07:18:20.599000] D [monitor(monitor):419:distribute] <top>:<br>
>worker specs: [({'host': 'master-node', 'uuid':<br>
>'023cdb20-2737-4278-93c2-0927917ee314', 'dir': '/srv/media-storage'},<br>
>('root@slave-node', 'cf34fc96-a08a-49c2-b8eb-a3df5a05f757'), '1',<br>
>False)]<br>
>[2020-03-12 07:18:20.609170] I<br>
>[gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker<br>
>Status<br>
>Change status=Initializing...<br>
>[2020-03-12 07:18:20.609390] I [monitor(monitor):159:monitor] Monitor:<br>
>starting gsyncd worker brick=/srv/media-storage slave_node=slave-node<br>
>[2020-03-12 07:18:20.616580] D [monitor(monitor):230:monitor] Monitor:<br>
>Worker would mount volume privately<br>
>[2020-03-12 07:18:20.698721] I [gsyncd(agent<br>
>/srv/media-storage):318:main]<br>
><top>: Using session config file<br>
>path=/var/lib/glusterd/geo-replication/media-storage_slave-node_dr-media/gsyncd.conf<br>
>[2020-03-12 07:18:20.700041] D [subcmds(agent<br>
>/srv/media-storage):107:subcmd_agent] <top>: RPC FD rpc_fd='5,11,10,9'<br>
>[2020-03-12 07:18:20.700860] I [changelogagent(agent<br>
>/srv/media-storage):72:__init__] ChangelogAgent: Agent listining...<br>
>[2020-03-12 07:18:20.706031] I [gsyncd(worker<br>
>/srv/media-storage):318:main]<br>
><top>: Using session config file<br>
>path=/var/lib/glusterd/geo-replication/media-storage_slave-node_dr-media/gsyncd.conf<br>
>[2020-03-12 07:18:20.727953] I [resource(worker<br>
>/srv/media-storage):1386:connect_remote] SSH: Initializing SSH<br>
>connection<br>
>between master and slave...<br>
>[2020-03-12 07:18:20.763588] D [repce(worker<br>
>/srv/media-storage):195:push]<br>
>RepceClient: call 31602:140109226157888:1583997500.76<br>
>__repce_version__()<br>
>...<br>
>[2020-03-12 07:18:22.261354] D [repce(worker<br>
>/srv/media-storage):215:__call__] RepceClient: call<br>
>31602:140109226157888:1583997500.76 __repce_version__ -> 1.0<br>
>[2020-03-12 07:18:22.261629] D [repce(worker<br>
>/srv/media-storage):195:push]<br>
>RepceClient: call 31602:140109226157888:1583997502.26 version() ...<br>
>[2020-03-12 07:18:22.269346] D [repce(worker<br>
>/srv/media-storage):215:__call__] RepceClient: call<br>
>31602:140109226157888:1583997502.26 version -> 1.0<br>
>[2020-03-12 07:18:22.269610] D [repce(worker<br>
>/srv/media-storage):195:push]<br>
>RepceClient: call 31602:140109226157888:1583997502.27 pid() ...<br>
>[2020-03-12 07:18:22.278055] D [repce(worker<br>
>/srv/media-storage):215:__call__] RepceClient: call<br>
>31602:140109226157888:1583997502.27 pid -> 50215<br>
>[2020-03-12 07:18:22.278273] I [resource(worker<br>
>/srv/media-storage):1435:connect_remote] SSH: SSH connection between<br>
>master<br>
>and slave established. duration=1.5501<br>
>[2020-03-12 07:18:22.278470] I [resource(worker<br>
>/srv/media-storage):1105:connect] GLUSTER: Mounting gluster volume<br>
>locally...<br>
>[2020-03-12 07:18:22.345194] D [resource(worker<br>
>/srv/media-storage):879:inhibit] DirectMounter: auxiliary glusterfs<br>
>mount<br>
>in place<br>
>[2020-03-12 07:18:23.352816] D [resource(worker<br>
>/srv/media-storage):953:inhibit] DirectMounter: auxiliary glusterfs<br>
>mount<br>
>prepared<br>
>[2020-03-12 07:18:23.353075] I [resource(worker<br>
>/srv/media-storage):1128:connect] GLUSTER: Mounted gluster volume<br>
>duration=1.0744<br>
>[2020-03-12 07:18:23.353261] I [subcmds(worker<br>
>/srv/media-storage):84:subcmd_worker] <top>: Worker spawn successful.<br>
>Acknowledging back to monitor<br>
>[2020-03-12 07:18:23.353602] D [master(worker<br>
>/srv/media-storage):104:gmaster_builder] <top>: setting up change<br>
>detection<br>
>mode mode=xsync<br>
>[2020-03-12 07:18:23.353697] D [monitor(monitor):273:monitor] Monitor:<br>
>worker(/srv/media-storage) connected<br>
>[2020-03-12 07:18:23.354954] D [master(worker<br>
>/srv/media-storage):104:gmaster_builder] <top>: setting up change<br>
>detection<br>
>mode mode=changelog<br>
>[2020-03-12 07:18:23.356203] D [master(worker<br>
>/srv/media-storage):104:gmaster_builder] <top>: setting up change<br>
>detection<br>
>mode mode=changeloghistory<br>
>[2020-03-12 07:18:23.359487] D [repce(worker<br>
>/srv/media-storage):195:push]<br>
>RepceClient: call 31602:140109226157888:1583997503.36 version() ...<br>
>[2020-03-12 07:18:23.360182] D [repce(worker<br>
>/srv/media-storage):215:__call__] RepceClient: call<br>
>31602:140109226157888:1583997503.36 version -> 1.0<br>
>[2020-03-12 07:18:23.360398] D [master(worker<br>
>/srv/media-storage):777:setup_working_dir] _GMaster: changelog working<br>
>dir<br>
>/var/lib/misc/gluster/gsyncd/media-storage_slave-node_dr-media/srv-media-storage<br>
>[2020-03-12 07:18:23.360609] D [repce(worker<br>
>/srv/media-storage):195:push]<br>
>RepceClient: call 31602:140109226157888:1583997503.36 init() ...<br>
>[2020-03-12 07:18:23.380367] D [repce(worker<br>
>/srv/media-storage):215:__call__] RepceClient: call<br>
>31602:140109226157888:1583997503.36 init -> None<br>
>[2020-03-12 07:18:23.380626] D [repce(worker<br>
>/srv/media-storage):195:push]<br>
>RepceClient: call 31602:140109226157888:1583997503.38<br>
>register('/srv/media-storage',<br>
>'/var/lib/misc/gluster/gsyncd/media-storage_slave-node.com_dr-media/srv-media-storage',<br>
>'/var/log/glusterfs/geo-replication/media-storage_slave-node_dr-media/changes-srv-media-storage.log',<br>
>7, 5) ...<br>
>[2020-03-12 07:18:25.384763] D [repce(worker<br>
>/srv/media-storage):215:__call__] RepceClient: call<br>
>31602:140109226157888:1583997503.38 register -> None<br>
>[2020-03-12 07:18:25.384988] D [master(worker<br>
>/srv/media-storage):777:setup_working_dir] _GMaster: changelog working<br>
>dir<br>
>/var/lib/misc/gluster/gsyncd/media-storage_slave-node_dr-media/srv-media-storage<br>
>[2020-03-12 07:18:25.385189] D [master(worker<br>
>/srv/media-storage):777:setup_working_dir] _GMaster: changelog working<br>
>dir<br>
>/var/lib/misc/gluster/gsyncd/media-storage_slave-node_dr-media/srv-media-storage<br>
>[2020-03-12 07:18:25.385427] D [master(worker<br>
>/srv/media-storage):777:setup_working_dir] _GMaster: changelog working<br>
>dir<br>
>/var/lib/misc/gluster/gsyncd/media-storage_slave-node_dr-media/srv-media-storage<br>
>[2020-03-12 07:18:25.385597] I [master(worker<br>
>/srv/media-storage):1640:register] _GMaster: Working dir<br>
>path=/var/lib/misc/gluster/gsyncd/media-storage_slave-node_dr-media/srv-media-storage<br>
>[2020-03-12 07:18:25.395163] I [resource(worker<br>
>/srv/media-storage):1291:service_loop] GLUSTER: Register time<br>
>time=1583997505<br>
>[2020-03-12 07:18:25.395838] D [repce(worker<br>
>/srv/media-storage):195:push]<br>
>RepceClient: call 31602:140108435007232:1583997505.4 keep_alive(None,)<br>
>...<br>
>[2020-03-12 07:18:25.399630] D [master(worker<br>
>/srv/media-storage):539:crawlwrap] _GMaster: primary master with volume<br>
>id<br>
>458c5247-1fb1-4e4a-88ec-6e3933f1cf3b ...<br>
>[2020-03-12 07:18:25.403887] I [gsyncdstatus(worker<br>
>/srv/media-storage):281:set_active] GeorepStatus: Worker Status Change<br>
>status=Active<br>
>[2020-03-12 07:18:25.404815] D [repce(worker<br>
>/srv/media-storage):215:__call__] RepceClient: call<br>
>31602:140108435007232:1583997505.4 keep_alive -> 1<br>
>[2020-03-12 07:18:25.405322] I [gsyncdstatus(worker<br>
>/srv/media-storage):253:set_worker_crawl_status] GeorepStatus: Crawl<br>
>Status<br>
>Change status=History Crawl<br>
>[2020-03-12 07:18:25.405780] I [master(worker<br>
>/srv/media-storage):1554:crawl] _GMaster: starting history crawl<br>
>turns=1<br>
>stime=None entry_stime=None etime=1583997505<br>
>[2020-03-12 07:18:25.405967] I [resource(worker<br>
>/srv/media-storage):1307:service_loop] GLUSTER: No stime available,<br>
>using<br>
>xsync crawl<br>
>[2020-03-12 07:18:25.406529] D [repce(worker<br>
>/srv/media-storage):195:push]<br>
>RepceClient: call 31602:140108426614528:1583997505.41 keep_alive(None,)<br>
>...<br>
>[2020-03-12 07:18:25.408334] D [master(worker<br>
>/srv/media-storage):539:crawlwrap] _GMaster: primary master with volume<br>
>id<br>
>458c5247-1fb1-4e4a-88ec-6e3933f1cf3b ...<br>
>[2020-03-12 07:18:25.411999] I [master(worker<br>
>/srv/media-storage):1670:crawl] _GMaster: starting hybrid crawl<br>
>stime=None<br>
>[2020-03-12 07:18:25.412503] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering .<br>
>[2020-03-12 07:18:25.414274] I [gsyncdstatus(worker<br>
>/srv/media-storage):253:set_worker_crawl_status] GeorepStatus: Crawl<br>
>Status<br>
>Change status=Hybrid Crawl<br>
>[2020-03-12 07:18:25.415570] D [repce(worker<br>
>/srv/media-storage):215:__call__] RepceClient: call<br>
>31602:140108426614528:1583997505.41 keep_alive -> 2<br>
>[2020-03-12 07:18:25.418721] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering ./api<br>
>[2020-03-12 07:18:25.419502] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering ./api/media<br>
>[2020-03-12 07:18:25.420666] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering ./api/media/listing<br>
>[2020-03-12 07:18:25.592319] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018<br>
>[2020-03-12 07:18:25.603673] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20<br>
>[2020-03-12 07:18:25.623689] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528270<br>
>[2020-03-12 07:18:25.650356] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528967<br>
>[2020-03-12 07:18:25.689970] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528854<br>
>[2020-03-12 07:18:25.728863] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528754<br>
>[2020-03-12 07:18:25.770190] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528462<br>
>[2020-03-12 07:18:25.797228] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528991<br>
>[2020-03-12 07:18:25.809954] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529014<br>
>[2020-03-12 07:18:25.822673] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528358<br>
>[2020-03-12 07:18:25.845856] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529072<br>
>[2020-03-12 07:18:25.856015] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528678<br>
>[2020-03-12 07:18:25.858510] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528857<br>
>[2020-03-12 07:18:25.869515] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528699<br>
>[2020-03-12 07:18:25.893805] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528348<br>
>[2020-03-12 07:18:25.916162] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528582<br>
>[2020-03-12 07:18:25.943183] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528548<br>
>[2020-03-12 07:18:25.956763] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528241<br>
>[2020-03-12 07:18:25.978117] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528864<br>
>[2020-03-12 07:18:25.990802] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528730<br>
>[2020-03-12 07:18:26.1740] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528367<br>
>[2020-03-12 07:18:26.24030] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529063<br>
>[2020-03-12 07:18:26.46653] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529058<br>
>[2020-03-12 07:18:26.76448] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528954<br>
>[2020-03-12 07:18:26.96426] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528466<br>
>[2020-03-12 07:18:26.112611] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528960<br>
>[2020-03-12 07:18:26.134823] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528620<br>
>[2020-03-12 07:18:26.157533] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528943<br>
>[2020-03-12 07:18:26.190406] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528274<br>
>[2020-03-12 07:18:26.222803] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528249<br>
>[2020-03-12 07:18:26.232880] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528711<br>
>[2020-03-12 07:18:26.253369] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528413<br>
>[2020-03-12 07:18:26.259948] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528853<br>
>[2020-03-12 07:18:26.278692] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528803<br>
>[2020-03-12 07:18:26.295695] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528831<br>
>[2020-03-12 07:18:26.322766] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528510<br>
>[2020-03-12 07:18:26.344959] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529041<br>
>[2020-03-12 07:18:26.356956] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528637<br>
>[2020-03-12 07:18:26.364748] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528519<br>
>[2020-03-12 07:18:26.369580] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528740<br>
>[2020-03-12 07:18:26.382257] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528990<br>
>[2020-03-12 07:18:26.393202] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528896<br>
>[2020-03-12 07:18:26.414587] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528808<br>
>[2020-03-12 07:18:26.424684] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529026<br>
>[2020-03-12 07:18:26.452891] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528508<br>
>[2020-03-12 07:18:26.471034] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529049<br>
>[2020-03-12 07:18:26.473328] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529056<br>
>[2020-03-12 07:18:26.490430] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528928<br>
>[2020-03-12 07:18:26.500461] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528516<br>
>[2020-03-12 07:18:26.515748] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528658<br>
>[2020-03-12 07:18:26.543150] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528800<br>
>[2020-03-12 07:18:26.564454] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528671<br>
>[2020-03-12 07:18:26.591042] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528529<br>
>[2020-03-12 07:18:26.611500] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528502<br>
>[2020-03-12 07:18:26.617207] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529006<br>
>[2020-03-12 07:18:26.629002] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528899<br>
>[2020-03-12 07:18:26.644531] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528482<br>
>[2020-03-12 07:18:26.666290] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528623<br>
>[2020-03-12 07:18:26.676486] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528424<br>
>[2020-03-12 07:18:26.687612] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528825<br>
>[2020-03-12 07:18:26.695039] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528756<br>
>[2020-03-12 07:18:26.706976] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529062<br>
>[2020-03-12 07:18:26.721376] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1529033<br>
>[2020-03-12 07:18:26.742405] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528988<br>
>[2020-03-12 07:18:26.743782] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528439<br>
>[2020-03-12 07:18:26.749468] D [master(worker<br>
>/srv/media-storage):1792:Xcrawl] _GMaster: entering<br>
>./api/media/listing/2018/05-20/1528347<br>
>[2020-03-12 07:18:27.416842] I [master(worker<br>
>/srv/media-storage):1681:crawl] _GMaster: processing xsync changelog<br>
>path=/var/lib/misc/gluster/gsyncd/media-storage_slave-node_dr-media/srv-media-storage/xsync/XSYNC-CHANGELOG.1583997505<br>
>[2020-03-12 07:18:27.417128] D [master(worker<br>
>/srv/media-storage):1326:process] _GMaster: processing change<br>
>changelog=/var/lib/misc/gluster/gsyncd/media-storage_slave-node_dr-media/srv-media-storage/xsync/XSYNC-CHANGELOG.1583997505<br>
>[2020-03-12 07:18:27.660609] D [master(worker<br>
>/srv/media-storage):1205:process_change] _GMaster: entries: [{'uid': 0,<br>
>'skip_entry': False, 'gfid': '55c74c7f-3609-44c4-b0d1-0e7032413169',<br>
>'gid':<br>
>0, 'mode': 16895, 'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/konga', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '77dc4f2c-3be2-4cc7-88ed-49fb4be12f73',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/zingat-comment', 'op':<br>
>'MKDIR'}, {'uid': 0, 'skip_entry': False, 'gfid':<br>
>'d1e67bf6-1495-42b0-b299-46f1e8fa75cd', 'gid': 0, 'mode': 16877,<br>
>'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/zfotobot', 'op': 'MKDIR'},<br>
>{'uid': 0, 'skip_entry': False, 'gfid':<br>
>'4d4e189a-9c8f-4137-ab49-23801b532788', 'gid': 0, 'mode': 16895,<br>
>'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/elastics', 'op': 'MKDIR'},<br>
>{'uid': 0, 'skip_entry': False, 'gfid':<br>
>'438c462a-74a3-48b4-95f8-7fdd18497c63', 'gid': 0, 'mode': 16895,<br>
>'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/zingrawler', 'op':<br>
>'MKDIR'},<br>
>{'uid': 0, 'skip_entry': False, 'gfid':<br>
>'7543966d-e153-424f-a311-4cab0adbd765', 'gid': 0, 'mode': 16895,<br>
>'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/notification', 'op':<br>
>'MKDIR'},<br>
>{'uid': 0, 'skip_entry': False, 'gfid':<br>
>'894b04c4-f006-4fdc-a089-4961e483077f', 'gid': 0, 'mode': 16877,<br>
>'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/zschools', 'op': 'MKDIR'},<br>
>{'uid': 0, 'skip_entry': False, 'gfid':<br>
>'74464ff5-d285-45f6-8417-638c313244b6', 'gid': 0, 'mode': 16895,<br>
>'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/zingat-akademi', 'op':<br>
>'MKDIR'}, {'uid': 0, 'skip_entry': False, 'gfid':<br>
>'63c30efd-0222-432f-a3b8-09dd5bd5173c', 'gid': 0, 'mode': 16877,<br>
>'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/webc', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': 'c1d865d8-48f1-4d69-a02a-b06cae6e7776',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/comment-ratings', 'op':<br>
>'MKDIR'}, {'uid': 0, 'skip_entry': False, 'gfid':<br>
>'09569e91-6223-4320-9a12-af7ad9f6e433', 'gid': 0, 'mode': 16895,<br>
>'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/airflow', 'op': 'MKDIR'},<br>
>{'uid': 0, 'skip_entry': False, 'gfid':<br>
>'8aeb9a6f-b223-4835-bccf-a11393b15bbc', 'gid': 0, 'mode': 16877,<br>
>'entry':<br>
>'.gfid/00000000-0000-0000-0000-000000000001/api', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '001e77e9-de31-4d73-8ac6-f03c1ae556b2',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/8aeb9a6f-b223-4835-bccf-a11393b15bbc/media', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '6bdf6593-8d18-42c0-a52c-dc344a2ded46',<br>
>'gid': 0, 'mode': 33188, 'entry':<br>
>'.gfid/001e77e9-de31-4d73-8ac6-f03c1ae556b2/testforgluster', 'op':<br>
>'MKNOD'}, {'stat': {'atime': 1583997329.2596, 'gid': 997, 'mtime':<br>
>1507210680.0, 'uid': 998, 'mode': 41471}, 'skip_entry': False, 'gfid':<br>
>'3282ed41-377e-472b-b4f4-96a62fafe709', 'link': '/srv/autofs/media',<br>
>'entry': '.gfid/001e77e9-de31-4d73-8ac6-f03c1ae556b2/media', 'op':<br>
>'SYMLINK'}, {'uid': 0, 'skip_entry': False, 'gfid':<br>
>'a29c9172-3b72-4175-9f1c-d1458434a7f0', 'gid': 0, 'mode': 16877,<br>
>'entry':<br>
>'.gfid/001e77e9-de31-4d73-8ac6-f03c1ae556b2/listing', 'op': 'MKDIR'},<br>
>{'uid': 0, 'skip_entry': False, 'gfid':<br>
>'5c392153-cc7c-4479-a801-9afdad0ecbcf', 'gid': 0, 'mode': 16877,<br>
>'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/95762', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '52dc2abe-4848-47b4-903e-b43cb5f2cec1',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/50117', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': 'f6a211b6-fdff-454e-956c-d4ba7e324164',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/4993', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '5bfafa60-522c-4e0f-93df-07343c2fb943',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/54531', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '80e50209-75b8-4e95-bb45-ff3ab5f879bb',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/12715', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '15b61eac-7b00-4af6-9322-1c75589cd692',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/21321', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': 'acfbb711-376f-4c66-bd5a-de0991073d27',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/87093', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': 'f54b0d9f-a0aa-479e-91ca-8c1b2d651ae6',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/51424', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '9b4a16b8-1f60-47b2-90ff-6c9ae44269a8',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/20729', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '6d4ccfdf-5815-4b28-9cba-cbb55ee610e0',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/12066', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': 'b682e238-6ce1-4699-9159-162c835cc25c',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/93511', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': 'd7d4f2fb-7eb7-4a07-9beb-d06cd080ea3c',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/16643', 'op': 'MKDIR'},<br>
>{'uid':<br>
>0, 'skip_entry': False, 'gfid': '0cc499c4-be50-49f1-a7e4-7dfdd331fd48',<br>
>'gid': 0, 'mode': 16877, 'entry':<br>
>'.gfid/a29c9172-3b72-4175-9f1c-d1458434a7f0/54979', 'op': 'MKDIR'},<br>
><br>
>Etem Bayoğlu <<a href="mailto:etembayoglu@gmail.com" target="_blank">etembayoglu@gmail.com</a>>, 12 Mar 2020 Per, 10:13 tarihinde<br>
>şunu<br>
>yazdı:<br>
><br>
>> Hi,<br>
>><br>
>> here my slave node logs at the time sync stopped:<br>
>><br>
>> [2020-03-08 03:33:01.489559] I<br>
>[glusterfsd-mgmt.c:2282:mgmt_getspec_cbk]<br>
>> 0-glusterfs: No change in volfile,continuing<br>
>> [2020-03-08 03:33:01.489298] I [MSGID: 100011]<br>
>> [glusterfsd.c:1679:reincarnate] 0-glusterfsd: Fetching the volume<br>
>file from<br>
>> server...<br>
>> [2020-03-08 09:49:37.991177] I [fuse-bridge.c:6083:fuse_thread_proc]<br>
>> 0-fuse: initiating unmount of /tmp/gsyncd-aux-mount-l3PR6o<br>
>> [2020-03-08 09:49:37.993978] W [glusterfsd.c:1596:cleanup_and_exit]<br>
>> (-->/lib64/libpthread.so.0(+0x7e65) [0x7f2f9f70ce65]<br>
>> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55cc67c20625]<br>
>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55cc67c2048b] ) 0-:<br>
>> received signum (15), shutting down<br>
>> [2020-03-08 09:49:37.994012] I [fuse-bridge.c:6871:fini] 0-fuse:<br>
>> Unmounting '/tmp/gsyncd-aux-mount-l3PR6o'.<br>
>> [2020-03-08 09:49:37.994022] I [fuse-bridge.c:6876:fini] 0-fuse:<br>
>Closing<br>
>> fuse connection to '/tmp/gsyncd-aux-mount-l3PR6o'.<br>
>> [2020-03-08 09:49:50.302806] I [MSGID: 100030]<br>
>[glusterfsd.c:2867:main]<br>
>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version<br>
>7.3<br>
>> (args: /usr/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO<br>
>><br>
>--log-file=/var/log/glusterfs/geo-replication-slaves/media-storage_slave-node_dr-media/mnt-master-node-srv-media-storage.log<br>
>> --volfile-server=localhost --volfile-id=dr-media --client-pid=-1<br>
>> /tmp/gsyncd-aux-mount-1AQBe4)<br>
>> [2020-03-08 09:49:50.311167] I [glusterfsd.c:2594:daemonize]<br>
>0-glusterfs:<br>
>> Pid of current running process is 55522<br>
>> [2020-03-08 09:49:50.352351] I [MSGID: 101190]<br>
>> [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started<br>
>thread<br>
>> with index 0<br>
>> [2020-03-08 09:49:50.352416] I [MSGID: 101190]<br>
>> [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started<br>
>thread<br>
>> with index 1<br>
>> [2020-03-08 09:49:50.373248] I [MSGID: 114020] [client.c:2436:notify]<br>
>> 0-dr-media-client-0: parent translators are ready, attempting connect<br>
>on<br>
>> transport<br>
>> Final graph:<br>
>><br>
>><br>
>+------------------------------------------------------------------------------+<br>
>> 1: volume dr-media-client-0<br>
>> 2: type protocol/client<br>
>> 3: option ping-timeout 42<br>
>> 4: option remote-host slave-node<br>
>> 5: option remote-subvolume /data/dr-media<br>
>> 6: option transport-type socket<br>
>> 7: option transport.address-family inet<br>
>> 8: option username 4aafadfa-6ccb-4c2f-920c-1f37ed9eef34<br>
>> 9: option password a8c0f88b-2621-4038-8f65-98068ea71bb0<br>
>> 10: option transport.socket.ssl-enabled off<br>
>> 11: option transport.tcp-user-timeout 0<br>
>> 12: option transport.socket.keepalive-time 20<br>
>> 13: option transport.socket.keepalive-interval 2<br>
>> 14: option transport.socket.keepalive-count 9<br>
>> 15: option send-gids true<br>
>> 16: end-volume<br>
>> 17:<br>
>> 18: volume dr-media-dht<br>
>> 19: type cluster/distribute<br>
>> 20: option lock-migration off<br>
>> 21: option force-migration off<br>
>> 22: subvolumes dr-media-client-0<br>
>> 23: end-volume<br>
>> 24:<br>
>> 25: volume dr-media-write-behind<br>
>> 26: type performance/write-behind<br>
>> 27: option cache-size 8MB<br>
>> 28: option aggregate-size 1MB<br>
>> 29: subvolumes dr-media-dht<br>
>> 30: end-volume<br>
>> 31:<br>
>> 32: volume dr-media-read-ahead<br>
>> 33: type performance/read-ahead<br>
>> 34: subvolumes dr-media-write-behind<br>
>> 35: end-volume<br>
>> 36:<br>
>> 37: volume dr-media-readdir-ahead<br>
>> 38: type performance/readdir-ahead<br>
>> 39: option parallel-readdir off<br>
>> 40: option rda-request-size 131072<br>
>> 41: option rda-cache-limit 10MB<br>
>> 42: subvolumes dr-media-read-ahead<br>
>> 43: end-volume<br>
>> 44:<br>
>> 45: volume dr-media-io-cache<br>
>> 46: type performance/io-cache<br>
>> 47: option cache-size 256MB<br>
>> 48: subvolumes dr-media-readdir-ahead<br>
>> 49: end-volume<br>
>> 50:<br>
>> 51: volume dr-media-open-behind<br>
>> 52: type performance/open-behind<br>
>> 53: subvolumes dr-media-io-cache<br>
>> 54: end-volume<br>
>> 55:<br>
>> 56: volume dr-media-quick-read<br>
>> 57: type performance/quick-read<br>
>> 58: option cache-size 256MB<br>
>> 59: subvolumes dr-media-open-behind<br>
>> 60: end-volume<br>
>> 61:<br>
>> 62: volume dr-media-md-cache<br>
>> 63: type performance/md-cache<br>
>> 64: option cache-posix-acl true<br>
>> 65: subvolumes dr-media-quick-read<br>
>> 66: end-volume<br>
>> 67:<br>
>> 68: volume dr-media-io-threads<br>
>> 69: type performance/io-threads<br>
>> 70: subvolumes dr-media-md-cache<br>
>> 71: end-volume<br>
>> 72:<br>
>> 73: volume dr-media<br>
>> 74: type debug/io-stats<br>
>> 75: option log-level INFO<br>
>> 76: option threads 16<br>
>> 77: option latency-measurement off<br>
>> 78: option count-fop-hits off<br>
>> 79: option global-threading off<br>
>> 80: subvolumes dr-media-io-threads<br>
>> 81: end-volume<br>
>> 82:<br>
>> 83: volume posix-acl-autoload<br>
>> 84: type system/posix-acl<br>
>> 85: subvolumes dr-media<br>
>> 86: end-volume<br>
>> 87:<br>
>> 88: volume gfid-access-autoload<br>
>> 89: type features/gfid-access<br>
>> 90: subvolumes posix-acl-autoload<br>
>> 91: end-volume<br>
>> 92:<br>
>> 93: volume meta-autoload<br>
>> 94: type meta<br>
>> 95: subvolumes gfid-access-autoload<br>
>> 96: end-volume<br>
>> 97:<br>
>><br>
>><br>
>+------------------------------------------------------------------------------+<br>
>> [2020-03-08 09:49:50.388102] I [rpc-clnt.c:1963:rpc_clnt_reconfig]<br>
>> 0-dr-media-client-0: changing port to 49152 (from 0)<br>
>> [2020-03-08 09:49:50.388132] I [socket.c:865:__socket_shutdown]<br>
>> 0-dr-media-client-0: intentional socket shutdown(12)<br>
>> [2020-03-08 09:49:50.401512] I [MSGID: 114057]<br>
>> [client-handshake.c:1375:select_server_supported_programs]<br>
>> 0-dr-media-client-0: Using Program GlusterFS 4.x v1, Num (1298437),<br>
>Version<br>
>> (400)<br>
>> [2020-03-08 09:49:50.401765] W [dict.c:999:str_to_data]<br>
>> (-->/usr/lib64/glusterfs/7.3/xlator/protocol/client.so(+0x381d4)<br>
>> [0x7f7f63daa1d4] -->/lib64/libglusterfs.so.0(dict_set_str+0x16)<br>
>> [0x7f7f76b3a2f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71)<br>
>> [0x7f7f76b36c11] ) 0-dict: value is NULL [Invalid argument]<br>
>> [2020-03-08 09:49:50.401783] I [MSGID: 114006]<br>
>> [client-handshake.c:1236:client_setvolume] 0-dr-media-client-0:<br>
>failed to<br>
>> set process-name in handshake msg<br>
>> [2020-03-08 09:49:50.404115] I [MSGID: 114046]<br>
>> [client-handshake.c:1105:client_setvolume_cbk] 0-dr-media-client-0:<br>
>> Connected to dr-media-client-0, attached to remote volume<br>
>'/data/dr-media'.<br>
>> [2020-03-08 09:49:50.405761] I [fuse-bridge.c:5166:fuse_init]<br>
>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24<br>
>kernel<br>
>> 7.22<br>
>> [2020-03-08 09:49:50.405780] I [fuse-bridge.c:5777:fuse_graph_sync]<br>
>> 0-fuse: switched to graph 0<br>
>> [2020-03-08 11:49:00.933168] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f7f76b438ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f7f6def1221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f7f6def2998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e65)[0x7f7f75984e65] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7f7f7524a88d] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 11:53:29.822876] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f7f76b438ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f7f6def1221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f7f6def2998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e65)[0x7f7f75984e65] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7f7f7524a88d] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 12:00:46.656170] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f7f76b438ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f7f6def1221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f7f6def2998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e65)[0x7f7f75984e65] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7f7f7524a88d] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>><br>
>><br>
>> master node logs here:<br>
>><br>
>> [2020-03-08 09:49:38.115108] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fb5ab2aa8ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fb5a265c221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8b3a)[0x7fb5a265cb3a]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fb5aa0ebe25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fb5a99b4bad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 09:49:38.932935] I [fuse-bridge.c:6083:fuse_thread_proc]<br>
>> 0-fuse: initiating unmount of /tmp/gsyncd-aux-mount-Xy8taN<br>
>> [2020-03-08 09:49:38.947634] W [glusterfsd.c:1596:cleanup_and_exit]<br>
>> (-->/lib64/libpthread.so.0(+0x7e25) [0x7fb5aa0ebe25]<br>
>> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x557ccdf55625]<br>
>> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x557ccdf5548b] ) 0-:<br>
>> received signum (15), shutting down<br>
>> [2020-03-08 09:49:38.947684] I [fuse-bridge.c:6871:fini] 0-fuse:<br>
>> Unmounting '/tmp/gsyncd-aux-mount-Xy8taN'.<br>
>> [2020-03-08 09:49:38.947704] I [fuse-bridge.c:6876:fini] 0-fuse:<br>
>Closing<br>
>> fuse connection to '/tmp/gsyncd-aux-mount-Xy8taN'.<br>
>> [2020-03-08 09:49:51.545529] I [MSGID: 100030]<br>
>[glusterfsd.c:2867:main]<br>
>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version<br>
>7.3<br>
>> (args: /usr/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO<br>
>><br>
>--log-file=/var/log/glusterfs/geo-replication/media-storage_slave-node_dr-media/mnt-srv-media-storage.log<br>
>> --volfile-server=localhost --volfile-id=media-storage --client-pid=-1<br>
>> /tmp/gsyncd-aux-mount-XT9WfC)<br>
>> [2020-03-08 09:49:51.559518] I [glusterfsd.c:2594:daemonize]<br>
>0-glusterfs:<br>
>> Pid of current running process is 18484<br>
>> [2020-03-08 09:49:51.645473] I [MSGID: 101190]<br>
>> [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started<br>
>thread<br>
>> with index 0<br>
>> [2020-03-08 09:49:51.645624] I [MSGID: 101190]<br>
>> [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started<br>
>thread<br>
>> with index 1<br>
>> [2020-03-08 09:49:51.701470] I [MSGID: 114020] [client.c:2436:notify]<br>
>> 0-media-storage-client-0: parent translators are ready, attempting<br>
>connect<br>
>> on transport<br>
>> [2020-03-08 09:49:51.702908] I [rpc-clnt.c:1963:rpc_clnt_reconfig]<br>
>> 0-media-storage-client-0: changing port to 49152 (from 0)<br>
>> [2020-03-08 09:49:51.702961] I [socket.c:865:__socket_shutdown]<br>
>> 0-media-storage-client-0: intentional socket shutdown(12)<br>
>> [2020-03-08 09:49:51.703807] I [MSGID: 114057]<br>
>> [client-handshake.c:1375:select_server_supported_programs]<br>
>> 0-media-storage-client-0: Using Program GlusterFS 4.x v1, Num<br>
>(1298437),<br>
>> Version (400)<br>
>> [2020-03-08 09:49:51.704147] W [dict.c:999:str_to_data]<br>
>> (-->/usr/lib64/glusterfs/7.3/xlator/protocol/client.so(+0x381d4)<br>
>> [0x7fc5c01031d4] -->/lib64/libglusterfs.so.0(dict_set_str+0x16)<br>
>> [0x7fc5ce8d82f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71)<br>
>> [0x7fc5ce8d4c11] ) 0-dict: value is NULL [Invalid argument]<br>
>> [2020-03-08 09:49:51.704178] I [MSGID: 114006]<br>
>> [client-handshake.c:1236:client_setvolume] 0-media-storage-client-0:<br>
>failed<br>
>> to set process-name in handshake msg<br>
>> [2020-03-08 09:49:51.705982] I [MSGID: 114046]<br>
>> [client-handshake.c:1105:client_setvolume_cbk]<br>
>0-media-storage-client-0:<br>
>> Connected to media-storage-client-0, attached to remote volume<br>
>> '/srv/media-storage'.<br>
>> [2020-03-08 09:49:51.707627] I [fuse-bridge.c:5166:fuse_init]<br>
>> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24<br>
>kernel<br>
>> 7.22<br>
>> [2020-03-08 09:49:51.707658] I [fuse-bridge.c:5777:fuse_graph_sync]<br>
>> 0-fuse: switched to graph 0<br>
>> [2020-03-08 09:56:26.875082] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>> [2020-03-08 10:12:35.190809] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>> [2020-03-08 10:25:06.240795] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>> [2020-03-08 10:40:33.946794] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>> [2020-03-08 10:43:50.459247] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 10:55:27.034947] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>> [2020-03-08 11:05:53.483207] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>> [2020-03-08 11:26:01.492270] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>> [2020-03-08 11:32:56.618737] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 11:37:50.475099] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 11:40:54.362173] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>> [2020-03-08 11:42:35.859423] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 11:44:24.906383] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 11:47:45.474723] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 11:50:58.127202] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 11:52:55.616968] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 11:56:24.039211] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>> [2020-03-08 11:57:56.031648] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 12:06:19.686974] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>> (--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221]<br>
>(--><br>
>><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998]<br>
>(--><br>
>> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--><br>
>> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse:<br>
>> writing to fuse device failed: No such file or directory<br>
>> [2020-03-08 12:12:41.888889] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>> 0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>><br>
>><br>
>><br>
>> I see just no such file or directory error logs for 5 days. I<br>
>shifted log<br>
>> level to DEBUG I will send them also.<br>
>><br>
>> Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>>, 12 Mar 2020 Per, 07:55<br>
>tarihinde<br>
>> şunu yazdı:<br>
>><br>
>>> On March 11, 2020 10:17:05 PM GMT+02:00, "Etem Bayoğlu" <<br>
>>> <a href="mailto:etembayoglu@gmail.com" target="_blank">etembayoglu@gmail.com</a>> wrote:<br>
>>> >Hi Strahil,<br>
>>> ><br>
>>> >Thank you for your response. when I tail logs on both master and<br>
>slave<br>
>>> >I<br>
>>> >get this:<br>
>>> ><br>
>>> >on slave, from<br>
>>> >/var/log/glusterfs/geo-replication-slaves/<geo-session>/mnt-XXX.log<br>
>>> >file:<br>
>>> ><br>
>>> >[2020-03-11 19:53:32.721509] E<br>
>>> >[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>>> >(--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f78e10488ea]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f78d83f6221]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f78d83f7998]<br>
>>> >(--><br>
>>> >/lib64/libpthread.so.0(+0x7e65)[0x7f78dfe89e65] (--><br>
>>> >/lib64/libc.so.6(clone+0x6d)[0x7f78df74f88d] )))))<br>
>0-glusterfs-fuse:<br>
>>> >writing to fuse device failed: No such file or directory<br>
>>> >[2020-03-11 19:53:32.723758] E<br>
>>> >[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>>> >(--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f78e10488ea]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f78d83f6221]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f78d83f7998]<br>
>>> >(--><br>
>>> >/lib64/libpthread.so.0(+0x7e65)[0x7f78dfe89e65] (--><br>
>>> >/lib64/libc.so.6(clone+0x6d)[0x7f78df74f88d] )))))<br>
>0-glusterfs-fuse:<br>
>>> >writing to fuse device failed: No such file or directory<br>
>>> ><br>
>>> >on master,<br>
>>> >from /var/log/glusterfs/geo-replication/<geo-session>/mnt-XXX.log<br>
>file:<br>
>>> ><br>
>>> >[2020-03-11 19:40:55.872002] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>>> >0-glusterfs-fuse: extended attribute not supported by the backend<br>
>>> >storage<br>
>>> >[2020-03-11 19:40:58.389748] E<br>
>>> >[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>>> >(--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f1f4b9108ea]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f1f42cc2221]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f1f42cc3998]<br>
>>> >(--><br>
>>> >/lib64/libpthread.so.0(+0x7e25)[0x7f1f4a751e25] (--><br>
>>> >/lib64/libc.so.6(clone+0x6d)[0x7f1f4a01abad] )))))<br>
>0-glusterfs-fuse:<br>
>>> >writing to fuse device failed: No such file or directory<br>
>>> >[2020-03-11 19:41:08.214591] E<br>
>>> >[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>>> >(--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f1f4b9108ea]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f1f42cc2221]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f1f42cc3998]<br>
>>> >(--><br>
>>> >/lib64/libpthread.so.0(+0x7e25)[0x7f1f4a751e25] (--><br>
>>> >/lib64/libc.so.6(clone+0x6d)[0x7f1f4a01abad] )))))<br>
>0-glusterfs-fuse:<br>
>>> >writing to fuse device failed: No such file or directory<br>
>>> >[2020-03-11 19:53:59.275469] E<br>
>>> >[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>>> >(--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f1f4b9108ea]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f1f42cc2221]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f1f42cc3998]<br>
>>> >(--><br>
>>> >/lib64/libpthread.so.0(+0x7e25)[0x7f1f4a751e25] (--><br>
>>> >/lib64/libc.so.6(clone+0x6d)[0x7f1f4a01abad] )))))<br>
>0-glusterfs-fuse:<br>
>>> >writing to fuse device failed: No such file or directory<br>
>>> ><br>
>>> >####################gsyncd.log outputs:######################<br>
>>> ><br>
>>> >from slave:<br>
>>> >[2020-03-11 08:55:16.384085] I [repce(slave<br>
>>> >master-node/srv/media-storage):96:service_loop] RepceServer:<br>
>>> >terminating on<br>
>>> >reaching EOF.<br>
>>> >[2020-03-11 08:57:55.87364] I [resource(slave<br>
>>> >master-node/srv/media-storage):1105:connect] GLUSTER: Mounting<br>
>gluster<br>
>>> >volume locally...<br>
>>> >[2020-03-11 08:57:56.171372] I [resource(slave<br>
>>> >master-node/srv/media-storage):1128:connect] GLUSTER: Mounted<br>
>gluster<br>
>>> >volume duration=1.0837<br>
>>> >[2020-03-11 08:57:56.173346] I [resource(slave<br>
>>> >master-node/srv/media-storage):1155:service_loop] GLUSTER: slave<br>
>>> >listening<br>
>>> ><br>
>>> >from master:<br>
>>> >[2020-03-11 20:08:55.145453] I [master(worker<br>
>>> >/srv/media-storage):1991:syncjob] Syncer: Sync Time Taken<br>
>>> >duration=134.9987num_files=4661 job=2 return_code=0<br>
>>> >[2020-03-11 20:08:55.285871] I [master(worker<br>
>>> >/srv/media-storage):1421:process] _GMaster: Entry Time Taken MKD=83<br>
>>> >MKN=8109 LIN=0 SYM=0 REN=0 RMD=0 CRE=0 duration=17.0358 UNL=0<br>
>>> >[2020-03-11 20:08:55.286082] I [master(worker<br>
>>> >/srv/media-storage):1431:process] _GMaster: Data/Metadata Time<br>
>Taken<br>
>>> >SETA=83 SETX=0 meta_duration=0.9334 data_duration=135.2497<br>
>DATA=8109<br>
>>> >XATT=0<br>
>>> >[2020-03-11 20:08:55.286410] I [master(worker<br>
>>> >/srv/media-storage):1441:process] _GMaster: Batch Completed<br>
>>> >changelog_end=1583917610 entry_stime=None<br>
>changelog_start=1583917610<br>
>>> >stime=None duration=153.5185 num_changelogs=1 mode=xsync<br>
>>> >[2020-03-11 20:08:55.315442] I [master(worker<br>
>>> >/srv/media-storage):1681:crawl] _GMaster: processing xsync<br>
>changelog<br>
>>><br>
>>><br>
>>path=/var/lib/misc/gluster/gsyncd/media-storage_daredevil01.zingat.com_dr-media/srv-media-storage/xsync/XSYNC-CHANGELOG.1583917613<br>
>>> ><br>
>>> ><br>
>>> >Thank you..<br>
>>> ><br>
>>> >Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>>, 11 Mar 2020 Çar, 12:28<br>
>>> >tarihinde<br>
>>> >şunu yazdı:<br>
>>> ><br>
>>> >> On March 11, 2020 10:09:27 AM GMT+02:00, "Etem Bayoğlu" <<br>
>>> >> <a href="mailto:etembayoglu@gmail.com" target="_blank">etembayoglu@gmail.com</a>> wrote:<br>
>>> >> >Hello community,<br>
>>> >> ><br>
>>> >> >I've set up a glusterfs geo-replication node for disaster<br>
>recovery.<br>
>>> >I<br>
>>> >> >manage about 10TB media data on a gluster volume and I want to<br>
>sync<br>
>>> >all<br>
>>> >> >data to remote location over WAN. So, I created a slave node<br>
>volume<br>
>>> >on<br>
>>> >> >disaster recovery center on remote location and I've started<br>
>geo-rep<br>
>>> >> >session. It has been transferred data fine up to about 800GB,<br>
>but<br>
>>> >> >syncing<br>
>>> >> >has stopped for three days despite gluster geo-rep status active<br>
>and<br>
>>> >> >hybrid<br>
>>> >> >crawl. There is no sending data. I've recreated session and<br>
>>> >restarted<br>
>>> >> >but<br>
>>> >> >still the same.<br>
>>> >> ><br>
>>> >> >#gluster volu geo-rep status<br>
>>> >> ><br>
>>> >> >MASTER NODE MASTER VOL MASTER BRICK <br>
>SLAVE<br>
>>> >> >USER<br>
>>> >> >SLAVE SLAVE NODE<br>
>>> >> >STATUS<br>
>>> >> > CRAWL STATUS LAST_SYNCED<br>
>>> >><br>
>>> >><br>
>>><br>
>>><br>
>>>------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
>>> >> >master-node media-storage /srv/media-storage root<br>
>>> >> > ssh://slave-node::dr-media slave-node Active<br>
>>> >> >Hybrid Crawl N/A<br>
>>> >> ><br>
>>> >> >Any idea? please. Thank you.<br>
>>> >><br>
>>> >> Hi Etem,<br>
>>> >><br>
>>> >> Have you checked the log on both source and destination. Maybe<br>
>they<br>
>>> >can<br>
>>> >> hint you what the issue is.<br>
>>> >><br>
>>> >> Best Regards,<br>
>>> >> Strahil Nikolov<br>
>>> >><br>
>>><br>
>>> Hi Etem,<br>
>>><br>
>>> Nothing obvious....<br>
>>> I don't like this one:<br>
>>><br>
>>> [2020-03-11 19:53:32.721509] E<br>
>>> >[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>>> >(--><br>
>/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f78e10488ea]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f78d83f6221]<br>
>>> >(--><br>
>>><br>
>>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f78d83f7998]<br>
>>> >(--><br>
>>> >/lib64/libpthread.so.0(+0x7e65)[0x7f78dfe89e65] (--><br>
>>> >/lib64/libc.so.6(clone+0x6d)[0x7f78df74f88d] )))))<br>
>0-glusterfs-fuse:<br>
>>> >writing to fuse device failed: No such file or directory<br>
>>><br>
>>> Can you check the health of the slave volume (splitbrains, brick<br>
>>> status,etc) ?<br>
>>><br>
>>> Maybe you can check the logs and find when exactly the master<br>
>stopped<br>
>>> replicating and then checking the logs of the slave at that exact<br>
>time .<br>
>>><br>
>>> Also, you can increase the log level on the slave and then recreate<br>
>the<br>
>>> georep.<br>
>>> For details, check:<br>
>>><br>
>>><br>
>>><br>
><a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level" rel="noreferrer" target="_blank">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level</a><br>
>>><br>
>>> P.S.: Trace/debug can fill up your /var/log, so enable them for a<br>
>short<br>
>>> period of time.<br>
>>><br>
>>> Best Regards,<br>
>>> Strahil Nikolov<br>
>>><br>
>><br>
<br>
Hi Etem,<br>
<br>
The log seems that it ends abruptly, while gluster is creating a list what should be done on the slave.<br>
Is it really ending there ?<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Thanks and Regards,<br></div>Kotresh H R<br></div></div>