<div dir="ltr"><div dir="ltr">Hi,<div><br></div><div>here my slave node logs at the time sync stopped:</div><div><br></div><div>[2020-03-08 03:33:01.489559] I [glusterfsd-mgmt.c:2282:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing<br>[2020-03-08 03:33:01.489298] I [MSGID: 100011] [glusterfsd.c:1679:reincarnate] 0-glusterfsd: Fetching the volume file from server...<br>[2020-03-08 09:49:37.991177] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /tmp/gsyncd-aux-mount-l3PR6o<br>[2020-03-08 09:49:37.993978] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e65) [0x7f2f9f70ce65] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55cc67c20625] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55cc67c2048b] ) 0-: received signum (15), shutting down<br>[2020-03-08 09:49:37.994012] I [fuse-bridge.c:6871:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-l3PR6o'.<br>[2020-03-08 09:49:37.994022] I [fuse-bridge.c:6876:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-l3PR6o'.<br>[2020-03-08 09:49:50.302806] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.3 (args: /usr/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication-slaves/media-storage_slave-node_dr-media/mnt-master-node-srv-media-storage.log --volfile-server=localhost --volfile-id=dr-media --client-pid=-1 /tmp/gsyncd-aux-mount-1AQBe4)<br>[2020-03-08 09:49:50.311167] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 55522<br>[2020-03-08 09:49:50.352351] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0<br>[2020-03-08 09:49:50.352416] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2020-03-08 09:49:50.373248] I [MSGID: 114020] [client.c:2436:notify] 0-dr-media-client-0: parent translators are ready, attempting connect on transport<br>Final graph:<br>+------------------------------------------------------------------------------+<br> 1: volume dr-media-client-0<br> 2: type protocol/client<br> 3: option ping-timeout 42<br> 4: option remote-host slave-node</div><div> 5: option remote-subvolume /data/dr-media<br> 6: option transport-type socket<br> 7: option transport.address-family inet<br> 8: option username 4aafadfa-6ccb-4c2f-920c-1f37ed9eef34<br> 9: option password a8c0f88b-2621-4038-8f65-98068ea71bb0<br> 10: option transport.socket.ssl-enabled off<br> 11: option transport.tcp-user-timeout 0<br> 12: option transport.socket.keepalive-time 20<br> 13: option transport.socket.keepalive-interval 2<br> 14: option transport.socket.keepalive-count 9<br> 15: option send-gids true<br> 16: end-volume<br> 17:<br> 18: volume dr-media-dht<br> 19: type cluster/distribute<br> 20: option lock-migration off<br> 21: option force-migration off<br> 22: subvolumes dr-media-client-0<br> 23: end-volume<br> 24:<br> 25: volume dr-media-write-behind<br> 26: type performance/write-behind<br> 27: option cache-size 8MB<br> 28: option aggregate-size 1MB<br> 29: subvolumes dr-media-dht<br> 30: end-volume<br> 31:<br> 32: volume dr-media-read-ahead<br> 33: type performance/read-ahead<br> 34: subvolumes dr-media-write-behind<br> 35: end-volume<br> 36:<br> 37: volume dr-media-readdir-ahead<br> 38: type performance/readdir-ahead<br> 39: option parallel-readdir off<br> 40: option rda-request-size 131072<br> 41: option rda-cache-limit 10MB<br> 42: subvolumes dr-media-read-ahead<br> 43: end-volume<br> 44:<br> 45: volume dr-media-io-cache<br> 46: type performance/io-cache<br> 47: option cache-size 256MB<br> 48: subvolumes dr-media-readdir-ahead<br> 49: end-volume<br> 50:<br> 51: volume dr-media-open-behind<br> 52: type performance/open-behind<br> 53: subvolumes dr-media-io-cache<br> 54: end-volume<br> 55:<br> 56: volume dr-media-quick-read<br> 57: type performance/quick-read<br> 58: option cache-size 256MB<br> 59: subvolumes dr-media-open-behind<br> 60: end-volume<br> 61:<br> 62: volume dr-media-md-cache<br> 63: type performance/md-cache<br> 64: option cache-posix-acl true<br> 65: subvolumes dr-media-quick-read<br> 66: end-volume<br> 67:<br> 68: volume dr-media-io-threads<br> 69: type performance/io-threads<br> 70: subvolumes dr-media-md-cache<br> 71: end-volume<br> 72:<br> 73: volume dr-media<br> 74: type debug/io-stats<br> 75: option log-level INFO<br> 76: option threads 16<br> 77: option latency-measurement off<br> 78: option count-fop-hits off<br> 79: option global-threading off<br> 80: subvolumes dr-media-io-threads<br> 81: end-volume<br> 82:<br> 83: volume posix-acl-autoload<br> 84: type system/posix-acl<br> 85: subvolumes dr-media<br> 86: end-volume<br> 87:<br> 88: volume gfid-access-autoload<br> 89: type features/gfid-access<br> 90: subvolumes posix-acl-autoload<br> 91: end-volume<br> 92:<br> 93: volume meta-autoload<br> 94: type meta<br> 95: subvolumes gfid-access-autoload<br> 96: end-volume<br> 97:<br>+------------------------------------------------------------------------------+<br>[2020-03-08 09:49:50.388102] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-dr-media-client-0: changing port to 49152 (from 0)<br>[2020-03-08 09:49:50.388132] I [socket.c:865:__socket_shutdown] 0-dr-media-client-0: intentional socket shutdown(12)<br>[2020-03-08 09:49:50.401512] I [MSGID: 114057] [client-handshake.c:1375:select_server_supported_programs] 0-dr-media-client-0: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)<br>[2020-03-08 09:49:50.401765] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.3/xlator/protocol/client.so(+0x381d4) [0x7f7f63daa1d4] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f7f76b3a2f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f7f76b36c11] ) 0-dict: value is NULL [Invalid argument]<br>[2020-03-08 09:49:50.401783] I [MSGID: 114006] [client-handshake.c:1236:client_setvolume] 0-dr-media-client-0: failed to set process-name in handshake msg<br>[2020-03-08 09:49:50.404115] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-dr-media-client-0: Connected to dr-media-client-0, attached to remote volume '/data/dr-media'.<br>[2020-03-08 09:49:50.405761] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22<br>[2020-03-08 09:49:50.405780] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0<br>[2020-03-08 11:49:00.933168] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f7f76b438ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f7f6def1221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f7f6def2998] (--> /lib64/libpthread.so.0(+0x7e65)[0x7f7f75984e65] (--> /lib64/libc.so.6(clone+0x6d)[0x7f7f7524a88d] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 11:53:29.822876] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f7f76b438ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f7f6def1221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f7f6def2998] (--> /lib64/libpthread.so.0(+0x7e65)[0x7f7f75984e65] (--> /lib64/libc.so.6(clone+0x6d)[0x7f7f7524a88d] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 12:00:46.656170] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f7f76b438ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f7f6def1221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f7f6def2998] (--> /lib64/libpthread.so.0(+0x7e65)[0x7f7f75984e65] (--> /lib64/libc.so.6(clone+0x6d)[0x7f7f7524a88d] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br></div><div><br></div><div><br></div><div>master node logs here:</div><div><br></div><div>[2020-03-08 09:49:38.115108] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fb5ab2aa8ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fb5a265c221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8b3a)[0x7fb5a265cb3a] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fb5aa0ebe25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fb5a99b4bad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 09:49:38.932935] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /tmp/gsyncd-aux-mount-Xy8taN<br>[2020-03-08 09:49:38.947634] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7e25) [0x7fb5aa0ebe25] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x557ccdf55625] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x557ccdf5548b] ) 0-: received signum (15), shutting down<br>[2020-03-08 09:49:38.947684] I [fuse-bridge.c:6871:fini] 0-fuse: Unmounting '/tmp/gsyncd-aux-mount-Xy8taN'.<br>[2020-03-08 09:49:38.947704] I [fuse-bridge.c:6876:fini] 0-fuse: Closing fuse connection to '/tmp/gsyncd-aux-mount-Xy8taN'.<br>[2020-03-08 09:49:51.545529] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.3 (args: /usr/sbin/glusterfs --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication/media-storage_slave-node_dr-media/mnt-srv-media-storage.log --volfile-server=localhost --volfile-id=media-storage --client-pid=-1 /tmp/gsyncd-aux-mount-XT9WfC)<br>[2020-03-08 09:49:51.559518] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 18484<br>[2020-03-08 09:49:51.645473] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0<br>[2020-03-08 09:49:51.645624] I [MSGID: 101190] [event-epoll.c:682:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>[2020-03-08 09:49:51.701470] I [MSGID: 114020] [client.c:2436:notify] 0-media-storage-client-0: parent translators are ready, attempting connect on transport<br>[2020-03-08 09:49:51.702908] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-media-storage-client-0: changing port to 49152 (from 0)<br>[2020-03-08 09:49:51.702961] I [socket.c:865:__socket_shutdown] 0-media-storage-client-0: intentional socket shutdown(12)<br>[2020-03-08 09:49:51.703807] I [MSGID: 114057] [client-handshake.c:1375:select_server_supported_programs] 0-media-storage-client-0: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)<br>[2020-03-08 09:49:51.704147] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.3/xlator/protocol/client.so(+0x381d4) [0x7fc5c01031d4] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7fc5ce8d82f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7fc5ce8d4c11] ) 0-dict: value is NULL [Invalid argument]<br>[2020-03-08 09:49:51.704178] I [MSGID: 114006] [client-handshake.c:1236:client_setvolume] 0-media-storage-client-0: failed to set process-name in handshake msg<br>[2020-03-08 09:49:51.705982] I [MSGID: 114046] [client-handshake.c:1105:client_setvolume_cbk] 0-media-storage-client-0: Connected to media-storage-client-0, attached to remote volume '/srv/media-storage'.<br>[2020-03-08 09:49:51.707627] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22<br>[2020-03-08 09:49:51.707658] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0<br>[2020-03-08 09:56:26.875082] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br>[2020-03-08 10:12:35.190809] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br>[2020-03-08 10:25:06.240795] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br>[2020-03-08 10:40:33.946794] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br>[2020-03-08 10:43:50.459247] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 10:55:27.034947] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br>[2020-03-08 11:05:53.483207] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br>[2020-03-08 11:26:01.492270] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br>[2020-03-08 11:32:56.618737] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 11:37:50.475099] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 11:40:54.362173] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br>[2020-03-08 11:42:35.859423] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 11:44:24.906383] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 11:47:45.474723] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 11:50:58.127202] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 11:52:55.616968] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 11:56:24.039211] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br>[2020-03-08 11:57:56.031648] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 12:06:19.686974] E [fuse-bridge.c:227:check_and_dump_fuse_W] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7fc5ce8e18ea] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7fc5c5c93221] (--> /usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7fc5c5c94998] (--> /lib64/libpthread.so.0(+0x7e25)[0x7fc5cd722e25] (--> /lib64/libc.so.6(clone+0x6d)[0x7fc5ccfebbad] ))))) 0-glusterfs-fuse: writing to fuse device failed: No such file or directory<br>[2020-03-08 12:12:41.888889] E [fuse-bridge.c:4188:fuse_xattr_cbk] 0-glusterfs-fuse: extended attribute not supported by the backend storage<br></div><div><br></div><div><br></div><div><br></div><div>I see just no such file or directory error logs for 5 days. I shifted log level to DEBUG I will send them also. </div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com">hunter86_bg@yahoo.com</a>>, 12 Mar 2020 Per, 07:55 tarihinde şunu yazdı:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On March 11, 2020 10:17:05 PM GMT+02:00, "Etem Bayoğlu" <<a href="mailto:etembayoglu@gmail.com" target="_blank">etembayoglu@gmail.com</a>> wrote:<br>
>Hi Strahil,<br>
><br>
>Thank you for your response. when I tail logs on both master and slave<br>
>I<br>
>get this:<br>
><br>
>on slave, from<br>
>/var/log/glusterfs/geo-replication-slaves/<geo-session>/mnt-XXX.log<br>
>file:<br>
><br>
>[2020-03-11 19:53:32.721509] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f78e10488ea]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f78d83f6221]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f78d83f7998]<br>
>(--><br>
>/lib64/libpthread.so.0(+0x7e65)[0x7f78dfe89e65] (--><br>
>/lib64/libc.so.6(clone+0x6d)[0x7f78df74f88d] ))))) 0-glusterfs-fuse:<br>
>writing to fuse device failed: No such file or directory<br>
>[2020-03-11 19:53:32.723758] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f78e10488ea]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f78d83f6221]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f78d83f7998]<br>
>(--><br>
>/lib64/libpthread.so.0(+0x7e65)[0x7f78dfe89e65] (--><br>
>/lib64/libc.so.6(clone+0x6d)[0x7f78df74f88d] ))))) 0-glusterfs-fuse:<br>
>writing to fuse device failed: No such file or directory<br>
><br>
>on master,<br>
>from /var/log/glusterfs/geo-replication/<geo-session>/mnt-XXX.log file:<br>
><br>
>[2020-03-11 19:40:55.872002] E [fuse-bridge.c:4188:fuse_xattr_cbk]<br>
>0-glusterfs-fuse: extended attribute not supported by the backend<br>
>storage<br>
>[2020-03-11 19:40:58.389748] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f1f4b9108ea]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f1f42cc2221]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f1f42cc3998]<br>
>(--><br>
>/lib64/libpthread.so.0(+0x7e25)[0x7f1f4a751e25] (--><br>
>/lib64/libc.so.6(clone+0x6d)[0x7f1f4a01abad] ))))) 0-glusterfs-fuse:<br>
>writing to fuse device failed: No such file or directory<br>
>[2020-03-11 19:41:08.214591] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f1f4b9108ea]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f1f42cc2221]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f1f42cc3998]<br>
>(--><br>
>/lib64/libpthread.so.0(+0x7e25)[0x7f1f4a751e25] (--><br>
>/lib64/libc.so.6(clone+0x6d)[0x7f1f4a01abad] ))))) 0-glusterfs-fuse:<br>
>writing to fuse device failed: No such file or directory<br>
>[2020-03-11 19:53:59.275469] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f1f4b9108ea]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f1f42cc2221]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f1f42cc3998]<br>
>(--><br>
>/lib64/libpthread.so.0(+0x7e25)[0x7f1f4a751e25] (--><br>
>/lib64/libc.so.6(clone+0x6d)[0x7f1f4a01abad] ))))) 0-glusterfs-fuse:<br>
>writing to fuse device failed: No such file or directory<br>
><br>
>####################gsyncd.log outputs:######################<br>
><br>
>from slave:<br>
>[2020-03-11 08:55:16.384085] I [repce(slave<br>
>master-node/srv/media-storage):96:service_loop] RepceServer:<br>
>terminating on<br>
>reaching EOF.<br>
>[2020-03-11 08:57:55.87364] I [resource(slave<br>
>master-node/srv/media-storage):1105:connect] GLUSTER: Mounting gluster<br>
>volume locally...<br>
>[2020-03-11 08:57:56.171372] I [resource(slave<br>
>master-node/srv/media-storage):1128:connect] GLUSTER: Mounted gluster<br>
>volume duration=1.0837<br>
>[2020-03-11 08:57:56.173346] I [resource(slave<br>
>master-node/srv/media-storage):1155:service_loop] GLUSTER: slave<br>
>listening<br>
><br>
>from master:<br>
>[2020-03-11 20:08:55.145453] I [master(worker<br>
>/srv/media-storage):1991:syncjob] Syncer: Sync Time Taken<br>
>duration=134.9987num_files=4661 job=2 return_code=0<br>
>[2020-03-11 20:08:55.285871] I [master(worker<br>
>/srv/media-storage):1421:process] _GMaster: Entry Time Taken MKD=83<br>
>MKN=8109 LIN=0 SYM=0 REN=0 RMD=0 CRE=0 duration=17.0358 UNL=0<br>
>[2020-03-11 20:08:55.286082] I [master(worker<br>
>/srv/media-storage):1431:process] _GMaster: Data/Metadata Time Taken<br>
>SETA=83 SETX=0 meta_duration=0.9334 data_duration=135.2497 DATA=8109<br>
>XATT=0<br>
>[2020-03-11 20:08:55.286410] I [master(worker<br>
>/srv/media-storage):1441:process] _GMaster: Batch Completed<br>
>changelog_end=1583917610 entry_stime=None changelog_start=1583917610<br>
>stime=None duration=153.5185 num_changelogs=1 mode=xsync<br>
>[2020-03-11 20:08:55.315442] I [master(worker<br>
>/srv/media-storage):1681:crawl] _GMaster: processing xsync changelog<br>
>path=/var/lib/misc/gluster/gsyncd/media-storage_daredevil01.zingat.com_dr-media/srv-media-storage/xsync/XSYNC-CHANGELOG.1583917613<br>
><br>
><br>
>Thank you..<br>
><br>
>Strahil Nikolov <<a href="mailto:hunter86_bg@yahoo.com" target="_blank">hunter86_bg@yahoo.com</a>>, 11 Mar 2020 Çar, 12:28<br>
>tarihinde<br>
>şunu yazdı:<br>
><br>
>> On March 11, 2020 10:09:27 AM GMT+02:00, "Etem Bayoğlu" <<br>
>> <a href="mailto:etembayoglu@gmail.com" target="_blank">etembayoglu@gmail.com</a>> wrote:<br>
>> >Hello community,<br>
>> ><br>
>> >I've set up a glusterfs geo-replication node for disaster recovery.<br>
>I<br>
>> >manage about 10TB media data on a gluster volume and I want to sync<br>
>all<br>
>> >data to remote location over WAN. So, I created a slave node volume<br>
>on<br>
>> >disaster recovery center on remote location and I've started geo-rep<br>
>> >session. It has been transferred data fine up to about 800GB, but<br>
>> >syncing<br>
>> >has stopped for three days despite gluster geo-rep status active and<br>
>> >hybrid<br>
>> >crawl. There is no sending data. I've recreated session and<br>
>restarted<br>
>> >but<br>
>> >still the same.<br>
>> ><br>
>> >#gluster volu geo-rep status<br>
>> ><br>
>> >MASTER NODE MASTER VOL MASTER BRICK SLAVE<br>
>> >USER<br>
>> >SLAVE SLAVE NODE<br>
>> >STATUS<br>
>> > CRAWL STATUS LAST_SYNCED<br>
>><br>
>><br>
>>------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------<br>
>> >master-node media-storage /srv/media-storage root<br>
>> > ssh://slave-node::dr-media slave-node Active<br>
>> >Hybrid Crawl N/A<br>
>> ><br>
>> >Any idea? please. Thank you.<br>
>><br>
>> Hi Etem,<br>
>><br>
>> Have you checked the log on both source and destination. Maybe they<br>
>can<br>
>> hint you what the issue is.<br>
>><br>
>> Best Regards,<br>
>> Strahil Nikolov<br>
>><br>
<br>
Hi Etem,<br>
<br>
Nothing obvious....<br>
I don't like this one:<br>
<br>
[2020-03-11 19:53:32.721509] E<br>
>[fuse-bridge.c:227:check_and_dump_fuse_W]<br>
>(--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13a)[0x7f78e10488ea]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x8221)[0x7f78d83f6221]<br>
>(--><br>
>/usr/lib64/glusterfs/7.3/xlator/mount/fuse.so(+0x9998)[0x7f78d83f7998]<br>
>(--><br>
>/lib64/libpthread.so.0(+0x7e65)[0x7f78dfe89e65] (--><br>
>/lib64/libc.so.6(clone+0x6d)[0x7f78df74f88d] ))))) 0-glusterfs-fuse:<br>
>writing to fuse device failed: No such file or directory<br>
<br>
Can you check the health of the slave volume (splitbrains, brick status,etc) ?<br>
<br>
Maybe you can check the logs and find when exactly the master stopped replicating and then checking the logs of the slave at that exact time .<br>
<br>
Also, you can increase the log level on the slave and then recreate the georep.<br>
For details, check: <br>
<br>
<a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level" rel="noreferrer" target="_blank">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level</a><br>
<br>
P.S.: Trace/debug can fill up your /var/log, so enable them for a short period of time.<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
</blockquote></div>