[Gluster-users] Faulty staus in geo-replication session of a sub-volume
Naranderan Ramakrishnan
rnaranbe at gmail.com
Tue May 26 07:38:44 UTC 2020
Dear Developers/Users,
A geo-rep session of a sub-volume is in 'faulty' status. Please find the
setup and log details below.
Setup Details:
Gluster version - 7.0
Volume configuration - 2x3 (DxR)
gysncd permission(master) - root
gysncd permission(slave) - sas (non-root)
glusterd, glusterfsd permissions(master) - root
glusterd, glusterfsd permissions(slave) - root
Log details:
In the *master gyncd* log, this traceback is printed *repeatedly*.
> [2020-05-22 12:09:43.838727] I [master(worker
> /home/sas/gluster/data/code-ide):1991:syncjob] Syncer: Sync Time Taken
> duration=0.4240 num_files=1 job=1 return_code=0
> [2020-05-22 12:09:43.944392] E [repce(worker
> /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed
> call=261471:140535761106752:1590149383.8 method=entry_ops error=OSError
> [2020-05-22 12:09:43.944746] E [syncdutils(worker
> /home/sas/gluster/data/code-ide):338:log_raise_exception] <top>: FAIL:
> Traceback (most recent call last):
> File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in
> main
> func(args)
> File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in
> subcmd_worker
> local.service_loop(remote)
> File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1305,
> in service_loop
> g3.crawlwrap(oneshot=True)
> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in
> crawlwrap
> self.crawl()
> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1592, in
> crawl
> self.changelogs_batch_process(changes)
> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1492, in
> changelogs_batch_process
> self.process(batch)
> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in
> process
> self.process_change(change, done, retry)
> File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in
> process_change
> failures = self.slave.server.entry_ops(entries)
> File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in
> __call__
> return self.ins(self.meth, *a)
> File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in
> __call__
> raise res
> OSError: [Errno 13] Permission denied:
> '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
> [2020-05-22 12:09:43.968710] I [repce(agent
> /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating
> on reaching EOF.
> [2020-05-22 12:09:44.912470] I [monitor(monitor):280:monitor] Monitor:
> worker died in startup phase brick=/home/sas/gluster/data/code-ide
> [2020-05-22 12:09:44.913692] I
> [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status
> Change status=Faulty
>
In salve end, these are printed *repeatedly*.
>
> [2020-05-22 11:23:26.65115] W [gsyncd(slave
> 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session
> config file not exists, using the default config
> path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
> [2020-05-22 11:23:26.77414] I [resource(slave
> 10.47.8.153/home/sas/gluster/data/code-ide):1105:connect] GLUSTER:
> Mounting gluster volume locally...
> [2020-05-22 11:23:27.297466] I [resource(slave
> 10.47.8.153/home/sas/gluster/data/code-ide):1128:connect] GLUSTER:
> Mounted gluster volume duration=1.2199
> [2020-05-22 11:23:27.298125] I [resource(slave
> 10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop] GLUSTER:
> slave listening
> [2020-05-22 11:23:32.654939] E [repce(slave
> 10.47.8.153/home/sas/gluster/data/code-ide):122:worker] <top>: call
> failed:
> Traceback (most recent call last):
> File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in
> worker
> res = getattr(self.obj, rmeth)(*in_data[2:])
> File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706,
> in entry_ops
> collect_failure(e, cmd_ret, uid, gid)
> File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444,
> in collect_failure
> disk_gfid)
> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687,
> in get_slv_dir_path
> [ENOENT], [ESTALE])
> File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546,
> in errno_wrap
> return call(*arg)
> OSError: [Errno 13] Permission denied:
> '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
> [2020-05-22 11:23:32.741317] I [repce(slave
> 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer:
> terminating on reaching EOF.
>
Additional info:
Parallel to this
GFID(/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e)
mentioned in master gyscnd log, there are some files with ---------T
permission & *trusted.glusterfs.dht.linkto* extended attribute in the
master subvolume for which geo-rep session is in faulty status. Not sure
whether this is related to this geo-rep issue or not.
I have attached a few screenshots and log stmts for further info. Please
let us know how we should solve this.
Thanks in advance.
Regards,
Naranderan R
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200526/49963943/attachment-0001.html>
-------------- next part --------------
sas at 10.37.11.246/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide>tail -f gsyncd.log
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, in entry_ops
collect_failure(e, cmd_ret, uid, gid)
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, in collect_failure
disk_gfid)
File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
[ENOENT], [ESTALE])
File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
return call(*arg)
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 11:23:14.762231] I [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.
[2020-05-22 11:23:26.65115] W [gsyncd(slave 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session config file not exists, using the default config path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 11:23:26.77414] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1105:connect] GLUSTER: Mounting gluster volume locally...
[2020-05-22 11:23:27.297466] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1128:connect] GLUSTER: Mounted gluster volume duration=1.2199
[2020-05-22 11:23:27.298125] I [resource(slave 10.47.8.153/home/sas/gluster/data/code-ide):1155:service_loop] GLUSTER: slave listening
[2020-05-22 11:23:32.654939] E [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):122:worker] <top>: call failed:
Traceback (most recent call last):
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 118, in worker
res = getattr(self.obj, rmeth)(*in_data[2:])
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 706, in entry_ops
collect_failure(e, cmd_ret, uid, gid)
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 444, in collect_failure
disk_gfid)
File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 687, in get_slv_dir_path
[ENOENT], [ESTALE])
File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 546, in errno_wrap
return call(*arg)
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 11:23:32.741317] I [repce(slave 10.47.8.153/home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.
[2020-05-22 11:23:44.797867] W [gsyncd(slave 10.47.8.153/home/sas/gluster/data/code-ide):307:main] <top>: Session config file not exists, using the default config path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Slave-mnt-10.47.8.153-home-sas-gluster-data-code-ide-log-tailf.png
Type: image/png
Size: 1827884 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200526/49963943/attachment-0001.png>
-------------- next part --------------
root at 10.37.11.246/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide#ls
gsyncd.log mnt-10.47.8.153-home-sas-gluster-data-code-ide.log
root at 10.37.11.246/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide#tail -f mnt-10.47.8.153-home-sas-gluster-data-code-ide.log
[2020-05-22 11:31:10.001901] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-4: Connected to code-ide-client-4, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:10.001955] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-1: Client-quorum is met
[2020-05-22 11:31:10.002114] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-1: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:10.003158] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-5: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:10.003215] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f8ec76fb244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f8ed62782f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f8ed6274c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:10.003227] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-5: failed to set process-name in handshake msg
[2020-05-22 11:31:10.003564] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-5: Connected to code-ide-client-5, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:10.004717] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2020-05-22 11:31:10.004746] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0
[2020-05-22 11:31:10.007121] I [MSGID: 108031] [afr-common.c:2580:afr_local_discovery_cbk] 0-code-ide-replicate-1: selecting local read_child code-ide-client-5
[2020-05-22 11:31:16.385568] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /var/mountbroker-root/user1000/mtpt-sas-IgRBaT
[2020-05-22 11:31:16.385666] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f8ed50c2dd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x56136fdfa625] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x56136fdfa48b] ) 0-: received signum (15), shutting down
[2020-05-22 11:31:16.385687] I [fuse-bridge.c:6871:fini] 0-fuse: Unmounting '/var/mountbroker-root/user1000/mtpt-sas-IgRBaT'.
[2020-05-22 11:31:16.385698] I [fuse-bridge.c:6876:fini] 0-fuse: Closing fuse connection to '/var/mountbroker-root/user1000/mtpt-sas-IgRBaT'.
[2020-05-22 11:31:27.695486] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.0 (args: /usr/sbin/glusterfs --user-map-root=sas --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide/mnt-10.47.8.153-home-sas-gluster-data-code-ide.log --volfile-server=localhost --volfile-id=code-ide --client-pid=-1 /var/mountbroker-root/user1000/mtpt-sas-gXOnl6)
[2020-05-22 11:31:27.698895] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 230635
[2020-05-22 11:31:27.707687] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-05-22 11:31:27.707741] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-05-22 11:31:27.714024] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-0: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:27.716381] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-1: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:27.716834] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-0: changing port to 49153 (from 0)
[2020-05-22 11:31:27.716863] I [socket.c:864:__socket_shutdown] 0-code-ide-client-0: intentional socket shutdown(12)
[2020-05-22 11:31:27.718763] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-2: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:27.721166] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-0: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:27.721451] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.721476] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-0: failed to set process-name in handshake msg
[2020-05-22 11:31:27.721587] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-1: changing port to 49152 (from 0)
[2020-05-22 11:31:27.721606] I [socket.c:864:__socket_shutdown] 0-code-ide-client-1: intentional socket shutdown(13)
[2020-05-22 11:31:27.723607] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-3: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:27.725797] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-0: Connected to code-ide-client-0, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:27.725826] I [MSGID: 108005] [afr-common.c:5280:__afr_handle_child_up_event] 0-code-ide-replicate-0: Subvolume 'code-ide-client-0' came back up; going online.
[2020-05-22 11:31:27.725971] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:27.726190] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:27.726263] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.726279] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-1: failed to set process-name in handshake msg
[2020-05-22 11:31:27.726404] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-2: changing port to 49152 (from 0)
[2020-05-22 11:31:27.726419] I [socket.c:864:__socket_shutdown] 0-code-ide-client-2: intentional socket shutdown(12)
[2020-05-22 11:31:27.728149] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-4: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:27.730264] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-1: Connected to code-ide-client-1, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:27.730284] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-0: Client-quorum is met
[2020-05-22 11:31:27.730395] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:27.730634] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-2: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:27.730714] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.730731] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-2: failed to set process-name in handshake msg
[2020-05-22 11:31:27.730867] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-3: changing port to 49153 (from 0)
[2020-05-22 11:31:27.730883] I [socket.c:864:__socket_shutdown] 0-code-ide-client-3: intentional socket shutdown(13)
[2020-05-22 11:31:27.732462] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-5: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:27.734593] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-2: Connected to code-ide-client-2, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:27.734935] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-3: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:27.735007] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.735026] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-3: failed to set process-name in handshake msg
[2020-05-22 11:31:27.735187] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-4: changing port to 49152 (from 0)
[2020-05-22 11:31:27.735203] I [socket.c:864:__socket_shutdown] 0-code-ide-client-4: intentional socket shutdown(12)
Final graph:
+------------------------------------------------------------------------------+
1: volume code-ide-client-0
2: type protocol/client
3: option opversion 70000
4: option clnt-lk-version 1
5: option volfile-checksum 0
6: option volfile-key code-ide
7: option client-version 7.0
8: option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-0-RECON_NO:-0
9: option fops-version 1298437
10: option ping-timeout 42
11: option remote-host 10.37.11.253
12: option remote-subvolume /home/sas/gluster/data/code-ide
13: option transport-type socket
14: option transport.address-family inet
15: option username 57f0d708-564c-4762-bcba-39bbbd32a314
16: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
17: option transport.socket.ssl-enabled off
18: option transport.tcp-user-timeout 0
19: option transport.socket.keepalive-time 20
20: option transport.socket.keepalive-interval 2
21: option transport.socket.keepalive-count 9
22: option send-gids true
23: end-volume
24:
25: volume code-ide-client-1
26: type protocol/client
27: option opversion 70000
28: option clnt-lk-version 1
29: option volfile-checksum 0
30: option volfile-key code-ide
31: option client-version 7.0
32: option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-1-RECON_NO:-0
33: option fops-version 1298437
34: option ping-timeout 42
35: option remote-host 10.37.11.252
36: option remote-subvolume /home/sas/gluster/data/code-ide
37: option transport-type socket
38: option transport.address-family inet
39: option username 57f0d708-564c-4762-bcba-39bbbd32a314
40: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
41: option transport.socket.ssl-enabled off
42: option transport.tcp-user-timeout 0
43: option transport.socket.keepalive-time 20
44: option transport.socket.keepalive-interval 2
45: option transport.socket.keepalive-count 9
46: option send-gids true
47: end-volume
48:
49: volume code-ide-client-2
50: type protocol/client
51: option opversion 70000
52: option clnt-lk-version 1
53: option volfile-checksum 0
54: option volfile-key code-ide
55: option client-version 7.0
56: option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-2-RECON_NO:-0
57: option fops-version 1298437
58: option ping-timeout 42
59: option remote-host 10.37.11.251
60: option remote-subvolume /home/sas/gluster/data/code-ide
61: option transport-type socket
62: option transport.address-family inet
63: option username 57f0d708-564c-4762-bcba-39bbbd32a314
64: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
65: option transport.socket.ssl-enabled off
66: option transport.tcp-user-timeout 0
67: option transport.socket.keepalive-time 20
68: option transport.socket.keepalive-interval 2
69: option transport.socket.keepalive-count 9
70: option send-gids true
71: end-volume
72:
73: volume code-ide-replicate-0
74: type cluster/replicate
75: option afr-pending-xattr code-ide-client-0,code-ide-client-1,code-ide-client-2
76: option use-compound-fops off
77: subvolumes code-ide-client-0 code-ide-client-1 code-ide-client-2
78: end-volume
79:
80: volume code-ide-client-3
81: type protocol/client
82: option opversion 70000
83: option clnt-lk-version 1
84: option volfile-checksum 0
85: option volfile-key code-ide
86: option client-version 7.0
87: option process-uuid CTX_ID:c97ffcc5-c4de-4f4e-a13e-71c83e8ce7ac-GRAPH_ID:0-PID:230635-HOST:10.37.11.246-PC_NAME:code-ide-client-3-RECON_NO:-0
88: option fops-version 1298437
89: option ping-timeout 42
90: option remote-host 10.37.11.250
91: option remote-subvolume /home/sas/gluster/data/code-ide
92: option transport-type socket
93: option transport.address-family inet
94: option username 57f0d708-564c-4762-bcba-39bbbd32a314
95: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
96: option transport.socket.ssl-enabled off
97: option transport.tcp-user-timeout 0
98: option transport.socket.keepalive-time 20
99: option transport.socket.keepalive-interval 2
100: option transport.socket.keepalive-count 9
101: option send-gids true
102: end-volume
103:
104: volume code-ide-client-4
105: type protocol/client
106: option ping-timeout 42
107: option remote-host 10.37.11.249
108: option remote-subvolume /home/sas/gluster/data/code-ide
109: option transport-type socket
110: option transport.address-family inet
111: option username 57f0d708-564c-4762-bcba-39bbbd32a314
112: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
113: option transport.socket.ssl-enabled off
114: option transport.tcp-user-timeout 0
115: option transport.socket.keepalive-time 20
116: option transport.socket.keepalive-interval 2
117: option transport.socket.keepalive-count 9
118: option send-gids true
119: end-volume
120:
121: volume code-ide-client-5
122: type protocol/client
123: option ping-timeout 42
124: option remote-host 10.37.11.246
125: option remote-subvolume /home/sas/gluster/data/code-ide
126: option transport-type socket
127: option transport.address-family inet
128: option username 57f0d708-564c-4762-bcba-39bbbd32a314
129: option password 23c1f9a2-31b1-4300-bf41-4b0dd55ecca1
130: option transport.socket.ssl-enabled off
131: option transport.tcp-user-timeout 0
132: option transport.socket.keepalive-time 20
133: option transport.socket.keepalive-interval 2
134: option transport.socket.keepalive-count 9
135: option send-gids true
136: end-volume
137:
138: volume code-ide-replicate-1
139: type cluster/replicate
140: option afr-pending-xattr code-ide-client-3,code-ide-client-4,code-ide-client-5
141: option use-compound-fops off
142: subvolumes code-ide-client-3 code-ide-client-4 code-ide-client-5
143: end-volume
144:
145: volume code-ide-dht
146: type cluster/distribute
147: option lock-migration off
148: option force-migration off
149: subvolumes code-ide-replicate-0 code-ide-replicate-1
150: end-volume
151:
152: volume code-ide-utime
153: type features/utime
154: option noatime on
155: subvolumes code-ide-dht
156: end-volume
157:
158: volume code-ide-write-behind
159: type performance/write-behind
160: subvolumes code-ide-utime
161: end-volume
162:
163: volume code-ide-read-ahead
164: type performance/read-ahead
165: subvolumes code-ide-write-behind
166: end-volume
167:
168: volume code-ide-readdir-ahead
169: type performance/readdir-ahead
170: option parallel-readdir off
171: option rda-request-size 131072
172: option rda-cache-limit 10MB
173: subvolumes code-ide-read-ahead
174: end-volume
175:
176: volume code-ide-io-cache
177: type performance/io-cache
178: subvolumes code-ide-readdir-ahead
179: end-volume
180:
181: volume code-ide-open-behind
182: type performance/open-behind
183: subvolumes code-ide-io-cache
184: end-volume
185:
186: volume code-ide-quick-read
187: type performance/quick-read
188: subvolumes code-ide-open-behind
189: end-volume
190:
191: volume code-ide-md-cache
192: type performance/md-cache
193: option cache-posix-acl true
194: subvolumes code-ide-quick-read
195: end-volume
196:
197: volume code-ide
198: type debug/io-stats
199: option log-level INFO
200: option threads 16
201: option latency-measurement off
202: option count-fop-hits off
203: option global-threading off
204: subvolumes code-ide-md-cache
205: end-volume
206:
207: volume posix-acl-autoload
208: type system/posix-acl
209: subvolumes code-ide
210: end-volume
211:
212: volume gfid-access-autoload
213: type features/gfid-access
214: subvolumes posix-acl-autoload
215: end-volume
216:
217: volume meta-autoload
218: type meta
219: subvolumes gfid-access-autoload
220: end-volume
221:
+------------------------------------------------------------------------------+
[2020-05-22 11:31:27.737910] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-3: Connected to code-ide-client-3, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:27.737926] I [MSGID: 108005] [afr-common.c:5280:__afr_handle_child_up_event] 0-code-ide-replicate-1: Subvolume 'code-ide-client-3' came back up; going online.
[2020-05-22 11:31:27.738026] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-1: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:27.738195] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-5: changing port to 49152 (from 0)
[2020-05-22 11:31:27.738212] I [socket.c:864:__socket_shutdown] 0-code-ide-client-5: intentional socket shutdown(13)
[2020-05-22 11:31:27.739521] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-4: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:27.739592] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.739608] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-4: failed to set process-name in handshake msg
[2020-05-22 11:31:27.740090] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-4: Connected to code-ide-client-4, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:27.740105] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-1: Client-quorum is met
[2020-05-22 11:31:27.740222] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-1: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:27.741492] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-5: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:27.741547] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f92fc0f8244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f930ac752f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f930ac71c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:27.741560] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-5: failed to set process-name in handshake msg
[2020-05-22 11:31:27.741931] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-5: Connected to code-ide-client-5, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:27.743285] I [fuse-bridge.c:5166:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.22
[2020-05-22 11:31:27.743338] I [fuse-bridge.c:5777:fuse_graph_sync] 0-fuse: switched to graph 0
[2020-05-22 11:31:27.746010] I [MSGID: 108031] [afr-common.c:2580:afr_local_discovery_cbk] 0-code-ide-replicate-1: selecting local read_child code-ide-client-5
[2020-05-22 11:31:34.214315] I [fuse-bridge.c:6083:fuse_thread_proc] 0-fuse: initiating unmount of /var/mountbroker-root/user1000/mtpt-sas-gXOnl6
[2020-05-22 11:31:34.214450] W [glusterfsd.c:1596:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7dd5) [0x7f9309abfdd5] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55dcea04e625] -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55dcea04e48b] ) 0-: received signum (15), shutting down
[2020-05-22 11:31:34.214485] I [fuse-bridge.c:6871:fini] 0-fuse: Unmounting '/var/mountbroker-root/user1000/mtpt-sas-gXOnl6'.
[2020-05-22 11:31:34.214499] I [fuse-bridge.c:6876:fini] 0-fuse: Closing fuse connection to '/var/mountbroker-root/user1000/mtpt-sas-gXOnl6'.
[2020-05-22 11:31:46.446624] I [MSGID: 100030] [glusterfsd.c:2867:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 7.0 (args: /usr/sbin/glusterfs --user-map-root=sas --aux-gfid-mount --acl --log-level=INFO --log-file=/var/log/glusterfs/geo-replication-slaves/code-ide_10.37.11.252_code-ide/mnt-10.47.8.153-home-sas-gluster-data-code-ide.log --volfile-server=localhost --volfile-id=code-ide --client-pid=-1 /var/mountbroker-root/user1000/mtpt-sas-oq0aUM)
[2020-05-22 11:31:46.450189] I [glusterfsd.c:2594:daemonize] 0-glusterfs: Pid of current running process is 230980
[2020-05-22 11:31:46.458835] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2020-05-22 11:31:46.458826] I [MSGID: 101190] [event-epoll.c:674:event_dispatch_epoll_worker] 0-epoll: Started thread with index 0
[2020-05-22 11:31:46.465054] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-0: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:46.467356] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-1: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:46.467849] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-0: changing port to 49153 (from 0)
[2020-05-22 11:31:46.467872] I [socket.c:864:__socket_shutdown] 0-code-ide-client-0: intentional socket shutdown(12)
[2020-05-22 11:31:46.469623] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-2: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:46.472059] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-1: changing port to 49152 (from 0)
[2020-05-22 11:31:46.472078] I [socket.c:864:__socket_shutdown] 0-code-ide-client-1: intentional socket shutdown(13)
[2020-05-22 11:31:46.473883] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-3: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:46.475942] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-0: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:46.476169] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f948df1a244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f949ca972f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f949ca93c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:46.476186] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-0: failed to set process-name in handshake msg
[2020-05-22 11:31:46.476472] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-1: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:46.476555] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f948df1a244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f949ca972f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f949ca93c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:46.476574] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-1: failed to set process-name in handshake msg
[2020-05-22 11:31:46.476837] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-0: Connected to code-ide-client-0, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:46.476858] I [MSGID: 108005] [afr-common.c:5280:__afr_handle_child_up_event] 0-code-ide-replicate-0: Subvolume 'code-ide-client-0' came back up; going online.
[2020-05-22 11:31:46.476979] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:46.477009] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-2: changing port to 49152 (from 0)
[2020-05-22 11:31:46.477025] I [socket.c:864:__socket_shutdown] 0-code-ide-client-2: intentional socket shutdown(12)
[2020-05-22 11:31:46.478210] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-4: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:46.480275] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-1: Connected to code-ide-client-1, attached to remote volume '/home/sas/gluster/data/code-ide'.
[2020-05-22 11:31:46.480294] I [MSGID: 108002] [afr-common.c:5647:afr_notify] 0-code-ide-replicate-0: Client-quorum is met
[2020-05-22 11:31:46.480406] E [MSGID: 101172] [events.c:89:_gf_event] 0-code-ide-replicate-0: inet_pton failed with return code 0 [Invalid argument]
[2020-05-22 11:31:46.480644] I [MSGID: 114057] [client-handshake.c:1376:select_server_supported_programs] 0-code-ide-client-2: Using Program GlusterFS 4.x v1, Num (1298437), Version (400)
[2020-05-22 11:31:46.480719] W [dict.c:999:str_to_data] (-->/usr/lib64/glusterfs/7.0/xlator/protocol/client.so(+0x38244) [0x7f948df1a244] -->/lib64/libglusterfs.so.0(dict_set_str+0x16) [0x7f949ca972f6] -->/lib64/libglusterfs.so.0(str_to_data+0x71) [0x7f949ca93c11] ) 0-dict: value is NULL [Invalid argument]
[2020-05-22 11:31:46.480744] I [MSGID: 114006] [client-handshake.c:1237:client_setvolume] 0-code-ide-client-2: failed to set process-name in handshake msg
[2020-05-22 11:31:46.480870] I [rpc-clnt.c:1963:rpc_clnt_reconfig] 0-code-ide-client-3: changing port to 49153 (from 0)
[2020-05-22 11:31:46.480887] I [socket.c:864:__socket_shutdown] 0-code-ide-client-3: intentional socket shutdown(13)
[2020-05-22 11:31:46.482509] I [MSGID: 114020] [client.c:2436:notify] 0-code-ide-client-5: parent translators are ready, attempting connect on transport
[2020-05-22 11:31:46.484519] I [MSGID: 114046] [client-handshake.c:1106:client_setvolume_cbk] 0-code-ide-client-2: Connected to code-ide-client-2, attached to remote volume '/home/sas/gluster/data/code-ide'.
-------------- next part --------------
sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>date
Fri May 22 05:09:23 PDT 2020
sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>
sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>tail -f gsyncd.log
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 12:09:25.176702] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.
[2020-05-22 12:09:26.144371] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-ide
[2020-05-22 12:09:26.145390] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty
[2020-05-22 12:09:36.496685] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...
[2020-05-22 12:09:36.496835] I [monitor(monitor):159:monitor] Monitor: starting gsyncd worker brick=/home/sas/gluster/data/code-ide slave_node=10.37.11.246
[2020-05-22 12:09:36.541825] I [gsyncd(agent /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 12:09:36.545790] I [changelogagent(agent /home/sas/gluster/data/code-ide):72:__init__] ChangelogAgent: Agent listining...
[2020-05-22 12:09:36.554223] I [gsyncd(worker /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 12:09:36.570023] I [resource(worker /home/sas/gluster/data/code-ide):1394:connect_remote] SSH: Initializing SSH connection between master and slave...
[2020-05-22 12:09:38.784018] I [resource(worker /home/sas/gluster/data/code-ide):1443:connect_remote] SSH: SSH connection between master and slave established. duration=2.2138
[2020-05-22 12:09:38.784291] I [resource(worker /home/sas/gluster/data/code-ide):1113:connect] GLUSTER: Mounting gluster volume locally...
[2020-05-22 12:09:39.906485] I [resource(worker /home/sas/gluster/data/code-ide):1136:connect] GLUSTER: Mounted gluster volume duration=1.1220
[2020-05-22 12:09:39.906748] I [subcmds(worker /home/sas/gluster/data/code-ide):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2020-05-22 12:09:42.84068] I [master(worker /home/sas/gluster/data/code-ide):1640:register] _GMaster: Working dir path=/home/sas/gluster/internals/misc/gluster/gsyncd/code-ide_10.37.11.252_code-ide/home-sas-gluster-data-code-ide
[2020-05-22 12:09:42.84353] I [resource(worker /home/sas/gluster/data/code-ide):1299:service_loop] GLUSTER: Register time time=1590149382
[2020-05-22 12:09:42.88851] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):281:set_active] GeorepStatus: Worker Status Change status=Active
[2020-05-22 12:09:42.149405] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl
[2020-05-22 12:09:42.149740] I [master(worker /home/sas/gluster/data/code-ide):1554:crawl] _GMaster: starting history crawl turns=1 stime=(1588918750, 0) entry_stime=(1588918779, 0) etime=1590149382
[2020-05-22 12:09:43.163168] I [master(worker /home/sas/gluster/data/code-ide):1583:crawl] _GMaster: slave's time stime=(1588918750, 0)
[2020-05-22 12:09:43.838727] I [master(worker /home/sas/gluster/data/code-ide):1991:syncjob] Syncer: Sync Time Taken duration=0.4240 num_files=1 job=1 return_code=0
[2020-05-22 12:09:43.944392] E [repce(worker /home/sas/gluster/data/code-ide):214:__call__] RepceClient: call failed call=261471:140535761106752:1590149383.8 method=entry_ops error=OSError
[2020-05-22 12:09:43.944746] E [syncdutils(worker /home/sas/gluster/data/code-ide):338:log_raise_exception] <top>: FAIL:
Traceback (most recent call last):
File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 325, in main
func(args)
File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 86, in subcmd_worker
local.service_loop(remote)
File "/usr/libexec/glusterfs/python/syncdaemon/resource.py", line 1305, in service_loop
g3.crawlwrap(oneshot=True)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 602, in crawlwrap
self.crawl()
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1592, in crawl
self.changelogs_batch_process(changes)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1492, in changelogs_batch_process
self.process(batch)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in process
self.process_change(change, done, retry)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in process_change
failures = self.slave.server.entry_ops(entries)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__
return self.ins(self.meth, *a)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__
raise res
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 12:09:43.968710] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.
[2020-05-22 12:09:44.912470] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-ide
[2020-05-22 12:09:44.913692] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty
^C
sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>
sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>date
Fri May 22 05:09:59 PDT 2020
sas at 10.47.8.153/var/log/glusterfs/geo-replication/code-ide_10.37.11.252_code-ide>tail -f gsyncd.log
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1327, in process
self.process_change(change, done, retry)
File "/usr/libexec/glusterfs/python/syncdaemon/master.py", line 1221, in process_change
failures = self.slave.server.entry_ops(entries)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 233, in __call__
return self.ins(self.meth, *a)
File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 215, in __call__
raise res
OSError: [Errno 13] Permission denied: '/home/sas/gluster/data/code-ide/.glusterfs/c2/bf/c2bff066-b10e-468a-a67e-b8b501a8951e'
[2020-05-22 12:10:02.710613] I [repce(agent /home/sas/gluster/data/code-ide):97:service_loop] RepceServer: terminating on reaching EOF.
[2020-05-22 12:10:03.681055] I [monitor(monitor):280:monitor] Monitor: worker died in startup phase brick=/home/sas/gluster/data/code-ide
[2020-05-22 12:10:03.682048] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Faulty
[2020-05-22 12:10:14.31925] I [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker Status Change status=Initializing...
[2020-05-22 12:10:14.32105] I [monitor(monitor):159:monitor] Monitor: starting gsyncd worker brick=/home/sas/gluster/data/code-ide slave_node=10.37.11.246
[2020-05-22 12:10:14.76786] I [gsyncd(agent /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 12:10:14.81006] I [changelogagent(agent /home/sas/gluster/data/code-ide):72:__init__] ChangelogAgent: Agent listining...
[2020-05-22 12:10:14.91418] I [gsyncd(worker /home/sas/gluster/data/code-ide):311:main] <top>: Using session config file path=/var/lib/glusterd/geo-replication/code-ide_10.37.11.252_code-ide/gsyncd.conf
[2020-05-22 12:10:14.109091] I [resource(worker /home/sas/gluster/data/code-ide):1394:connect_remote] SSH: Initializing SSH connection between master and slave...
[2020-05-22 12:10:16.334167] I [resource(worker /home/sas/gluster/data/code-ide):1443:connect_remote] SSH: SSH connection between master and slave established. duration=2.2248
[2020-05-22 12:10:16.334379] I [resource(worker /home/sas/gluster/data/code-ide):1113:connect] GLUSTER: Mounting gluster volume locally...
[2020-05-22 12:10:17.444447] I [resource(worker /home/sas/gluster/data/code-ide):1136:connect] GLUSTER: Mounted gluster volume duration=1.1099
[2020-05-22 12:10:17.444658] I [subcmds(worker /home/sas/gluster/data/code-ide):84:subcmd_worker] <top>: Worker spawn successful. Acknowledging back to monitor
[2020-05-22 12:10:19.612169] I [master(worker /home/sas/gluster/data/code-ide):1640:register] _GMaster: Working dir path=/home/sas/gluster/internals/misc/gluster/gsyncd/code-ide_10.37.11.252_code-ide/home-sas-gluster-data-code-ide
[2020-05-22 12:10:19.612401] I [resource(worker /home/sas/gluster/data/code-ide):1299:service_loop] GLUSTER: Register time time=1590149419
[2020-05-22 12:10:19.616991] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):281:set_active] GeorepStatus: Worker Status Change status=Active
[2020-05-22 12:10:19.669262] I [gsyncdstatus(worker /home/sas/gluster/data/code-ide):253:set_worker_crawl_status] GeorepStatus: Crawl Status Change status=History Crawl
[2020-05-22 12:10:19.669564] I [master(worker /home/sas/gluster/data/code-ide):1554:crawl] _GMaster: starting history crawl turns=1 stime=(1588918750, 0) entry_stime=(1588918779, 0) etime=1590149419
More information about the Gluster-users
mailing list