[2017-08-22 13:45:33.082336] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.11 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/45a71b11283e2887e6df8e37c178a760.socket --xlator-option *replicate*.node-uuid=5f4ccbf4-33f6-4298-8b31-213553223349)
[2017-08-22 13:45:33.089861] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-08-22 13:45:33.089932] E [socket.c:2309:socket_connect_finish] 0-glusterfs: connection to ::1:24007 failed (Connection refused)
[2017-08-22 13:45:33.089981] E [glusterfsd-mgmt.c:1908:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport endpoint is not connected)
[2017-08-22 13:45:37.090406] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-othervol-replicate-0: adding option 'node-uuid' for volume 'othervol-replicate-0' with value '5f4ccbf4-33f6-4298-8b31-213553223349'
[2017-08-22 13:45:37.090446] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-myvol-private-replicate-0: adding option 'node-uuid' for volume 'myvol-private-replicate-0' with value '5f4ccbf4-33f6-4298-8b31-213553223349'
[2017-08-22 13:45:37.090468] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-myvol-pro-replicate-0: adding option 'node-uuid' for volume 'myvol-pro-replicate-0' with value '5f4ccbf4-33f6-4298-8b31-213553223349'
[2017-08-22 13:45:37.093963] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2017-08-22 13:45:37.094055] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4
[2017-08-22 13:45:37.094059] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2017-08-22 13:45:37.104141] I [MSGID: 114020] [client.c:2356:notify] 0-othervol-client-0: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:37.104779] I [MSGID: 114020] [client.c:2356:notify] 0-othervol-client-1: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:37.104905] I [MSGID: 101191] [event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 3
[2017-08-22 13:45:37.105107] I [MSGID: 101191] [event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 4
[2017-08-22 13:45:37.105167] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-othervol-client-0: changing port to 49160 (from 0)
[2017-08-22 13:45:37.105765] I [MSGID: 114020] [client.c:2356:notify] 0-othervol-client-2: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:37.106384] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-private-client-0: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:37.106458] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-othervol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:37.106735] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-othervol-client-1: changing port to 49162 (from 0)
[2017-08-22 13:45:37.106992] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-private-client-1: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:37.107409] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-othervol-client-2: changing port to 49155 (from 0)
[2017-08-22 13:45:37.107493] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-othervol-client-0: Connected to othervol-client-0, attached to remote volume '/data/othervol/brick'.
[2017-08-22 13:45:37.107520] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-othervol-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:37.107673] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-othervol-replicate-0: Subvolume 'othervol-client-0' came back up; going online.
[2017-08-22 13:45:37.107709] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-private-client-2: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:37.108392] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-othervol-client-0: Server lk version = 1
[2017-08-22 13:45:37.108589] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-othervol-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:37.108648] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-pro-client-0: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:37.108713] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-private-client-0: changing port to 49161 (from 0)
[2017-08-22 13:45:37.108784] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-othervol-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:37.109217] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-pro-client-1: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:37.109487] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-private-client-1: changing port to 49163 (from 0)
[2017-08-22 13:45:37.109598] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-othervol-client-1: Connected to othervol-client-1, attached to remote volume '/data/othervol/brick'.
[2017-08-22 13:45:37.109626] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-othervol-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:37.109713] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-othervol-client-2: Connected to othervol-client-2, attached to remote volume '/srv/glusterfs/othervol/brick'.
[2017-08-22 13:45:37.109738] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-othervol-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:37.109843] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-pro-client-2: parent translators are ready, attempting connect on transport
Final graph:
+------------------------------------------------------------------------------+
  1: volume othervol-client-0
  2:     type protocol/client
  3:     option clnt-lk-version 1
  4:     option volfile-checksum 0
  5:     option volfile-key gluster/glustershd
  6:     option client-version 3.8.11
  7:     option process-uuid gfs1a-29019-2017/08/22-13:45:33:75672-othervol-client-0-0-0
  8:     option fops-version 1298437
  9:     option ping-timeout 42
 10:     option remote-host gfs1a.domain.tld
 11:     option remote-subvolume /data/othervol/brick
 12:     option transport-type socket
 13:     option transport.address-family inet
 14:     option username f9e3bdde-ca87-4d66-99f1-cd4467381566
 15:     option password ff0ef273-bcf7-41a3-acee-fedce0d9c831
 16:     option event-threads 4
 17: end-volume
 18:
 19: volume othervol-client-1
 20:     type protocol/client
 21:     option clnt-lk-version 1
 22:     option volfile-checksum 0
 23:     option volfile-key gluster/glustershd
 24:     option client-version 3.8.11
 25:     option process-uuid gfs1a-29019-2017/08/22-13:45:33:75672-othervol-client-1-0-0
 26:     option fops-version 1298437
 27:     option ping-timeout 42
 28:     option remote-host gfs1b.domain.tld
 29:     option remote-subvolume /data/othervol/brick
 30:     option transport-type socket
 31:     option transport.address-family inet
 32:     option username f9e3bdde-ca87-4d66-99f1-cd4467381566
 33:     option password ff0ef273-bcf7-41a3-acee-fedce0d9c831
 34:     option event-threads 4
 35: end-volume
 36:
 37: volume othervol-client-2
 38:     type protocol/client
 39:     option clnt-lk-version 1
 40:     option volfile-checksum 0
 41:     option volfile-key gluster/glustershd
 42:     option client-version 3.8.11
 43:     option process-uuid gfs1a-29019-2017/08/22-13:45:33:75672-othervol-client-2-0-0
 44:     option fops-version 1298437
 45:     option ping-timeout 42
 46:     option remote-host gfs1c.domain.tld
 47:     option remote-subvolume /srv/glusterfs/othervol/brick
 48:     option transport-type socket
[2017-08-22 13:45:37.110750] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-private-client-2: changing port to 49157 (from 0)
 49:     option transport.address-family inet
 50:     option username f9e3bdde-ca87-4d66-99f1-cd4467381566
 51:     option password ff0ef273-bcf7-41a3-acee-fedce0d9c831
 52:     option event-threads 4
 53: end-volume
 54:
 55: volume othervol-replicate-0
 56:     type cluster/replicate
[2017-08-22 13:45:37.111169] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-othervol-client-1: Server lk version = 1
 57:     option node-uuid 5f4ccbf4-33f6-4298-8b31-213553223349
 58:     option arbiter-count 1
 59:     option background-self-heal-count 0
 60:     option metadata-self-heal on
 61:     option data-self-heal on
 62:     option entry-self-heal on
 63:     option self-heal-daemon enable
 64:     option iam-self-heal-daemon yes
 65:     subvolumes othervol-client-0 othervol-client-1 othervol-client-2
 66: end-volume
 67:
 68: volume myvol-private-client-0
 69:     type protocol/client
 70:     option ping-timeout 42
 71:     option remote-host gfs1a.domain.tld
[2017-08-22 13:45:37.111317] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-othervol-client-2: Server lk version = 1
 72:     option remote-subvolume /data/myvol-private/brick
 73:     option transport-type socket
 74:     option username bea3e634-e174-4bb3-a1d6-25b09d03b536
 75:     option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6
 76:     option event-threads 4
 77: end-volume
 78:
 79: volume myvol-private-client-1
 80:     type protocol/client
 81:     option ping-timeout 42
 82:     option remote-host gfs1b.domain.tld
 83:     option remote-subvolume /data/myvol-private/brick
 84:     option transport-type socket
 85:     option username bea3e634-e174-4bb3-a1d6-25b09d03b536
[2017-08-22 13:45:37.111547] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-private-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
 86:     option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6
 87:     option event-threads 4
 88: end-volume
 89:
 90: volume myvol-private-client-2
 91:     type protocol/client
 92:     option ping-timeout 42
 93:     option remote-host gfs1c.domain.tld
 94:     option remote-subvolume /srv/glusterfs/myvol-private/brick
 95:     option transport-type socket
 96:     option username bea3e634-e174-4bb3-a1d6-25b09d03b536
 97:     option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6
 98:     option event-threads 4
 99: end-volume
100:
101: volume myvol-private-replicate-0
102:     type cluster/replicate
103:     option node-uuid 5f4ccbf4-33f6-4298-8b31-213553223349
104:     option arbiter-count 1
105:     option background-self-heal-count 0
106:     option metadata-self-heal on
107:     option data-self-heal on
108:     option entry-self-heal on
109:     option self-heal-daemon enable
110:     option iam-self-heal-daemon yes
111:     subvolumes myvol-private-client-0 myvol-private-client-1 myvol-private-client-2
112: end-volume
113:
114: volume myvol-pro-client-0
115:     type protocol/client
116:     option ping-timeout 42
117:     option remote-host gfs1a.domain.tld
118:     option remote-subvolume /data/myvol-pro/brick
119:     option transport-type socket
120:     option username c53c23ad-b53c-470c-841d-85e33e052814
121:     option password 7c16ba18-47e3-47a9-b770-8ce33df8f68c
122: end-volume
123:
124: volume myvol-pro-client-1
125:     type protocol/client
126:     option ping-timeout 42
127:     option remote-host gfs1b.domain.tld
128:     option remote-subvolume /data/myvol-pro/brick
129:     option transport-type socket
130:     option username c53c23ad-b53c-470c-841d-85e33e052814
131:     option password 7c16ba18-47e3-47a9-b770-8ce33df8f68c
132: end-volume
133:
134: volume myvol-pro-client-2
135:     type protocol/client
136:     option ping-timeout 42
137:     option remote-host gfs1c.domain.tld
138:     option remote-subvolume /srv/glusterfs/myvol-pro/brick
139:     option transport-type socket
140:     option username c53c23ad-b53c-470c-841d-85e33e052814
141:     option password 7c16ba18-47e3-47a9-b770-8ce33df8f68c
142: end-volume
143:
144: volume myvol-pro-replicate-0
145:     type cluster/replicate
146:     option node-uuid 5f4ccbf4-33f6-4298-8b31-213553223349
147:     option arbiter-count 1
148:     option background-self-heal-count 0
149:     option metadata-self-heal on
150:     option data-self-heal on
151:     option entry-self-heal on
152:     option self-heal-daemon enable
153:     option iam-self-heal-daemon yes
154:     subvolumes myvol-pro-client-0 myvol-pro-client-1 myvol-pro-client-2
155: end-volume
156:
157: volume glustershd
158:     type debug/io-stats
159:     option log-level INFO
160:     subvolumes othervol-replicate-0 myvol-private-replicate-0 myvol-pro-replicate-0
161: end-volume
162:
+------------------------------------------------------------------------------+
[2017-08-22 13:45:37.112408] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-pro-client-0: changing port to 49162 (from 0)
[2017-08-22 13:45:37.112478] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-private-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:37.112575] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-private-client-0: Connected to myvol-private-client-0, attached to remote volume '/data/myvol-private/brick'.
[2017-08-22 13:45:37.112610] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-private-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:37.112703] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-myvol-private-replicate-0: Subvolume 'myvol-private-client-0' came back up; going online.
[2017-08-22 13:45:37.112785] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-pro-client-1: changing port to 49164 (from 0)
[2017-08-22 13:45:37.112825] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-pro-client-2: changing port to 49156 (from 0)
[2017-08-22 13:45:37.112859] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-private-client-0: Server lk version = 1
[2017-08-22 13:45:37.113550] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-private-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:37.113683] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-private-client-1: Connected to myvol-private-client-1, attached to remote volume '/data/myvol-private/brick'.
[2017-08-22 13:45:37.113709] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-private-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:37.113926] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-pro-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:37.113919] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-private-client-1: Server lk version = 1
[2017-08-22 13:45:37.114060] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-pro-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:37.114140] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-private-client-2: Connected to myvol-private-client-2, attached to remote volume '/srv/glusterfs/myvol-private/brick'.
[2017-08-22 13:45:37.114215] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-private-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:37.114218] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-pro-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:37.114465] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-private-client-2: Server lk version = 1
[2017-08-22 13:45:37.114545] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-pro-client-0: Connected to myvol-pro-client-0, attached to remote volume '/data/myvol-pro/brick'.
[2017-08-22 13:45:37.114588] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-pro-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:37.114596] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-pro-client-1: Connected to myvol-pro-client-1, attached to remote volume '/data/myvol-pro/brick'.
[2017-08-22 13:45:37.114661] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-pro-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:37.114722] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-myvol-pro-replicate-0: Subvolume 'myvol-pro-client-0' came back up; going online.
[2017-08-22 13:45:37.114832] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-pro-client-2: Connected to myvol-pro-client-2, attached to remote volume '/srv/glusterfs/myvol-pro/brick'.
[2017-08-22 13:45:37.114860] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-pro-client-0: Server lk version = 1
[2017-08-22 13:45:37.114896] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-pro-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:37.114967] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-pro-client-1: Server lk version = 1
[2017-08-22 13:45:37.115172] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-pro-client-2: Server lk version = 1
[2017-08-24 09:48:45.276906] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2017-08-24 09:48:45.354990] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2017-08-24 09:48:45.365659] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-othervol-replicate-0: adding option 'node-uuid' for volume 'othervol-replicate-0' with value '5f4ccbf4-33f6-4298-8b31-213553223349'
[2017-08-24 09:48:45.365689] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-myvol-private-replicate-0: adding option 'node-uuid' for volume 'myvol-private-replicate-0' with value '5f4ccbf4-33f6-4298-8b31-213553223349'
[2017-08-24 09:48:45.365707] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-myvol-pro-replicate-0: adding option 'node-uuid' for volume 'myvol-pro-replicate-0' with value '5f4ccbf4-33f6-4298-8b31-213553223349'
[2017-08-24 09:48:45.366865] D [io-stats.c:3574:reconfigure] 0-glustershd: reconfigure returning 0
[2017-08-24 09:48:45.366894] D [glusterfsd-mgmt.c:1628:mgmt_getspec_cbk] 0-glusterfsd-mgmt: No need to re-load volfile, reconfigure done
[2017-08-24 09:48:45.368142] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2017-08-24 09:49:50.218681] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present
[2017-08-24 09:49:50.218747] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
[2017-08-24 09:50:01.000541] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f6273504a63] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7f62732d0747] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13d6c)[0x7f62732d0d6c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29f)[0x7f62732cda9f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xf4ad)[0x7f626d2994ad] ))))) 0-: 127.0.1.1:49162: ping timer event already removed
[2017-08-24 09:50:01.000575] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f6273504a63] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7f62732d0747] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13d6c)[0x7f62732d0d6c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29f)[0x7f62732cda9f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xf4ad)[0x7f626d2994ad] ))))) 0-: 127.0.1.1:49160: ping timer event already removed
[2017-08-24 09:50:01.000689] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-pro-client-0: subvol myvol-pro-client-0 is local
[2017-08-24 09:50:01.000719] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-othervol-client-0: subvol othervol-client-0 is local
[2017-08-24 09:50:01.000720] D [MSGID: 0] [afr-self-heald.c:589:afr_shd_index_healer] 0-myvol-pro-replicate-0: starting index sweep on subvol myvol-pro-client-0
[2017-08-24 09:50:01.000772] D [MSGID: 0] [afr-self-heald.c:589:afr_shd_index_healer] 0-othervol-replicate-0: starting index sweep on subvol othervol-client-0
[2017-08-24 09:50:01.000972] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_index_gfid dir gfid for myvol-pro-client-0: 0a06c686-bcf4-43f5-bcd6-7a7666f77275
[2017-08-24 09:50:01.001002] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-othervol-replicate-0: glusterfs.xattrop_index_gfid dir gfid for othervol-client-0: 1ea2c5f3-9ae7-4825-89a5-d114022bfc65
[2017-08-24 09:50:01.005069] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_dirty_gfid dir gfid for myvol-pro-client-0: 860ef039-30f8-42cf-8ba9-e563acb0e01a
[2017-08-24 09:50:01.005071] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-othervol-replicate-0: glusterfs.xattrop_dirty_gfid dir gfid for othervol-client-0: 66fc9eea-e9ed-4b5d-b973-0236f0725f0e
[2017-08-24 09:50:01.008136] D [MSGID: 0] [afr-self-heald.c:432:afr_shd_index_heal] 0-myvol-pro-replicate-0: got entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
[2017-08-24 09:50:01.008820] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f6273504a63] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7f62732d0747] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13d6c)[0x7f62732d0d6c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29f)[0x7f62732cda9f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xf4ad)[0x7f626d2994ad] ))))) 0-: 192.168.10.33:49164: ping timer event already removed
[2017-08-24 09:50:01.009073] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f6273504a63] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7f62732d0747] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13d6c)[0x7f62732d0d6c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29f)[0x7f62732cda9f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xf4ad)[0x7f626d2994ad] ))))) 0-: 192.168.10.35:49156: ping timer event already removed
[2017-08-24 09:50:01.009130] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-othervol-replicate-0: glusterfs.xattrop_entry_changes_gfid dir gfid for othervol-client-0: ea81dd8d-755a-4453-9303-64e84e83f49a
[2017-08-24 09:50:01.012574] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_entry_changes_gfid dir gfid for myvol-pro-client-0: 762a5802-ceca-4b3a-a757-490115f8ec98
[2017-08-24 09:50:01.012848] D [MSGID: 0] [afr-self-heald.c:606:afr_shd_index_healer] 0-othervol-replicate-0: finished index sweep on subvol othervol-client-0
[2017-08-24 09:50:01.016027] D [MSGID: 0] [afr-self-heald.c:606:afr_shd_index_healer] 0-myvol-pro-replicate-0: finished index sweep on subvol myvol-pro-client-0
[2017-08-24 09:50:04.000428] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f6273504a63] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7f62732d0747] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13d6c)[0x7f62732d0d6c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29f)[0x7f62732cda9f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xf4ad)[0x7f626d2994ad] ))))) 0-: 127.0.1.1:49161: ping timer event already removed
[2017-08-24 09:50:04.000581] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-private-client-0: subvol myvol-private-client-0 is local
[2017-08-24 09:50:04.000616] D [MSGID: 0] [afr-self-heald.c:589:afr_shd_index_healer] 0-myvol-private-replicate-0: starting index sweep on subvol myvol-private-client-0
[2017-08-24 09:50:04.000858] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-private-replicate-0: glusterfs.xattrop_index_gfid dir gfid for myvol-private-client-0: 00dabbf1-79a9-4a44-b483-b1621cb79848
[2017-08-24 09:50:04.041820] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-private-replicate-0: glusterfs.xattrop_dirty_gfid dir gfid for myvol-private-client-0: f274bc66-7cf6-42a2-bf51-889149c538ed
[2017-08-24 09:50:04.044972] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-private-replicate-0: glusterfs.xattrop_entry_changes_gfid dir gfid for myvol-private-client-0: 6ec2f9db-d96f-4e6c-b583-c63ad29c8abd
[2017-08-24 09:50:04.047792] D [MSGID: 0] [afr-self-heald.c:606:afr_shd_index_healer] 0-myvol-private-replicate-0: finished index sweep on subvol myvol-private-client-0
[2017-08-24 09:50:43.224388] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:50:43.224440] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-othervol-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:50:43.224455] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:50:43.224467] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:50:46.224779] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-private-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:51:50.231620] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present
[2017-08-24 09:51:50.231686] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
[2017-08-24 09:52:26.301966] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f6273504a63] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7f62732d0747] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13d6c)[0x7f62732d0d6c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29f)[0x7f62732cda9f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xf4ad)[0x7f626d2994ad] ))))) 0-: 127.0.1.1:49162: ping timer event already removed
[2017-08-24 09:52:26.302361] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-pro-client-0: subvol myvol-pro-client-0 is local
[2017-08-24 09:52:26.302669] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f6273504a63] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7f62732d0747] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13d6c)[0x7f62732d0d6c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29f)[0x7f62732cda9f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xf4ad)[0x7f626d2994ad] ))))) 0-: 192.168.10.33:49164: ping timer event already removed
[2017-08-24 09:52:26.302692] D [MSGID: 0] [afr-self-heald.c:589:afr_shd_index_healer] 0-myvol-pro-replicate-0: starting index sweep on subvol myvol-pro-client-0
[2017-08-24 09:52:26.302964] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_index_gfid dir gfid for myvol-pro-client-0: 0a06c686-bcf4-43f5-bcd6-7a7666f77275
[2017-08-24 09:52:26.303076] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-pro-client-1: subvol myvol-pro-client-1 is local
[2017-08-24 09:52:26.303354] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7f6273504a63] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7f62732d0747] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13d6c)[0x7f62732d0d6c] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29f)[0x7f62732cda9f] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xf4ad)[0x7f626d2994ad] ))))) 0-: 192.168.10.35:49156: ping timer event already removed
[2017-08-24 09:52:26.303821] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-pro-client-2: subvol myvol-pro-client-2 is local
[2017-08-24 09:52:26.305618] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_dirty_gfid dir gfid for myvol-pro-client-0: 860ef039-30f8-42cf-8ba9-e563acb0e01a
[2017-08-24 09:52:26.307369] D [MSGID: 0] [afr-self-heald.c:432:afr_shd_index_heal] 0-myvol-pro-replicate-0: got entry: 16cf5457-fcda-40be-a982-c4307dffcda2
[2017-08-24 09:52:26.308440] D [MSGID: 0] [afr-self-heal-common.c:1886:afr_selfheal_unlocked_inspect] 0-myvol-pro-replicate-0: SIZE mismatch 8192 vs 0 on myvol-pro-client-2 for gfid:16cf5457-fcda-40be-a982-c4307dffcda2
[2017-08-24 09:52:26.309535] D [MSGID: 114031] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-myvol-pro-client-1: remote operation failed [Resource temporarily unavailable]
[2017-08-24 09:52:26.309533] D [MSGID: 114031] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-myvol-pro-client-0: remote operation failed [Resource temporarily unavailable]
[2017-08-24 09:52:26.309589] D [MSGID: 0] [client-rpc-fops.c:1548:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f6270e0e498, myvol-pro-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable]
[2017-08-24 09:52:26.309667] D [MSGID: 114031] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-myvol-pro-client-2: remote operation failed [Resource temporarily unavailable]
[2017-08-24 09:52:26.309703] D [MSGID: 0] [client-rpc-fops.c:1548:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f6270e0e498, myvol-pro-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable]
[2017-08-24 09:52:26.309855] D [MSGID: 0] [client-rpc-fops.c:1548:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7f6270e0e498, myvol-pro-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable]
[2017-08-24 09:52:27.491001] D [MSGID: 0] [afr-self-heald.c:432:afr_shd_index_heal] 0-myvol-pro-replicate-0: got entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
[2017-08-24 09:52:27.494268] D [MSGID: 0] [afr-self-heald.c:432:afr_shd_index_heal] 0-myvol-pro-replicate-0: got entry: 6167bf72-c9a2-476d-a41b-96ca1507e2f5
[2017-08-24 09:52:27.495304] D [MSGID: 0] [afr-self-heal-common.c:1886:afr_selfheal_unlocked_inspect] 0-myvol-pro-replicate-0: SIZE mismatch 13752 vs 0 on myvol-pro-client-2 for gfid:6167bf72-c9a2-476d-a41b-96ca1507e2f5
[2017-08-24 09:52:27.498031] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_entry_changes_gfid dir gfid for myvol-pro-client-0: 762a5802-ceca-4b3a-a757-490115f8ec98
[2017-08-24 09:52:27.501469] D [MSGID: 0] [afr-self-heald.c:606:afr_shd_index_healer] 0-myvol-pro-replicate-0: finished index sweep on subvol myvol-pro-client-0
[2017-08-24 09:53:09.239860] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:53:09.239920] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:53:09.239936] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
