[2017-08-22 13:45:34.113843] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.11 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/lib/glusterd/glustershd/run/glustershd.pid -l /var/log/glusterfs/glustershd.log -S /var/run/gluster/00be81b9852533e4fed4a6ecab7bfe60.socket --xlator-option *replicate*.node-uuid=60a03a81-ba92-4b84-90fe-7b6e35a10975)
[2017-08-22 13:45:34.122215] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-08-22 13:45:34.124808] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-othervol-replicate-0: adding option 'node-uuid' for volume 'othervol-replicate-0' with value '60a03a81-ba92-4b84-90fe-7b6e35a10975'
[2017-08-22 13:45:34.124830] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-myvol-private-replicate-0: adding option 'node-uuid' for volume 'myvol-private-replicate-0' with value '60a03a81-ba92-4b84-90fe-7b6e35a10975'
[2017-08-22 13:45:34.124841] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-myvol-pro-replicate-0: adding option 'node-uuid' for volume 'myvol-pro-replicate-0' with value '60a03a81-ba92-4b84-90fe-7b6e35a10975'
[2017-08-22 13:45:34.128701] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2
[2017-08-22 13:45:34.128769] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 3
[2017-08-22 13:45:34.128819] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 4
[2017-08-22 13:45:34.139347] I [MSGID: 114020] [client.c:2356:notify] 0-othervol-client-0: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:34.140275] I [MSGID: 114020] [client.c:2356:notify] 0-othervol-client-1: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:34.140478] I [MSGID: 101191] [event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 4
[2017-08-22 13:45:34.140664] I [MSGID: 114020] [client.c:2356:notify] 0-othervol-client-2: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:34.140698] I [MSGID: 101191] [event-epoll.c:659:event_dispatch_epoll_worker] 0-epoll: Exited thread with index 3
[2017-08-22 13:45:34.140808] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-othervol-client-0: changing port to 49160 (from 0)
[2017-08-22 13:45:34.140922] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-private-client-0: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:34.141305] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-private-client-1: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:34.141616] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-othervol-client-2: changing port to 49155 (from 0)
[2017-08-22 13:45:34.141639] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-private-client-2: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:34.141887] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-pro-client-0: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:34.141939] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-othervol-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:34.142034] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-private-client-0: changing port to 49161 (from 0)
[2017-08-22 13:45:34.142337] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-pro-client-1: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:34.142570] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-othervol-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:34.142674] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-othervol-client-0: Connected to othervol-client-0, attached to remote volume '/data/othervol/brick'.
[2017-08-22 13:45:34.142692] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-othervol-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:34.142755] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-othervol-replicate-0: Subvolume 'othervol-client-0' came back up; going online.
[2017-08-22 13:45:34.142807] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-private-client-2: changing port to 49157 (from 0)
[2017-08-22 13:45:34.142865] I [MSGID: 114020] [client.c:2356:notify] 0-myvol-pro-client-2: parent translators are ready, attempting connect on transport
[2017-08-22 13:45:34.143128] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-othervol-client-0: Server lk version = 1
[2017-08-22 13:45:34.143163] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-othervol-client-2: Connected to othervol-client-2, attached to remote volume '/srv/glusterfs/othervol/brick'.
[2017-08-22 13:45:34.143176] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-othervol-client-2: Server and Client lk-version numbers are not same, reopening the fds
Final graph:
+------------------------------------------------------------------------------+
  1: volume othervol-client-0
  2:     type protocol/client
  3:     option clnt-lk-version 1
  4:     option volfile-checksum 0
  5:     option volfile-key gluster/glustershd
  6:     option client-version 3.8.11
  7:     option process-uuid gfs1c-1289-2017/08/22-13:45:34:104854-othervol-client-0-0-0
  8:     option fops-version 1298437
  9:     option ping-timeout 42
[2017-08-22 13:45:34.143271] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-private-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
 10:     option remote-host gfs1a.domain.tld
 11:     option remote-subvolume /data/othervol/brick
 12:     option transport-type socket
 13:     option transport.address-family inet
 14:     option username f9e3bdde-ca87-4d66-99f1-cd4467381566
 15:     option password ff0ef273-bcf7-41a3-acee-fedce0d9c831
 16:     option event-threads 4
 17: end-volume
 18:
 19: volume othervol-client-1
[2017-08-22 13:45:34.143324] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-pro-client-0: changing port to 49162 (from 0)
 20:     type protocol/client
 21:     option ping-timeout 42
 22:     option remote-host gfs1b.domain.tld
 23:     option remote-subvolume /data/othervol/brick
 24:     option transport-type socket
 25:     option transport.address-family inet
 26:     option username f9e3bdde-ca87-4d66-99f1-cd4467381566
 27:     option password ff0ef273-bcf7-41a3-acee-fedce0d9c831
 28:     option event-threads 4
 29: end-volume
 30:
 31: volume othervol-client-2
 32:     type protocol/client
 33:     option clnt-lk-version 1
 34:     option volfile-checksum 0
 35:     option volfile-key gluster/glustershd
 36:     option client-version 3.8.11
 37:     option process-uuid gfs1c-1289-2017/08/22-13:45:34:104854-othervol-client-2-0-0
 38:     option fops-version 1298437
 39:     option ping-timeout 42
[2017-08-22 13:45:34.143469] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-othervol-client-2: Server lk version = 1
 40:     option remote-host gfs1c.domain.tld
 41:     option remote-subvolume /srv/glusterfs/othervol/brick
 42:     option transport-type socket
 43:     option transport.address-family inet
 44:     option username f9e3bdde-ca87-4d66-99f1-cd4467381566
 45:     option password ff0ef273-bcf7-41a3-acee-fedce0d9c831
 46:     option event-threads 4
 47: end-volume
 48:
 49: volume othervol-replicate-0
 50:     type cluster/replicate
 51:     option node-uuid 60a03a81-ba92-4b84-90fe-7b6e35a10975
 52:     option arbiter-count 1
 53:     option background-self-heal-count 0
 54:     option metadata-self-heal on
 55:     option data-self-heal on
 56:     option entry-self-heal on
 57:     option self-heal-daemon enable
 58:     option iam-self-heal-daemon yes
 59:     subvolumes othervol-client-0 othervol-client-1 othervol-client-2
 60: end-volume
 61:
 62: volume myvol-private-client-0
 63:     type protocol/client
 64:     option clnt-lk-version 1
 65:     option volfile-checksum 0
 66:     option volfile-key gluster/glustershd
 67:     option client-version 3.8.11
 68:     option process-uuid gfs1c-1289-2017/08/22-13:45:34:104854-myvol-private-client-0-0-0
 69:     option fops-version 1298437
 70:     option ping-timeout 42
 71:     option remote-host gfs1a.domain.tld
 72:     option remote-subvolume /data/myvol-private/brick
 73:     option transport-type socket
 74:     option username bea3e634-e174-4bb3-a1d6-25b09d03b536
 75:     option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6
 76:     option event-threads 4
 77: end-volume
 78:
 79: volume myvol-private-client-1
 80:     type protocol/client
 81:     option ping-timeout 42
 82:     option remote-host gfs1b.domain.tld
 83:     option remote-subvolume /data/myvol-private/brick
 84:     option transport-type socket
 85:     option username bea3e634-e174-4bb3-a1d6-25b09d03b536
 86:     option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6
 87:     option event-threads 4
 88: end-volume
 89:
 90: volume myvol-private-client-2
 91:     type protocol/client
 92:     option ping-timeout 42
 93:     option remote-host gfs1c.domain.tld
 94:     option remote-subvolume /srv/glusterfs/myvol-private/brick
 95:     option transport-type socket
 96:     option username bea3e634-e174-4bb3-a1d6-25b09d03b536
 97:     option password 3a6f98bd-795e-4ec4-adfe-42c61ccfa0a6
 98:     option event-threads 4
 99: end-volume
100:
101: volume myvol-private-replicate-0
102:     type cluster/replicate
103:     option node-uuid 60a03a81-ba92-4b84-90fe-7b6e35a10975
104:     option arbiter-count 1
105:     option background-self-heal-count 0
106:     option metadata-self-heal on
[2017-08-22 13:45:34.143912] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-private-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
107:     option data-self-heal on
108:     option entry-self-heal on
109:     option self-heal-daemon enable
110:     option iam-self-heal-daemon yes
111:     subvolumes myvol-private-client-0 myvol-private-client-1 myvol-private-client-2
112: end-volume
113:
114: volume myvol-pro-client-0
115:     type protocol/client
116:     option ping-timeout 42
117:     option remote-host gfs1a.domain.tld
118:     option remote-subvolume /data/myvol-pro/brick
119:     option transport-type socket
120:     option username c53c23ad-b53c-470c-841d-85e33e052814
121:     option password 7c16ba18-47e3-47a9-b770-8ce33df8f68c
[2017-08-22 13:45:34.144058] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-private-client-0: Connected to myvol-private-client-0, attached to remote volume '/data/myvol-private/brick'.
122: end-volume
[2017-08-22 13:45:34.144087] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-private-client-0: Server and Client lk-version numbers are not same, reopening the fds
123:
124: volume myvol-pro-client-1
125:     type protocol/client
126:     option ping-timeout 42
127:     option remote-host gfs1b.domain.tld
128:     option remote-subvolume /data/myvol-pro/brick
129:     option transport-type socket
130:     option username c53c23ad-b53c-470c-841d-85e33e052814
131:     option password 7c16ba18-47e3-47a9-b770-8ce33df8f68c
132: end-volume
[2017-08-22 13:45:34.144138] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-myvol-private-replicate-0: Subvolume 'myvol-private-client-0' came back up; going online.
133:
134: volume myvol-pro-client-2
135:     type protocol/client
136:     option ping-timeout 42
137:     option remote-host gfs1c.domain.tld
138:     option remote-subvolume /srv/glusterfs/myvol-pro/brick
139:     option transport-type socket
[2017-08-22 13:45:34.144223] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-pro-client-2: changing port to 49156 (from 0)
140:     option username c53c23ad-b53c-470c-841d-85e33e052814
[2017-08-22 13:45:34.144297] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-private-client-2: Connected to myvol-private-client-2, attached to remote volume '/srv/glusterfs/myvol-private/brick'.
141:     option password 7c16ba18-47e3-47a9-b770-8ce33df8f68c
142: end-volume
[2017-08-22 13:45:34.144326] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-private-client-2: Server and Client lk-version numbers are not same, reopening the fds
143:
144: volume myvol-pro-replicate-0
145:     type cluster/replicate
146:     option node-uuid 60a03a81-ba92-4b84-90fe-7b6e35a10975
147:     option arbiter-count 1
148:     option background-self-heal-count 0
149:     option metadata-self-heal on
150:     option data-self-heal on
151:     option entry-self-heal on
152:     option self-heal-daemon enable
153:     option iam-self-heal-daemon yes
154:     subvolumes myvol-pro-client-0 myvol-pro-client-1 myvol-pro-client-2
155: end-volume
156:
157: volume glustershd
158:     type debug/io-stats
[2017-08-22 13:45:34.144445] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-private-client-0: Server lk version = 1
159:     option log-level INFO
160:     subvolumes othervol-replicate-0 myvol-private-replicate-0 myvol-pro-replicate-0
161: end-volume
162:
+------------------------------------------------------------------------------+
[2017-08-22 13:45:34.144634] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-pro-client-0: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:34.144705] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-private-client-2: Server lk version = 1
[2017-08-22 13:45:34.144841] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-pro-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:34.145198] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-pro-client-0: Connected to myvol-pro-client-0, attached to remote volume '/data/myvol-pro/brick'.
[2017-08-22 13:45:34.145241] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-pro-client-0: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:34.145276] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-pro-client-2: Connected to myvol-pro-client-2, attached to remote volume '/srv/glusterfs/myvol-pro/brick'.
[2017-08-22 13:45:34.145305] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-pro-client-2: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:34.145285] I [MSGID: 108005] [afr-common.c:4387:afr_notify] 0-myvol-pro-replicate-0: Subvolume 'myvol-pro-client-0' came back up; going online.
[2017-08-22 13:45:34.145407] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-pro-client-2: Server lk version = 1
[2017-08-22 13:45:34.145435] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-pro-client-0: Server lk version = 1
[2017-08-22 13:45:34.156576] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-othervol-client-1: changing port to 49162 (from 0)
[2017-08-22 13:45:34.156652] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-private-client-1: changing port to 49163 (from 0)
[2017-08-22 13:45:34.157041] I [rpc-clnt.c:1965:rpc_clnt_reconfig] 0-myvol-pro-client-1: changing port to 49164 (from 0)
[2017-08-22 13:45:34.157376] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-othervol-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:34.157436] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-private-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:34.157824] I [MSGID: 114057] [client-handshake.c:1440:select_server_supported_programs] 0-myvol-pro-client-1: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2017-08-22 13:45:34.157879] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-othervol-client-1: Connected to othervol-client-1, attached to remote volume '/data/othervol/brick'.
[2017-08-22 13:45:34.157906] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-othervol-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:34.157977] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-private-client-1: Connected to myvol-private-client-1, attached to remote volume '/data/myvol-private/brick'.
[2017-08-22 13:45:34.157995] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-private-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:34.158145] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-othervol-client-1: Server lk version = 1
[2017-08-22 13:45:34.158276] I [MSGID: 114046] [client-handshake.c:1216:client_setvolume_cbk] 0-myvol-pro-client-1: Connected to myvol-pro-client-1, attached to remote volume '/data/myvol-pro/brick'.
[2017-08-22 13:45:34.158311] I [MSGID: 114047] [client-handshake.c:1227:client_setvolume_cbk] 0-myvol-pro-client-1: Server and Client lk-version numbers are not same, reopening the fds
[2017-08-22 13:45:34.158405] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-private-client-1: Server lk version = 1
[2017-08-22 13:45:34.158771] I [MSGID: 114035] [client-handshake.c:202:client_set_lk_version_cbk] 0-myvol-pro-client-1: Server lk version = 1
[2017-08-24 09:48:45.377538] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2017-08-24 09:48:45.390058] I [glusterfsd-mgmt.c:54:mgmt_cbk_spec] 0-mgmt: Volume file changed
[2017-08-24 09:48:45.399196] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-othervol-replicate-0: adding option 'node-uuid' for volume 'othervol-replicate-0' with value '60a03a81-ba92-4b84-90fe-7b6e35a10975'
[2017-08-24 09:48:45.399215] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-myvol-private-replicate-0: adding option 'node-uuid' for volume 'myvol-private-replicate-0' with value '60a03a81-ba92-4b84-90fe-7b6e35a10975'
[2017-08-24 09:48:45.399225] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-myvol-pro-replicate-0: adding option 'node-uuid' for volume 'myvol-pro-replicate-0' with value '60a03a81-ba92-4b84-90fe-7b6e35a10975'
[2017-08-24 09:48:45.399446] D [io-stats.c:3574:reconfigure] 0-glustershd: reconfigure returning 0
[2017-08-24 09:48:45.399464] D [glusterfsd-mgmt.c:1628:mgmt_getspec_cbk] 0-glusterfsd-mgmt: No need to re-load volfile, reconfigure done
[2017-08-24 09:48:45.399524] I [glusterfsd-mgmt.c:1600:mgmt_getspec_cbk] 0-glusterfs: No change in volfile, continuing
[2017-08-24 09:49:26.000680] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7ff11c17df13] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7ff11bf4b397] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13997)[0x7ff11bf4b997] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29d)[0x7ff11bf4825d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xef6e)[0x7ff116002f6e] ))))) 0-: 127.0.1.1:49157: ping timer event already removed
[2017-08-24 09:49:26.000817] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-private-client-2: subvol myvol-private-client-2 is local
[2017-08-24 09:49:26.000857] D [MSGID: 0] [afr-self-heald.c:589:afr_shd_index_healer] 0-myvol-private-replicate-0: starting index sweep on subvol myvol-private-client-2
[2017-08-24 09:49:26.001056] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-private-replicate-0: glusterfs.xattrop_index_gfid dir gfid for myvol-private-client-2: eac16f26-6e85-42f8-933c-996e9754c4da
[2017-08-24 09:49:26.002639] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-private-replicate-0: glusterfs.xattrop_dirty_gfid dir gfid for myvol-private-client-2: c86b2b89-a82a-4601-adbd-b3482240d1dd
[2017-08-24 09:49:26.003877] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-private-replicate-0: glusterfs.xattrop_entry_changes_gfid dir gfid for myvol-private-client-2: eecd5be5-0704-41b1-a1ec-fe5776c834eb
[2017-08-24 09:49:26.004972] D [MSGID: 0] [afr-self-heald.c:606:afr_shd_index_healer] 0-myvol-private-replicate-0: finished index sweep on subvol myvol-private-client-2
[2017-08-24 09:49:47.276389] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present
[2017-08-24 09:49:47.276468] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
[2017-08-24 09:49:58.000432] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7ff11c17df13] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7ff11bf4b397] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13997)[0x7ff11bf4b997] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29d)[0x7ff11bf4825d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xef6e)[0x7ff116002f6e] ))))) 0-: 127.0.1.1:49156: ping timer event already removed
[2017-08-24 09:49:58.000559] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-pro-client-2: subvol myvol-pro-client-2 is local
[2017-08-24 09:49:58.000579] D [MSGID: 0] [afr-self-heald.c:589:afr_shd_index_healer] 0-myvol-pro-replicate-0: starting index sweep on subvol myvol-pro-client-2
[2017-08-24 09:49:58.000715] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_index_gfid dir gfid for myvol-pro-client-2: 2f14d9b5-e319-4f7e-8628-031746eb692b
[2017-08-24 09:49:58.001832] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_dirty_gfid dir gfid for myvol-pro-client-2: 0cf56943-8424-4e07-af22-80263afecaf7
[2017-08-24 09:49:58.002685] D [MSGID: 0] [afr-self-heald.c:432:afr_shd_index_heal] 0-myvol-pro-replicate-0: got entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
[2017-08-24 09:49:58.003108] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7ff11c17df13] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7ff11bf4b397] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13997)[0x7ff11bf4b997] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29d)[0x7ff11bf4825d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xef6e)[0x7ff116002f6e] ))))) 0-: 192.168.10.31:49162: ping timer event already removed
[2017-08-24 09:49:58.003321] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7ff11c17df13] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7ff11bf4b397] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13997)[0x7ff11bf4b997] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29d)[0x7ff11bf4825d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xef6e)[0x7ff116002f6e] ))))) 0-: 192.168.10.33:49164: ping timer event already removed
[2017-08-24 09:49:58.005730] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_entry_changes_gfid dir gfid for myvol-pro-client-2: 9bfe4325-bd11-4975-9915-071c63e6db4f
[2017-08-24 09:49:58.006511] D [MSGID: 0] [afr-self-heald.c:606:afr_shd_index_healer] 0-myvol-pro-replicate-0: finished index sweep on subvol myvol-pro-client-2
[2017-08-24 09:50:08.278263] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-private-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:50:40.280982] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:50:40.281084] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:50:40.281103] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:51:47.286786] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present
[2017-08-24 09:51:47.286867] D [logging.c:1791:__gf_log_inject_timer_event] 0-logging-infra: Starting timer now. Timeout = 120, current buf size = 5
[2017-08-24 09:52:26.305406] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7ff11c17df13] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7ff11bf4b397] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13997)[0x7ff11bf4b997] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29d)[0x7ff11bf4825d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xef6e)[0x7ff116002f6e] ))))) 0-: 192.168.10.31:49162: ping timer event already removed
[2017-08-24 09:52:26.305662] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-pro-client-0: subvol myvol-pro-client-0 is local
[2017-08-24 09:52:26.305827] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7ff11c17df13] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7ff11bf4b397] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13997)[0x7ff11bf4b997] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29d)[0x7ff11bf4825d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xef6e)[0x7ff116002f6e] ))))) 0-: 192.168.10.33:49164: ping timer event already removed
[2017-08-24 09:52:26.306029] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-pro-client-1: subvol myvol-pro-client-1 is local
[2017-08-24 09:52:26.306197] D [rpc-clnt-ping.c:93:rpc_clnt_remove_ping_timer_locked] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x1a3)[0x7ff11c17df13] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_remove_ping_timer_locked+0x87)[0x7ff11bf4b397] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0x13997)[0x7ff11bf4b997] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_submit+0x29d)[0x7ff11bf4825d] (--> /usr/lib/x86_64-linux-gnu/glusterfs/3.8.11/xlator/protocol/client.so(+0xef6e)[0x7ff116002f6e] ))))) 0-: 127.0.1.1:49156: ping timer event already removed
[2017-08-24 09:52:26.306399] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-pro-client-2: subvol myvol-pro-client-2 is local
[2017-08-24 09:52:26.306678] D [MSGID: 0] [afr-self-heald.c:589:afr_shd_index_healer] 0-myvol-pro-replicate-0: starting index sweep on subvol myvol-pro-client-2
[2017-08-24 09:52:26.306891] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_index_gfid dir gfid for myvol-pro-client-2: 2f14d9b5-e319-4f7e-8628-031746eb692b
[2017-08-24 09:52:26.308717] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_dirty_gfid dir gfid for myvol-pro-client-2: 0cf56943-8424-4e07-af22-80263afecaf7
[2017-08-24 09:52:26.310081] D [MSGID: 0] [afr-self-heald.c:432:afr_shd_index_heal] 0-myvol-pro-replicate-0: got entry: 1985e233-d5ee-4e3e-a51a-cf0b5f9f2aea
[2017-08-24 09:52:26.313648] D [MSGID: 0] [afr-self-heald.c:432:afr_shd_index_heal] 0-myvol-pro-replicate-0: got entry: 16cf5457-fcda-40be-a982-c4307dffcda2
[2017-08-24 09:52:26.314602] D [MSGID: 0] [afr-self-heal-common.c:1886:afr_selfheal_unlocked_inspect] 0-myvol-pro-replicate-0: SIZE mismatch 8192 vs 0 on myvol-pro-client-2 for gfid:16cf5457-fcda-40be-a982-c4307dffcda2
[2017-08-24 09:52:26.315100] D [MSGID: 114031] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-myvol-pro-client-2: remote operation failed [Resource temporarily unavailable]
[2017-08-24 09:52:26.315138] D [MSGID: 0] [client-rpc-fops.c:1548:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7ff1199ff73c, myvol-pro-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable]
[2017-08-24 09:52:26.315142] D [MSGID: 114031] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-myvol-pro-client-0: remote operation failed [Resource temporarily unavailable]
[2017-08-24 09:52:26.315178] D [MSGID: 0] [client-rpc-fops.c:1548:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7ff1199ff73c, myvol-pro-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable]
[2017-08-24 09:52:26.315182] D [MSGID: 114031] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-myvol-pro-client-1: remote operation failed [Resource temporarily unavailable]
[2017-08-24 09:52:26.315209] D [MSGID: 0] [client-rpc-fops.c:1548:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7ff1199ff73c, myvol-pro-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable]
[2017-08-24 09:52:26.315295] D [MSGID: 0] [afr-self-heal-data.c:825:afr_selfheal_data] 0-myvol-pro-replicate-0: 16cf5457-fcda-40be-a982-c4307dffcda2: Skipping self-heal as only 0 number of subvolumes could be locked
[2017-08-24 09:52:26.315383] D [MSGID: 0] [afr-self-heald.c:432:afr_shd_index_heal] 0-myvol-pro-replicate-0: got entry: 6167bf72-c9a2-476d-a41b-96ca1507e2f5
[2017-08-24 09:52:26.316215] D [MSGID: 0] [afr-self-heal-common.c:1886:afr_selfheal_unlocked_inspect] 0-myvol-pro-replicate-0: SIZE mismatch 8192 vs 0 on myvol-pro-client-2 for gfid:6167bf72-c9a2-476d-a41b-96ca1507e2f5
[2017-08-24 09:52:26.316963] D [MSGID: 114031] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-myvol-pro-client-2: remote operation failed [Resource temporarily unavailable]
[2017-08-24 09:52:26.316994] D [MSGID: 0] [client-rpc-fops.c:1548:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7ff1199fa7a8, myvol-pro-client-2 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable]
[2017-08-24 09:52:26.317001] D [MSGID: 114031] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-myvol-pro-client-1: remote operation failed [Resource temporarily unavailable]
[2017-08-24 09:52:26.317032] D [MSGID: 0] [client-rpc-fops.c:1548:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7ff1199fa7a8, myvol-pro-client-1 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable]
[2017-08-24 09:52:26.317037] D [MSGID: 114031] [client-rpc-fops.c:1545:client3_3_inodelk_cbk] 0-myvol-pro-client-0: remote operation failed [Resource temporarily unavailable]
[2017-08-24 09:52:26.317054] D [MSGID: 0] [client-rpc-fops.c:1548:client3_3_inodelk_cbk] 0-stack-trace: stack-address: 0x7ff1199fa7a8, myvol-pro-client-0 returned -1 error: Resource temporarily unavailable [Resource temporarily unavailable]
[2017-08-24 09:52:26.548802] D [MSGID: 0] [afr-self-heald.c:227:afr_shd_index_inode] 0-myvol-pro-replicate-0: glusterfs.xattrop_entry_changes_gfid dir gfid for myvol-pro-client-2: 9bfe4325-bd11-4975-9915-071c63e6db4f
[2017-08-24 09:52:26.549905] D [MSGID: 0] [afr-self-heald.c:606:afr_shd_index_healer] 0-myvol-pro-replicate-0: finished index sweep on subvol myvol-pro-client-2
[2017-08-24 09:53:09.293533] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-0: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:53:09.293592] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-1: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:53:09.293610] D [rpc-clnt-ping.c:295:rpc_clnt_start_ping] 0-myvol-pro-client-2: returning as transport is already disconnected OR there are no frames (0 || 0)
[2017-08-24 09:53:47.296769] D [logging.c:1829:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer timed out. About to flush outstanding messages if present
[2017-08-24 09:52:26.306674] D [MSGID: 0] [syncop-utils.c:548:syncop_is_subvol_local] 0-myvol-pro-client-2: subvol myvol-pro-client-2 is local
