[2018-07-11 19:16:12.488634] W [socket.c:593:__socket_rwv] 0-management: readv on 10.4.16.12:24007 failed (Connection timed out)
[2018-07-11 19:16:12.503577] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <10.4.16.12> (<dfe01058-5bea-4b67-8859-382a2c8854f4>), in state <Peer in Cluster>, has disconnected from glusterd.
[2018-07-11 19:16:12.592253] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0x2322a) [0x7f60f376a22a] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0x2d198) [0x7f60f3774198] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xe4765) [0x7f60f382b765] ) 0-management: Lock for vol EXPORTB not held
[2018-07-11 19:16:12.602249] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for EXPORTB
[2018-07-11 19:16:12.602448] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0x2322a) [0x7f60f376a22a] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0x2d198) [0x7f60f3774198] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xe4765) [0x7f60f382b765] ) 0-management: Lock for vol gv1 not held
[2018-07-11 19:16:12.602478] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv1
[2018-07-11 19:16:12.618832] W [socket.c:593:__socket_rwv] 0-management: readv on 10.4.16.11:24007 failed (Connection timed out)
[2018-07-11 19:16:12.618887] I [MSGID: 106004] [glusterd-handler.c:6317:__glusterd_peer_rpc_notify] 0-management: Peer <10.4.16.11> (<0f3090ee-080b-4a6b-9964-0ca86d801469>), in state <Peer in Cluster>, has disconnected from glusterd.
[2018-07-11 19:16:12.619010] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0x2322a) [0x7f60f376a22a] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0x2d198) [0x7f60f3774198] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xe4765) [0x7f60f382b765] ) 0-management: Lock for vol EXPORTB not held
[2018-07-11 19:16:12.619057] W [glusterd-locks.c:843:glusterd_mgmt_v3_unlock] (-->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0x2322a) [0x7f60f376a22a] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0x2d198) [0x7f60f3774198] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xe4765) [0x7f60f382b765] ) 0-management: Lock for vol gv1 not held
[2018-07-11 19:16:12.619152] C [MSGID: 106002] [glusterd-server-quorum.c:360:glusterd_do_volume_quorum_action] 0-management: Server quorum lost for volume gv1. Stopping local bricks.
[2018-07-11 19:16:12.641346] I [MSGID: 106542] [glusterd-utils.c:8099:glusterd_brick_signal] 0-glusterd: sending signal 15 to brick with pid 2115
[2018-07-11 19:16:13.651358] I [MSGID: 106144] [glusterd-pmap.c:396:pmap_registry_remove] 0-pmap: removing brick /glusterfs/data1/gv1 on port 49153
[2018-07-11 19:16:13.661287] W [glusterd-handler.c:6064:__glusterd_brick_rpc_notify] 0-management: got disconnect from stale rpc on /glusterfs/data1/gv1
[2018-07-11 19:16:23.332759] W [rpc-clnt-ping.c:246:rpc_clnt_ping_cbk] 0-management: RPC_CLNT_PING notify failed
[2018-07-11 19:16:23.664421] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600
[2018-07-11 19:16:23.666255] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 0f3090ee-080b-4a6b-9964-0ca86d801469
[2018-07-11 19:16:23.873295] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: dfe01058-5bea-4b67-8859-382a2c8854f4, host: 10.4.16.12, port: 0
[2018-07-11 19:16:23.877403] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.4.16.11 (0), ret: 0, op_ret: 0
[2018-07-11 19:16:23.891916] C [MSGID: 106003] [glusterd-server-quorum.c:354:glusterd_do_volume_quorum_action] 0-management: Server quorum regained for volume gv1. Starting local bricks.
[2018-07-11 19:16:23.892149] I [glusterd-utils.c:5941:glusterd_brick_start] 0-management: starting a fresh brick process for brick /glusterfs/data1/gv1
[2018-07-11 19:16:23.896812] I [MSGID: 106144] [glusterd-pmap.c:396:pmap_registry_remove] 0-pmap: removing brick /glusterfs/data1/gv1 on port 49153
[2018-07-11 19:16:23.902276] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-07-11 19:16:23.931403] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: dfe01058-5bea-4b67-8859-382a2c8854f4
[2018-07-11 19:16:23.932357] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-07-11 19:16:23.940052] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping nfs daemon running in pid: 25885
[2018-07-11 19:16:24.940457] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped
[2018-07-11 19:16:24.965434] I [MSGID: 106540] [glusterd-utils.c:4939:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered MOUNTV3 successfully
[2018-07-11 19:16:24.966193] I [MSGID: 106540] [glusterd-utils.c:4948:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered MOUNTV1 successfully
[2018-07-11 19:16:24.966780] I [MSGID: 106540] [glusterd-utils.c:4957:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NFSV3 successfully
[2018-07-11 19:16:24.967380] I [MSGID: 106540] [glusterd-utils.c:4966:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NLM v4 successfully
[2018-07-11 19:16:24.968005] I [MSGID: 106540] [glusterd-utils.c:4975:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NLM v1 successfully
[2018-07-11 19:16:24.968558] I [MSGID: 106540] [glusterd-utils.c:4984:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered ACL v3 successfully
[2018-07-11 19:16:24.992325] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting nfs service
[2018-07-11 19:16:24.998379] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 25898
[2018-07-11 19:16:25.999008] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped
[2018-07-11 19:16:25.999203] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service
[2018-07-11 19:16:26.002669] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2018-07-11 19:16:26.002762] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped
[2018-07-11 19:16:26.002942] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-07-11 19:16:26.002984] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped
[2018-07-11 19:16:26.003142] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-07-11 19:16:26.003179] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped
[2018-07-11 19:16:26.011122] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 0f3090ee-080b-4a6b-9964-0ca86d801469
[2018-07-11 19:16:26.012277] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-07-11 19:16:26.013303] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2018-07-11 19:16:26.014041] I [MSGID: 106005] [glusterd-handler.c:6071:__glusterd_brick_rpc_notify] 0-management: Brick 10.4.16.19:/glusterfs/data1/gv1 has disconnected from glusterd.
[2018-07-11 19:16:26.014710] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: dfe01058-5bea-4b67-8859-382a2c8854f4
[2018-07-11 19:16:26.023253] I [MSGID: 106163] [glusterd-handshake.c:1316:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30600
[2018-07-11 19:16:26.025365] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: dfe01058-5bea-4b67-8859-382a2c8854f4
[2018-07-11 19:16:26.041670] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 10.4.16.12 (0), ret: 0, op_ret: 0
[2018-07-11 19:16:26.044042] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: dfe01058-5bea-4b67-8859-382a2c8854f4
[2018-07-11 19:16:26.044765] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-07-11 19:16:26.045026] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: dfe01058-5bea-4b67-8859-382a2c8854f4
[2018-07-11 19:16:26.193249] I [MSGID: 106143] [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick /glusterfs/data1/gv1 on port 49152
[2018-07-11 19:16:26.941324] W [rpc-clnt-ping.c:246:rpc_clnt_ping_cbk] 0-management: RPC_CLNT_PING notify failed
[2018-07-11 19:16:26.958973] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 0f3090ee-080b-4a6b-9964-0ca86d801469, host: 10.4.16.11, port: 0
[2018-07-11 19:16:26.961398] I [glusterd-utils.c:5847:glusterd_brick_start] 0-management: discovered already-running brick /glusterfs/data1/gv1
[2018-07-11 19:16:26.961437] I [MSGID: 106143] [glusterd-pmap.c:295:pmap_registry_bind] 0-pmap: adding brick /glusterfs/data1/gv1 on port 49152
[2018-07-11 19:16:26.961628] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 0f3090ee-080b-4a6b-9964-0ca86d801469
[2018-07-11 19:16:26.962590] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-07-11 19:16:26.962901] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping nfs daemon running in pid: 5954
[2018-07-11 19:16:27.963233] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: nfs service is stopped
[2018-07-11 19:16:27.964124] I [MSGID: 106540] [glusterd-utils.c:4939:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered MOUNTV3 successfully
[2018-07-11 19:16:27.964691] I [MSGID: 106540] [glusterd-utils.c:4948:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered MOUNTV1 successfully
[2018-07-11 19:16:27.965331] I [MSGID: 106540] [glusterd-utils.c:4957:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NFSV3 successfully
[2018-07-11 19:16:27.965974] I [MSGID: 106540] [glusterd-utils.c:4966:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NLM v4 successfully
[2018-07-11 19:16:27.966587] I [MSGID: 106540] [glusterd-utils.c:4975:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered NLM v1 successfully
[2018-07-11 19:16:27.967239] I [MSGID: 106540] [glusterd-utils.c:4984:glusterd_nfs_pmap_deregister] 0-glusterd: De-registered ACL v3 successfully
[2018-07-11 19:16:27.971364] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting nfs service
[2018-07-11 19:16:28.977134] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 5963
[2018-07-11 19:16:29.977492] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: glustershd service is stopped
[2018-07-11 19:16:29.977596] I [MSGID: 106567] [glusterd-svc-mgmt.c:197:glusterd_svc_start] 0-management: Starting glustershd service
[2018-07-11 19:16:29.981043] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2018-07-11 19:16:29.981116] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped
[2018-07-11 19:16:29.981275] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-07-11 19:16:29.981326] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped
[2018-07-11 19:16:29.981481] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-07-11 19:16:29.981514] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped
[2018-07-11 19:16:29.989084] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 0f3090ee-080b-4a6b-9964-0ca86d801469
[2018-07-11 19:16:12.619036] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for EXPORTB
[2018-07-11 19:16:12.619077] W [MSGID: 106118] [glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv1
[2018-07-11 20:31:06.707428] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2018-07-11 20:31:06.707494] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped
[2018-07-11 20:31:06.707798] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-07-11 20:31:06.707841] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped
[2018-07-11 20:31:06.708122] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-07-11 20:31:06.708158] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped
[2018-07-11 20:31:06.757167] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xdef0a) [0x7f60f3825f0a] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xde9cd) [0x7f60f38259cd] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f60fed25e05] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gv1 -o network.ping-timeout=75 --gd-workdir=/var/lib/glusterd
[2018-07-11 20:31:06.779545] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xdef0a) [0x7f60f3825f0a] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xde9cd) [0x7f60f38259cd] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f60fed25e05] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gv1 -o network.ping-timeout=75 --gd-workdir=/var/lib/glusterd
[2018-07-11 20:31:29.645512] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped
[2018-07-11 20:31:29.645769] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-07-11 20:32:00.557947] I [MSGID: 106488] [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2018-07-11 20:37:41.348320] I [MSGID: 106499] [glusterd-handler.c:4303:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv1
[2018-07-11 20:37:41.350482] W [rpc-clnt-ping.c:246:rpc_clnt_ping_cbk] 0-management: RPC_CLNT_PING notify failed
[2018-07-11 20:37:41.351249] W [rpc-clnt-ping.c:246:rpc_clnt_ping_cbk] 0-management: RPC_CLNT_PING notify failed
[2018-07-11 20:45:09.131629] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2018-07-11 20:45:09.131673] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: quotad service is stopped
[2018-07-11 20:45:09.131916] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-07-11 20:45:09.131949] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: bitd service is stopped
[2018-07-11 20:45:09.132180] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-07-11 20:45:09.132210] I [MSGID: 106568] [glusterd-svc-mgmt.c:229:glusterd_svc_stop] 0-management: scrub service is stopped
[2018-07-11 20:45:09.150888] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xdef0a) [0x7f60f3825f0a] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xde9cd) [0x7f60f38259cd] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f60fed25e05] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gv1 -o network.ping-timeout=50 --gd-workdir=/var/lib/glusterd
[2018-07-11 20:45:09.160539] I [run.c:190:runner_log] (-->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xdef0a) [0x7f60f3825f0a] -->/usr/lib64/glusterfs/3.12.6/xlator/mgmt/glusterd.so(+0xde9cd) [0x7f60f38259cd] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f60fed25e05] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gv1 -o network.ping-timeout=50 --gd-workdir=/var/lib/glusterd
[2018-07-11 20:45:28.971567] I [MSGID: 106488] [glusterd-handler.c:1548:__glusterd_handle_cli_get_volume] 0-management: Received get vol req