[2017-04-27 13:53:11.966640] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid)
[2017-04-27 13:53:11.969733] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-04-27 13:53:11.969748] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory
[2017-04-27 13:53:11.972015] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2017-04-27 13:53:11.972030] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device
[2017-04-27 13:53:11.972036] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2017-04-27 13:53:11.972065] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-04-27 13:53:11.972074] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-04-27 13:53:13.118479] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706
[2017-04-27 13:53:13.148442] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc
[2017-04-27 13:53:13.150180] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-04-27 13:53:13.150312] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-04-27 13:53:13.150326] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-04-27 13:53:13.150345] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-04-27 13:53:13.150800] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.socket.listen-backlog 128
  8:     option event-threads 1
  9:     option ping-timeout 0
 10:     option transport.socket.read-fail-log off
 11:     option transport.socket.keepalive-interval 2
 12:     option transport.socket.keepalive-time 10
 13:     option transport-type rdma
 14:     option working-directory /var/lib/glusterd
 15: end-volume
 16:  
+------------------------------------------------------------------------------+
[2017-04-27 13:53:13.150796] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-04-27 13:53:13.152013] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-04-27 13:53:13.687393] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-27 13:53:13.688892] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 13:53:13.689174] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-27 13:53:13.693990] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 13:53:13.694097] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst1.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0
[2017-04-27 13:53:13.694872] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0
[2017-04-27 13:53:13.696982] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 13:53:16.147785] E [socket.c:2310:socket_connect_finish] 0-management: connection to 2a01:4f8:b15:1023::2:24007 failed (No route to host)
[2017-04-27 13:53:16.147844] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-04-27 13:53:16.148656] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0
[2017-04-27 13:53:16.150042] E [socket.c:2310:socket_connect_finish] 0-management: connection to 2a01:4f8:b0a:101b::2:24007 failed (No route to host)
[2017-04-27 13:53:16.150088] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst1> (<5a2198de-f536-4328-a278-7f746f276e35>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-04-27 13:53:16.151683] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 13:53:16.151725] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-04-27 13:53:16.985462] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0
[2017-04-27 13:53:16.987764] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 13:53:16.987812] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-04-27 13:53:16.987901] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 13:53:16.988211] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2017-04-27 13:53:16.988437] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-04-27 13:53:16.988479] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-04-27 13:53:16.991101] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2017-04-27 13:53:16.992191] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-04-27 13:53:16.992233] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-04-27 13:53:16.992275] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service
[2017-04-27 13:53:17.994617] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2017-04-27 13:53:17.994907] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-04-27 13:53:17.994947] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-04-27 13:53:17.995007] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2017-04-27 13:53:17.995239] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-04-27 13:53:17.995270] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-04-27 13:53:17.995324] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2017-04-27 13:53:17.995543] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-04-27 13:53:17.995572] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-04-27 13:53:17.997781] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-04-27 13:53:17.998064] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-04-27 13:53:17.998531] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0
[2017-04-27 13:53:17.999620] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 13:53:17.999712] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-04-27 13:53:18.001367] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-04-27 13:53:18.002771] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick sst0:/var/glusterfs has disconnected from glusterd.
[2017-04-27 13:53:18.002993] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 13:53:22.012073] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/glusterfs on port 49152
[2017-04-27 13:53:44.017756] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-27 14:41:27.972234] I [MSGID: 106533] [glusterd-volume-ops.c:916:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gv0
[2017-04-27 14:41:34.404273] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-27 14:44:27.822099] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-27 14:54:08.030062] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fcf739d46ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x55bb4de5f7b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x55bb4de5f624] ) 0-: received signum (15), shutting down
[2017-04-27 14:54:29.766745] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid)
[2017-04-27 14:54:29.770355] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-04-27 14:54:29.770374] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory
[2017-04-27 14:54:29.772949] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2017-04-27 14:54:29.772967] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device
[2017-04-27 14:54:29.772974] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2017-04-27 14:54:29.773006] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-04-27 14:54:29.773016] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-04-27 14:54:30.980463] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706
[2017-04-27 14:54:31.015698] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc
[2017-04-27 14:54:31.017131] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-04-27 14:54:31.017267] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-04-27 14:54:31.017281] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-04-27 14:54:31.017309] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-04-27 14:54:31.017651] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.socket.listen-backlog 128
  8:     option event-threads 1
  9:     option ping-timeout 0
 10:     option transport.socket.read-fail-log off
 11:     option transport.socket.keepalive-interval 2
 12:     option transport.socket.keepalive-time 10
 13:     option transport-type rdma
 14:     option working-directory /var/lib/glusterd
 15: end-volume
 16:  
+------------------------------------------------------------------------------+
[2017-04-27 14:54:31.017646] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-04-27 14:54:31.018602] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-04-27 14:54:31.024673] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-27 14:54:31.028369] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 14:54:31.028858] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0
[2017-04-27 14:54:31.029630] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2017-04-27 14:54:31.029715] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-04-27 14:54:31.029731] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-04-27 14:54:31.030598] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2017-04-27 14:54:31.030981] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-04-27 14:54:31.030998] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-04-27 14:54:31.031015] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service
[2017-04-27 14:54:32.032199] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2017-04-27 14:54:32.032504] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-04-27 14:54:32.032543] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-04-27 14:54:32.032603] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2017-04-27 14:54:32.032840] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-04-27 14:54:32.032870] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-04-27 14:54:32.032923] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2017-04-27 14:54:32.033141] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-04-27 14:54:32.033170] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-04-27 14:54:32.035494] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-04-27 14:54:32.035825] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-04-27 14:54:32.038639] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-27 14:54:32.040678] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-04-27 14:54:32.041627] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick sst0:/var/glusterfs has disconnected from glusterd.
[2017-04-27 14:54:32.041914] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0
[2017-04-27 14:54:32.042781] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 14:54:32.042828] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-04-27 14:54:32.045661] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 14:54:32.086515] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0
[2017-04-27 14:54:32.094695] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 14:54:32.094753] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-04-27 14:54:32.096411] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 14:54:32.096616] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 14:54:32.097496] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0
[2017-04-27 14:54:32.099173] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 14:54:36.049986] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/glusterfs on port 49153
[2017-04-27 14:59:37.402947] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-27 15:00:50.647128] I [MSGID: 106533] [glusterd-volume-ops.c:916:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gv0
[2017-04-27 15:15:09.894711] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fc8e95e16ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x56485ac0e7b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x56485ac0e624] ) 0-: received signum (15), shutting down
[2017-04-27 15:16:51.130160] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid)
[2017-04-27 15:16:51.134590] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-04-27 15:16:51.134613] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory
[2017-04-27 15:16:51.137835] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2017-04-27 15:16:51.137858] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device
[2017-04-27 15:16:51.137868] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2017-04-27 15:16:51.137910] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-04-27 15:16:51.137929] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-04-27 15:16:52.334108] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706
[2017-04-27 15:16:52.364194] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc
[2017-04-27 15:16:52.365653] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-04-27 15:16:52.365782] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2017-04-27 15:16:52.365794] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-04-27 15:16:52.365813] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-04-27 15:16:52.366121] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
Final graph:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option rpc-auth.auth-glusterfs on
  4:     option rpc-auth.auth-unix on
  5:     option rpc-auth.auth-null on
  6:     option rpc-auth-allow-insecure on
  7:     option transport.socket.listen-backlog 128
  8:     option event-threads 1
  9:     option ping-timeout 0
 10:     option transport.socket.read-fail-log off
 11:     option transport.socket.keepalive-interval 2
 12:     option transport.socket.keepalive-time 10
 13:     option transport-type rdma
 14:     option working-directory /var/lib/glusterd
 15: end-volume
 16:  
+------------------------------------------------------------------------------+
[2017-04-27 15:16:52.366117] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-04-27 15:16:52.366995] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-04-27 15:16:53.247237] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0
[2017-04-27 15:16:53.248077] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2017-04-27 15:16:53.248177] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-04-27 15:16:53.248195] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-04-27 15:16:53.249126] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2017-04-27 15:16:53.249534] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-04-27 15:16:53.249551] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-04-27 15:16:53.249569] I [MSGID: 106567] [glusterd-svc-mgmt.c:196:glusterd_svc_start] 0-management: Starting glustershd service
[2017-04-27 15:16:54.251325] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2017-04-27 15:16:54.251621] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-04-27 15:16:54.251690] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-04-27 15:16:54.251751] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2017-04-27 15:16:54.251984] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-04-27 15:16:54.252016] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-04-27 15:16:54.252084] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2017-04-27 15:16:54.252304] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-04-27 15:16:54.252334] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-04-27 15:16:54.254557] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-04-27 15:16:54.254859] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-04-27 15:16:54.255157] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 15:16:54.255227] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-04-27 15:16:54.255367] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 15:16:54.255459] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0
[2017-04-27 15:16:54.257751] I [socket.c:2404:socket_event_handler] 0-transport: disconnecting now
[2017-04-27 15:16:54.258679] I [MSGID: 106005] [glusterd-handler.c:5691:__glusterd_brick_rpc_notify] 0-management: Brick sst0:/var/glusterfs has disconnected from glusterd.
[2017-04-27 15:16:54.259058] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 15:16:54.259195] I [MSGID: 106492] [glusterd-handler.c:2784:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 15:16:54.259253] I [MSGID: 106502] [glusterd-handler.c:2829:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-04-27 15:16:54.259604] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-27 15:16:54.260863] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 15:16:54.261834] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0
[2017-04-27 15:16:54.263351] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-27 15:16:54.264200] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-27 15:16:54.267379] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 15:16:54.268157] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0
[2017-04-27 15:16:54.272611] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-27 15:16:58.067807] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-27 15:16:58.267770] I [MSGID: 106143] [glusterd-pmap.c:277:pmap_registry_bind] 0-pmap: adding brick /var/glusterfs on port 49152
[2017-04-27 15:17:08.145868] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-27 15:17:11.346625] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 07:07:08.939198] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 07:07:22.465221] I [MSGID: 106533] [glusterd-volume-ops.c:916:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume gv0
[2017-04-28 07:07:23.219676] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-04-28 07:07:23.219740] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-04-28 07:07:23.219839] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-04-28 07:07:23.219867] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-04-28 07:07:23.219959] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-04-28 07:07:23.219983] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-04-28 07:07:23.255153] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd
[2017-04-28 07:07:23.272213] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd
[2017-04-28 07:26:57.666373] W [socket.c:593:__socket_rwv] 0-management: readv on sst1:24007 failed (No data available)
[2017-04-28 07:26:57.666447] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst1> (<5a2198de-f536-4328-a278-7f746f276e35>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-04-28 07:26:57.666602] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held
[2017-04-28 07:26:57.666630] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0
[2017-04-28 07:27:09.534588] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst1:24007 failed (Connection refused)
[2017-04-28 07:28:04.734953] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-28 07:28:04.739439] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-28 07:28:05.548023] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 1131418324, remote cksum = 4224731823 on peer sst1
[2017-04-28 07:28:05.548159] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: -1
[2017-04-28 07:28:05.560095] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 0
[2017-04-28 07:31:35.775788] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-04-28 07:31:35.775843] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-04-28 07:31:35.775940] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-04-28 07:31:35.775966] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-04-28 07:31:35.776055] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-04-28 07:31:35.776077] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-04-28 07:31:35.790917] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd
[2017-04-28 07:31:35.808099] I [run.c:191:runner_log] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0d7a) [0x7f4b94bd6d7a] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd0816) [0x7f4b94bd6816] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105) [0x7f4b99e179e5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh --volname=gv0 -o cluster.self-heal-daemon=enable --gd-workdir=/var/lib/glusterd
[2017-04-28 07:32:19.702837] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2017-04-28 07:32:45.754974] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst1> (<5a2198de-f536-4328-a278-7f746f276e35>), in state <Peer Rejected>, has disconnected from glusterd.
[2017-04-28 07:32:45.755166] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held
[2017-04-28 07:32:45.755197] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0
[2017-04-28 07:32:57.566578] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst1:24007 failed (Connection refused)
[2017-04-28 07:36:18.548429] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 5a2198de-f536-4328-a278-7f746f276e35, host: sst1, port: 24007
[2017-04-28 07:36:25.529048] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-28 07:36:25.532889] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-28 07:36:25.533005] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst1.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0
[2017-04-28 07:36:25.535830] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 5a2198de-f536-4328-a278-7f746f276e35
[2017-04-28 07:36:25.536507] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst1 (0), ret: 0, op_ret: 0
[2017-04-28 07:37:34.559221] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2017-04-28 07:39:52.354497] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2017-04-28 07:44:36.988131] W [socket.c:593:__socket_rwv] 0-management: readv on sst2:24007 failed (No data available)
[2017-04-28 07:44:36.988232] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Peer in Cluster>, has disconnected from glusterd.
[2017-04-28 07:44:36.988390] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held
[2017-04-28 07:44:36.988420] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0
[2017-04-28 07:44:48.647919] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused)
[2017-04-28 07:45:51.983723] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-28 07:45:51.985981] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-28 07:45:51.986141] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 59001066, remote cksum = 1132736709 on peer sst2
[2017-04-28 07:45:51.986269] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: -1
[2017-04-28 07:45:51.987609] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 0
[2017-04-28 07:48:15.716415] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2017-04-28 07:48:28.863531] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 07:51:03.525725] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Peer Rejected>, has disconnected from glusterd.
[2017-04-28 07:51:03.525969] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held
[2017-04-28 07:51:03.526019] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0
[2017-04-28 07:51:13.688993] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused)
[2017-04-28 07:51:50.710054] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2, port: 24007
[2017-04-28 07:51:57.559226] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-28 07:51:57.560345] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-28 07:51:57.560460] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0
[2017-04-28 07:51:57.561004] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-28 07:51:57.561841] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0
[2017-04-28 07:52:31.170428] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 07:52:56.783783] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2017-04-28 07:53:42.149638] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 07:55:03.447810] I [MSGID: 106482] [glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management: Received add brick req
[2017-04-28 07:55:03.447852] I [MSGID: 106578] [glusterd-brick-ops.c:499:__glusterd_handle_add_brick] 0-management: replica-count is 2
[2017-04-28 07:55:03.447865] E [MSGID: 106291] [glusterd-brick-ops.c:626:__glusterd_handle_add_brick] 0-management: 
[2017-04-28 07:55:31.925551] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 07:56:56.035196] I [MSGID: 106482] [glusterd-brick-ops.c:447:__glusterd_handle_add_brick] 0-management: Received add brick req
[2017-04-28 07:56:56.035247] I [MSGID: 106578] [glusterd-brick-ops.c:499:__glusterd_handle_add_brick] 0-management: replica-count is 2
[2017-04-28 07:56:56.035262] E [MSGID: 106291] [glusterd-brick-ops.c:626:__glusterd_handle_add_brick] 0-management: 
[2017-04-28 07:58:57.730694] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Accepted peer request>, has disconnected from glusterd.
[2017-04-28 07:58:57.730885] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held
[2017-04-28 07:58:57.730916] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0
[2017-04-28 07:59:07.732465] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused)
[2017-04-28 08:09:32.693558] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-28 08:09:32.718118] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-28 08:09:32.718248] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0
[2017-04-28 08:09:32.721359] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2
[2017-04-28 08:09:32.721401] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
[2017-04-28 08:09:32.731133] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f
[2017-04-28 08:09:53.596619] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 08:10:53.987182] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2017-04-28 08:15:03.737152] E [MSGID: 106062] [glusterd-utils.c:9338:glusterd_max_opversion_use_rsp_dict] 0-management: Maximum supported op-version not set in destination dictionary
[2017-04-28 08:15:26.819219] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 08:15:54.288580] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 08:19:24.708030] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 08:19:38.912746] E [rpc-clnt.c:200:call_bail] 0-management: bailing out frame type(Peer mgmt) op(--(2)) xid = 0x2f sent = 2017-04-28 08:09:32.721691. timeout = 600 for sst2:24007
[2017-04-28 08:49:13.071592] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer <sst2> (<bcfdb4e4-e1a7-4493-b18b-b3456f706f6f>), in state <Sent and Received peer request>, has disconnected from glusterd.
[2017-04-28 08:49:13.071837] W [glusterd-locks.c:675:glusterd_mgmt_v3_unlock] (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x1f859) [0x7f4b94b25859] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0x29080) [0x7f4b94b2f080] -->/usr/lib/x86_64-linux-gnu/glusterfs/3.10.1/xlator/mgmt/glusterd.so(+0xd50e3) [0x7f4b94bdb0e3] ) 0-management: Lock for vol gv0 not held
[2017-04-28 08:49:13.071868] W [MSGID: 106118] [glusterd-handler.c:5907:__glusterd_peer_rpc_notify] 0-management: Lock not released for gv0
[2017-04-28 08:49:24.046693] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused)
[2017-04-28 08:53:14.944791] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706
[2017-04-28 08:59:59.771701] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2017-04-28 09:00:35.795605] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0
[2017-04-28 08:59:59.772167] I [MSGID: 106488] [glusterd-handler.c:1538:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2017-04-28 09:01:11.179203] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req