[2017-04-28 07:28:03.196004] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:28:03.203614] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:28:03.203654] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:28:03.208918] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:28:03.208954] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:28:03.208968] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:28:03.209036] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:28:03.209056] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:28:04.693741] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:28:04.694694] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:28:04.696166] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:28:04.696294] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:28:04.696307] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:28:04.696327] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:28:04.696652] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:28:04.696648] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:28:04.697145] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:28:04.740957] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:28:04.742308] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2 [2017-04-28 07:28:04.742322] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:28:04.777048] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:28:04.777150] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:28:04.822564] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:28:05.551442] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received RJT from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0, port: 0 [2017-04-28 07:28:05.556904] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:28:05.560526] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:28:05.560596] E [MSGID: 106010] [glusterd-utils.c:3011:glusterd_compare_friend_volume] 0-management: Version of Cksums gv0 differ. local cksum = 4224731823, remote cksum = 1131418324 on peer sst0 [2017-04-28 07:28:05.560663] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0 (0), ret: 0, op_ret: -1 [2017-04-28 07:28:39.446867] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst0 24007 [2017-04-28 07:28:41.333950] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:28:46.326386] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:28:56.471370] I [MSGID: 106499] [glusterd-handler.c:4363:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv0 [2017-04-28 07:32:45.753176] W [glusterfsd.c:1329:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f2653de26ba] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xe5) [0x557fad3967b5] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x557fad396624] ) 0-: received signum (15), shutting down [2017-04-28 07:34:36.198249] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:34:36.205642] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:34:36.205682] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:34:36.211006] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:34:36.211041] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:34:36.211054] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:34:36.211119] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:34:36.211139] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:34:37.795217] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:34:37.797039] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:34:37.799291] I [MSGID: 106498] [glusterd-handler.c:3669:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0 [2017-04-28 07:34:37.799333] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:34:37.799355] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:34:37.800437] E [MSGID: 106187] [glusterd-store.c:4559:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore [2017-04-28 07:34:37.800479] E [MSGID: 101019] [xlator.c:503:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again [2017-04-28 07:34:37.800496] E [MSGID: 101066] [graph.c:325:glusterfs_graph_init] 0-management: initializing translator failed [2017-04-28 07:34:37.800516] E [MSGID: 101176] [graph.c:681:glusterfs_graph_activate] 0-graph: init failed [2017-04-28 07:34:37.800962] W [glusterfsd.c:1329:cleanup_and_exit] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xe5) [0x563af9820185] -->/usr/sbin/glusterd(glusterfs_process_volfp+0x1b7) [0x563af9820047] -->/usr/sbin/glusterd(cleanup_and_exit+0x54) [0x563af981f624] ) 0-: received signum (1), shutting down [2017-04-28 07:36:16.994152] I [MSGID: 100030] [glusterfsd.c:2460:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.10.1 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid) [2017-04-28 07:36:17.002052] I [MSGID: 106478] [glusterd.c:1449:init] 0-management: Maximum allowed open file descriptors set to 65536 [2017-04-28 07:36:17.002096] I [MSGID: 106479] [glusterd.c:1496:init] 0-management: Using /var/lib/glusterd as working directory [2017-04-28 07:36:17.008031] W [MSGID: 103071] [rdma.c:4590:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device] [2017-04-28 07:36:17.008072] W [MSGID: 103055] [rdma.c:4897:init] 0-rdma.management: Failed to initialize IB Device [2017-04-28 07:36:17.008088] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed [2017-04-28 07:36:17.008160] W [rpcsvc.c:1661:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed [2017-04-28 07:36:17.008181] E [MSGID: 106243] [glusterd.c:1720:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport [2017-04-28 07:36:18.537736] I [MSGID: 106513] [glusterd-store.c:2197:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30706 [2017-04-28 07:36:18.537844] I [MSGID: 106194] [glusterd-store.c:3776:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list. [2017-04-28 07:36:18.537898] E [MSGID: 101032] [store.c:433:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory] Final graph: +------------------------------------------------------------------------------+ 1: volume management 2: type mgmt/glusterd 3: option rpc-auth.auth-glusterfs on 4: option rpc-auth.auth-unix on 5: option rpc-auth.auth-null on 6: option rpc-auth-allow-insecure on 7: option transport.socket.listen-backlog 128 8: option event-threads 1 9: option ping-timeout 0 10: option transport.socket.read-fail-log off 11: option transport.socket.keepalive-interval 2 12: option transport.socket.keepalive-time 10 13: option transport-type rdma 14: option working-directory /var/lib/glusterd 15: end-volume 16: +------------------------------------------------------------------------------+ [2017-04-28 07:36:18.538873] I [MSGID: 101190] [event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1 [2017-04-28 07:36:18.545097] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:36:18.545144] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 5a2198de-f536-4328-a278-7f746f276e35 [2017-04-28 07:36:18.548869] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc [2017-04-28 07:36:18.549154] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst0.webkontrol.ru (24007), ret: 0, op_ret: -1 [2017-04-28 07:36:19.871627] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:36:25.519090] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst0 24007 [2017-04-28 07:36:25.519575] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst0 (24007) [2017-04-28 07:36:25.520506] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:36:25.520540] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:36:25.521090] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:36:25.536539] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0 [2017-04-28 07:36:25.536568] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:36:25.540030] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 26b35bd7-ad7e-4a25-a3f9-70002771e1fc, host: sst0, port: 0 [2017-04-28 07:36:27.783553] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:36:27.784256] I [MSGID: 106129] [glusterd-handler.c:3690:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: sst2 (24007) [2017-04-28 07:36:27.785385] W [MSGID: 106062] [glusterd-handler.c:3466:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout [2017-04-28 07:36:27.785419] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600 [2017-04-28 07:36:27.785850] I [MSGID: 106498] [glusterd-handler.c:3616:glusterd_friend_add] 0-management: connect returned 0 [2017-04-28 07:36:27.801417] I [MSGID: 106511] [glusterd-rpc-ops.c:261:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f, host: sst2 [2017-04-28 07:36:27.801448] I [MSGID: 106511] [glusterd-rpc-ops.c:421:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req [2017-04-28 07:37:50.228006] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:38:04.539534] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:39:30.570346] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:39:36.600198] I [MSGID: 106487] [glusterd-handler.c:1242:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req sst2 24007 [2017-04-28 07:39:40.535823] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:41:40.679683] I [MSGID: 106487] [glusterd-handler.c:1475:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req [2017-04-28 07:44:36.991808] W [socket.c:593:__socket_rwv] 0-management: readv on sst2:24007 failed (No data available) [2017-04-28 07:44:36.991882] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer (), in state , has disconnected from glusterd. [2017-04-28 07:44:36.992313] E [rpc-clnt.c:365:saved_frames_unwind] (--> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fb0a3cc35fb] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_unwind+0x1d1)[0x7fb0a3a90eb1] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7fb0a3a90fce] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x94)[0x7fb0a3a92654] (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_notify+0x288)[0x7fb0a3a93138] ))))) 0-management: forced unwinding frame type(Peer mgmt) op(--(2)) called at 2017-04-28 07:36:27.801497 (xid=0x5) [2017-04-28 07:44:36.992398] I [socket.c:3550:socket_submit_reply] 0-socket.management: not connected (priv->connected = -1) [2017-04-28 07:44:36.992416] E [rpcsvc.c:1348:rpcsvc_submit_generic] 0-rpc-service: failed to submit message (XID: 0x1, Program: GlusterD svc cli, ProgVers: 2, Proc: 1) to rpc-transport (socket.management) [2017-04-28 07:44:36.992450] E [MSGID: 106430] [glusterd-utils.c:497:glusterd_submit_reply] 0-glusterd: Reply submission failed [2017-04-28 07:44:48.057303] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:45:51.993206] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:51:03.529476] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer (), in state , has disconnected from glusterd. [2017-04-28 07:51:15.129289] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 07:53:15.259543] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 07:53:15.263540] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:53:15.263668] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 07:53:15.266626] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 07:53:15.267540] I [MSGID: 106493] [glusterd-handler.c:3866:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to sst2 (0), ret: 0, op_ret: 0 [2017-04-28 07:58:57.733476] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer (), in state , has disconnected from glusterd. [2017-04-28 07:59:09.177314] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:09:32.718239] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706 [2017-04-28 08:09:32.727764] I [MSGID: 106490] [glusterd-handler.c:2957:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:09:32.727897] I [MSGID: 106493] [glusterd-handler.c:3020:__glusterd_handle_probe_query] 0-glusterd: Responded to sst2.webkontrol.ru, op_ret: 0, op_errno: 0, ret: 0 [2017-04-28 08:09:32.754625] I [MSGID: 106490] [glusterd-handler.c:2606:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: bcfdb4e4-e1a7-4493-b18b-b3456f706f6f [2017-04-28 08:49:13.070794] I [MSGID: 106004] [glusterd-handler.c:5882:__glusterd_peer_rpc_notify] 0-management: Peer (), in state , has disconnected from glusterd. [2017-04-28 08:49:24.601282] E [socket.c:2310:socket_connect_finish] 0-management: connection to sst2:24007 failed (Connection refused) [2017-04-28 08:53:14.954821] I [MSGID: 106163] [glusterd-handshake.c:1304:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30706