[Gluster-users] [3.11.2] Bricks disconnect from gluster with 0-transport: EPOLLERR
Leonid Isaev
leonid.isaev at jila.colorado.edu
Sun Aug 6 03:54:43 UTC 2017
Hi,
I have a distributed volume which runs on Fedora 26 systems with
glusterfs 3.11.2 from gluster.org repos:
----------
[root at taupo ~]# glusterd --version
glusterfs 3.11.2
gluster> volume info gv2
Volume Name: gv2
Type: Distribute
Volume ID: 6b468f43-3857-4506-917c-7eaaaef9b6ee
Status: Started
Snapshot Count: 0
Number of Bricks: 6
Transport-type: tcp
Bricks:
Brick1: kiwi:/srv/gluster/gv2/brick1/gvol
Brick2: kiwi:/srv/gluster/gv2/brick2/gvol
Brick3: taupo:/srv/gluster/gv2/brick1/gvol
Brick4: fox:/srv/gluster/gv2/brick1/gvol
Brick5: fox:/srv/gluster/gv2/brick2/gvol
Brick6: logan:/srv/gluster/gv2/brick1/gvol
Options Reconfigured:
performance.readdir-ahead: on
nfs.disable: on
gluster> volume status gv2
Status of volume: gv2
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick kiwi:/srv/gluster/gv2/brick1/gvol 49152 0 Y 1128
Brick kiwi:/srv/gluster/gv2/brick2/gvol 49153 0 Y 1134
Brick taupo:/srv/gluster/gv2/brick1/gvol N/A N/A N N/A
Brick fox:/srv/gluster/gv2/brick1/gvol 49152 0 Y 1169
Brick fox:/srv/gluster/gv2/brick2/gvol 49153 0 Y 1175
Brick logan:/srv/gluster/gv2/brick1/gvol 49152 0 Y 1003
----------
The machine in question is TAUPO which has one brick that refuses to connect to
the cluster. All installations were migrated from glusterfs 3.8.14 on Fedora
24: I simply rsync'ed /var/lib/glusterd to new systems. On all other machines
glusterd starts fine and all bricks come up. Hence I suspect a race condition
somewhere. The glusterd.log file (attached) shows that the brick connects, and
then suddenly disconnects from the cluster:
----------
[2017-08-06 03:12:38.536409] I [glusterd-utils.c:5468:glusterd_brick_start] 0-management: discovered already-running brick /srv/gluster/gv2/brick1/gvol
[2017-08-06 03:12:38.536414] I [MSGID: 106143] [glusterd-pmap.c:279:pmap_registry_bind] 0-pmap: adding brick /srv/gluster/gv2/brick1/gvol on port 49153
[2017-08-06 03:12:38.536427] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-08-06 03:12:38.536500] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-08-06 03:12:38.536556] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-08-06 03:12:38.536616] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9
[2017-08-06 03:12:38.584598] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:38.599340] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:12:38.613745] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd.
----------
I checked that cluster.brick-multiplex is off. How can I debug this further?
Thanks in advance,
--
Leonid Isaev
-------------- next part --------------
[2017-08-06 03:12:36.559795] I [MSGID: 100030] [glusterfsd.c:2476:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.11.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2017-08-06 03:12:36.564906] I [MSGID: 106478] [glusterd.c:1422:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-08-06 03:12:36.564936] I [MSGID: 106479] [glusterd.c:1469:init] 0-management: Using /var/lib/glusterd as working directory
[2017-08-06 03:12:36.567903] E [rpc-transport.c:283:rpc_transport_load] 0-rpc-transport: /usr/lib64/glusterfs/3.11.2/rpc-transport/rdma.so: cannot open shared object file: No such file or directory
[2017-08-06 03:12:36.567916] W [rpc-transport.c:287:rpc_transport_load] 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not valid or not found on this machine
[2017-08-06 03:12:36.567922] W [rpcsvc.c:1660:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2017-08-06 03:12:36.567929] E [MSGID: 106243] [glusterd.c:1693:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2017-08-06 03:12:36.568821] I [MSGID: 106228] [glusterd.c:500:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system [No such file or directory]
[2017-08-06 03:12:36.573114] I [MSGID: 106513] [glusterd-store.c:2193:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 30800
[2017-08-06 03:12:36.691076] I [MSGID: 106498] [glusterd-handler.c:3602:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
The message "I [MSGID: 106498] [glusterd-handler.c:3602:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0" repeated 6 times between [2017-08-06 03:12:36.691076] and [2017-08-06 03:12:36.713022]
[2017-08-06 03:12:36.713040] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-08-06 03:12:36.713057] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-08-06 03:12:36.713102] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2017-08-06 03:12:36.715561] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-08-06 03:12:36.715573] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-08-06 03:12:36.715612] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2017-08-06 03:12:36.716599] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-08-06 03:12:36.716610] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-08-06 03:12:36.716647] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2017-08-06 03:12:36.717624] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-08-06 03:12:36.717634] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-08-06 03:12:36.717672] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2017-08-06 03:12:36.718664] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-08-06 03:12:36.718673] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-08-06 03:12:36.718710] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2017-08-06 03:12:36.719693] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-08-06 03:12:36.719703] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-08-06 03:12:36.719740] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2017-08-06 03:12:36.720718] W [MSGID: 106062] [glusterd-handler.c:3399:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2017-08-06 03:12:36.720728] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-08-06 03:12:36.720764] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2017-08-06 03:12:36.736985] I [MSGID: 106544] [glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID: 4fc4d794-e463-4424-8531-13c639fedfe9
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option rpc-auth-allow-insecure on
7: option transport.socket.listen-backlog 128
8: option event-threads 1
9: option ping-timeout 0
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2017-08-06 03:12:36.737096] I [MSGID: 101190] [event-epoll.c:602:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2017-08-06 03:12:38.525090] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9, host: 128.138.140.58, port: 0
[2017-08-06 03:12:38.535858] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to 600
[2017-08-06 03:12:38.535924] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2017-08-06 03:12:38.535934] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: nfs service is stopped
[2017-08-06 03:12:38.535941] I [MSGID: 106600] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2017-08-06 03:12:38.535956] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting frame-timeout to 600
[2017-08-06 03:12:38.536002] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd already stopped
[2017-08-06 03:12:38.536008] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: glustershd service is stopped
[2017-08-06 03:12:38.536023] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout to 600
[2017-08-06 03:12:38.536088] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: quotad already stopped
[2017-08-06 03:12:38.536096] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: quotad service is stopped
[2017-08-06 03:12:38.536109] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to 600
[2017-08-06 03:12:38.536175] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2017-08-06 03:12:38.536182] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: bitd service is stopped
[2017-08-06 03:12:38.536196] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout to 600
[2017-08-06 03:12:38.536257] I [MSGID: 106132] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2017-08-06 03:12:38.536267] I [MSGID: 106568] [glusterd-svc-mgmt.c:228:glusterd_svc_stop] 0-management: scrub service is stopped
[2017-08-06 03:12:38.536409] I [glusterd-utils.c:5468:glusterd_brick_start] 0-management: discovered already-running brick /srv/gluster/gv2/brick1/gvol
[2017-08-06 03:12:38.536414] I [MSGID: 106143] [glusterd-pmap.c:279:pmap_registry_bind] 0-pmap: adding brick /srv/gluster/gv2/brick1/gvol on port 49153
[2017-08-06 03:12:38.536427] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2017-08-06 03:12:38.536500] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-08-06 03:12:38.536556] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout to 600
[2017-08-06 03:12:38.536616] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9
[2017-08-06 03:12:38.584598] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:38.599340] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:12:38.613745] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd.
[2017-08-06 03:12:38.613783] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9
[2017-08-06 03:12:38.614144] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800
[2017-08-06 03:12:38.622382] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9
[2017-08-06 03:12:38.636536] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.58 (0), ret: 0, op_ret: 0
[2017-08-06 03:12:38.654277] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225, host: 128.138.140.223, port: 0
[2017-08-06 03:12:38.661217] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9
[2017-08-06 03:12:38.719619] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:38.719700] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225
[2017-08-06 03:12:38.719719] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d5a487e3-4c9b-4e5a-91ff-b8d85fd51da9
[2017-08-06 03:12:38.719732] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225
[2017-08-06 03:12:38.719749] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:38.789917] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800
[2017-08-06 03:12:38.800810] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225
[2017-08-06 03:12:38.808300] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.223 (0), ret: 0, op_ret: 0
[2017-08-06 03:12:38.821959] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225
[2017-08-06 03:12:38.821982] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:38.873314] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a, host: takahe.colorado.edu, port: 0
[2017-08-06 03:12:38.880937] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a
[2017-08-06 03:12:38.880974] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:38.929723] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 7da17b36-d09e-4e8f-870c-f4780ed61225
[2017-08-06 03:12:38.975223] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5, host: 128.138.140.48, port: 0
[2017-08-06 03:12:38.982264] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a
[2017-08-06 03:12:38.982651] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800
[2017-08-06 03:12:38.993222] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5
[2017-08-06 03:12:38.993246] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:39.061315] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a
[2017-08-06 03:12:39.067929] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to takahe.colorado.edu (0), ret: 0, op_ret: 0
[2017-08-06 03:12:39.089534] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5
[2017-08-06 03:12:39.089579] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5, host: 128.138.140.227, port: 0
[2017-08-06 03:12:39.100236] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a
[2017-08-06 03:12:39.100258] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:39.151399] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800
[2017-08-06 03:12:39.158128] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5
[2017-08-06 03:12:39.158154] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c, host: 128.138.140.154, port: 0
[2017-08-06 03:12:39.165171] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5
[2017-08-06 03:12:39.185496] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:39.213247] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5
[2017-08-06 03:12:39.223893] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.48 (0), ret: 0, op_ret: 0
[2017-08-06 03:12:39.242085] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 87a12be2-5bb0-47cd-88c4-b2c28c03066a
[2017-08-06 03:12:39.242115] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c
[2017-08-06 03:12:39.290703] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:39.290740] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c
[2017-08-06 03:12:39.290751] I [MSGID: 106493] [glusterd-rpc-ops.c:485:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b, host: pukeko.colorado.edu, port: 0
[2017-08-06 03:12:39.297783] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5
[2017-08-06 03:12:39.297806] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:39.350544] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800
[2017-08-06 03:12:39.357462] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: c4e38a4f-f61f-4bf5-aee1-e33e4daf4ef5
[2017-08-06 03:12:39.357480] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b
[2017-08-06 03:12:39.364344] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:39.405649] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800
[2017-08-06 03:12:39.415886] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5
[2017-08-06 03:12:39.426685] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.227 (0), ret: 0, op_ret: 0
[2017-08-06 03:12:39.443877] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b
[2017-08-06 03:12:39.443916] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c
[2017-08-06 03:12:39.455961] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 128.138.140.154 (0), ret: 0, op_ret: 0
[2017-08-06 03:12:39.473703] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5
[2017-08-06 03:12:39.493665] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:39.518102] I [MSGID: 106163] [glusterd-handshake.c:1309:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 30800
[2017-08-06 03:12:39.525093] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 9b89d233-4cb1-425e-8f50-a13ced4ffbc5
[2017-08-06 03:12:39.525112] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c
[2017-08-06 03:12:39.576708] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:39.576744] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 1bbced1a-1ad9-4f31-9421-346e6dc99b0c
[2017-08-06 03:12:39.576761] I [MSGID: 106490] [glusterd-handler.c:2539:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b
[2017-08-06 03:12:39.587450] I [MSGID: 106493] [glusterd-handler.c:3799:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to pukeko.colorado.edu (0), ret: 0, op_ret: 0
[2017-08-06 03:12:39.609156] I [MSGID: 106492] [glusterd-handler.c:2717:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b
[2017-08-06 03:12:39.616031] I [MSGID: 106502] [glusterd-handler.c:2762:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2017-08-06 03:12:39.658011] I [MSGID: 106493] [glusterd-rpc-ops.c:700:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 111012fe-701f-4102-879b-6ae7ab6b126b
[2017-08-06 03:12:41.583404] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:12:41.602493] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd.
[2017-08-06 03:12:44.578101] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:12:47.582732] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:12:50.583198] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:12:53.584918] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:12:56.583692] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:12:59.584040] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:02.585104] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:05.583899] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:08.583690] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:11.585503] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:14.585056] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:17.586200] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:20.586345] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:23.586469] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:26.584786] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:29.585524] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:32.587889] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:35.588441] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:38.588356] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:41.589152] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:44.589795] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:47.590841] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:50.590175] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:53.591069] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:56.589404] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:13:59.591044] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:02.592658] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:05.594516] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:08.593540] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:11.592057] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:14.592725] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:17.593480] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:20.593690] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:23.593413] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:26.592297] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:29.596041] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:32.595171] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:35.594801] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 38 times between [2017-08-06 03:12:41.602493] and [2017-08-06 03:14:35.612349]
[2017-08-06 03:14:38.595965] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:38.612558] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd.
[2017-08-06 03:14:41.594507] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:44.595193] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:47.596333] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:50.595273] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:53.596662] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:56.596728] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:14:59.596660] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:02.598157] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:05.599272] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:08.598349] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:11.597804] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:14.598448] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:17.599280] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:20.600455] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:23.599751] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:26.600506] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:29.601173] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:32.601107] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:35.601696] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:38.602591] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:41.601936] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:44.599341] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:47.603964] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:50.604358] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:53.600733] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:56.601888] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:15:59.605305] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:02.601723] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:05.605308] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:08.605716] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:11.606025] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:12.478137] I [MSGID: 100030] [glusterfsd.c:2476:main] 0-glusterd: Started running glusterd version 3.11.2 (args: glusterd)
[2017-08-06 03:16:12.484758] I [MSGID: 106478] [glusterd.c:1422:init] 0-management: Maximum allowed open file descriptors set to 65536
[2017-08-06 03:16:12.484797] I [MSGID: 106479] [glusterd.c:1469:init] 0-management: Using /var/lib/glusterd as working directory
[2017-08-06 03:16:12.489459] E [socket.c:796:__socket_server_bind] 0-socket.management: binding to failed: Address already in use
[2017-08-06 03:16:12.489485] E [socket.c:799:__socket_server_bind] 0-socket.management: Port is already in use
[2017-08-06 03:16:12.489503] W [rpcsvc.c:1667:rpcsvc_create_listener] 0-rpc-service: listening on transport failed
[2017-08-06 03:16:12.489515] E [MSGID: 106243] [glusterd.c:1686:init] 0-management: creation of listener failed
[2017-08-06 03:16:12.489535] E [MSGID: 101019] [xlator.c:503:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2017-08-06 03:16:12.489546] E [MSGID: 101066] [graph.c:325:glusterfs_graph_init] 0-management: initializing translator failed
[2017-08-06 03:16:12.489558] E [MSGID: 101176] [graph.c:696:glusterfs_graph_activate] 0-graph: init failed
[2017-08-06 03:16:12.490488] W [glusterfsd.c:1331:cleanup_and_exit] (-->glusterd(glusterfs_volumes_init+0xf7) [0x5604900d02c7] -->glusterd(glusterfs_process_volfp+0x183) [0x5604900d0183] -->glusterd(cleanup_and_exit+0x5a) [0x5604900ccb7a] ) 0-: received signum (-1), shutting down
[2017-08-06 03:16:14.605509] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:17.604803] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:20.608876] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:23.603836] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:26.607734] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:29.609353] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:32.609900] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:35.608544] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:14:38.612558] and [2017-08-06 03:16:35.625771]
[2017-08-06 03:16:38.609485] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:38.625303] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd.
[2017-08-06 03:16:41.610589] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:44.610563] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:47.610414] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:50.611649] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:53.609751] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:56.611203] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:16:59.613105] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:02.611623] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:05.611041] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:08.611022] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:11.614079] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:14.613345] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:17.616445] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:20.613447] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:23.614405] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:26.615806] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:29.616985] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:32.617296] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:35.617071] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:38.618057] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:41.613480] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:44.618631] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:47.618622] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:50.619449] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:53.619548] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:56.619893] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:17:59.620349] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:02.620491] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:05.622153] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:08.618072] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:11.620160] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:14.620953] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:17.623081] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:20.624232] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:23.623405] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:26.621556] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:29.622174] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:32.624101] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:35.624159] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:16:38.625303] and [2017-08-06 03:18:35.643628]
[2017-08-06 03:18:38.623154] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:38.638874] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd.
[2017-08-06 03:18:41.625459] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:44.625504] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:47.624487] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:50.625552] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:53.627476] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:56.626804] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:18:59.625783] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:02.625682] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:05.628513] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:08.629161] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:11.628698] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:14.627382] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:17.628452] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:20.624656] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:23.629982] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:26.630140] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:29.631511] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:32.631489] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:33.030413] I [MSGID: 106488] [glusterd-handler.c:1547:__glusterd_handle_cli_get_volume] 0-management: Received get vol req
[2017-08-06 03:19:35.630630] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:38.628652] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:41.633954] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:44.629301] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:47.634167] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:50.632232] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:53.630952] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:56.630006] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:19:56.645139] I [MSGID: 106499] [glusterd-handler.c:4302:__glusterd_handle_status_volume] 0-management: Received status volume req for volume gv2
[2017-08-06 03:19:59.635084] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:02.636198] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:05.635633] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:08.636366] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:11.637104] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:14.635935] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:17.638419] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:20.638921] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:23.638394] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:26.637497] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:29.638240] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:32.640077] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:35.639375] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
The message "I [MSGID: 106488] [glusterd-handler.c:1547:__glusterd_handle_cli_get_volume] 0-management: Received get vol req" repeated 2 times between [2017-08-06 03:19:33.030413] and [2017-08-06 03:19:33.032367]
The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:18:38.638874] and [2017-08-06 03:20:35.657497]
[2017-08-06 03:20:38.638998] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:38.655345] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd.
[2017-08-06 03:20:41.639249] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:44.641530] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:47.640078] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:50.639738] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:53.642584] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:56.639418] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:20:59.641479] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:02.641310] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:05.642192] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:08.643265] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:11.645152] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:14.643147] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:17.642636] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:20.644147] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:23.644211] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:26.645538] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:29.646314] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:32.644232] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:35.646143] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:38.646014] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:41.647145] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:44.645480] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:47.646920] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:50.645961] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:53.648376] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:56.647314] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:21:59.647240] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:02.649525] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:05.650677] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:08.649927] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:11.651573] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:14.650324] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:17.651756] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:20.648295] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:23.652112] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:26.651214] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:29.649616] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:32.653143] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:35.651358] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 39 times between [2017-08-06 03:20:38.655345] and [2017-08-06 03:22:35.667266]
[2017-08-06 03:22:38.653869] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:38.670345] I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd.
[2017-08-06 03:22:41.654410] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:44.652675] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:47.654514] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:50.654210] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:53.654301] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:56.655478] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:22:59.654460] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:02.657118] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:05.655457] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:08.656861] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:11.656538] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:14.657619] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:17.659736] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:20.655873] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:23.658645] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:26.657419] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:29.659674] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:32.659161] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:35.660123] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:38.660054] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:41.660019] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:44.659646] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:47.660895] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:50.661367] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:53.663192] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:56.662402] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:23:59.661518] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:24:02.663579] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:24:05.664634] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:24:08.663792] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:24:11.664684] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:24:14.663878] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:24:17.665416] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:24:20.664696] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:24:23.662762] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
[2017-08-06 03:24:26.663935] I [socket.c:2474:socket_event_handler] 0-transport: EPOLLERR - disconnecting now
The message "I [MSGID: 106005] [glusterd-handler.c:5846:__glusterd_brick_rpc_notify] 0-management: Brick taupo:/srv/gluster/gv2/brick1/gvol has disconnected from glusterd." repeated 36 times between [2017-08-06 03:22:38.670345] and [2017-08-06 03:24:26.680441]
[2017-08-06 03:24:28.820960] W [glusterfsd.c:1331:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x736d) [0x7f8adc8c036d] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xf5) [0x55ceb1ffcd85] -->/usr/sbin/glusterd(cleanup_and_exit+0x5a) [0x55ceb1ffcb7a] ) 0-: received signum (15), shutting down
More information about the Gluster-users
mailing list