[Gluster-users] glusterfs crash when the one of replicate node restart

Wed Dec 14 09:36:59 UTC 2011

Hi,we have use glusterfs for two years. After upgraded to 3.2.5,we discover
that when one of replicate node reboot and startup the glusterd daemon,the
gluster will crash cause by the other

replicate node cpu usage reach 100%.

Our gluster info:

Type: Distributed-Replicate
Status: Started
Number of Bricks: 5 x 2 = 10
Transport-type: tcp
Options Reconfigured:
performance.cache-size: 3GB
performance.cache-max-file-size: 512KB
network.frame-timeout: 30
network.ping-timeout: 25
cluster.min-free-disk: 10%

Our device：

Dell R710
600Gsas *6
3*8Gmem

The error info:

[2011-12-14 13:24:10.483812] E [rdma.c:4813:init] 0-rdma.management: Failed
to initialize IB Device
[2011-12-14 13:24:10.483828] E [rpc-transport.c:742:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2011-12-14 13:24:10.483841] W [rpcsvc.c:1288:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2011-12-14 13:24:11.967621] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-0
[2011-12-14 13:24:11.967665] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-1
[2011-12-14 13:24:11.967681] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-2
[2011-12-14 13:24:11.967695] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-3
[2011-12-14 13:24:11.967709] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-4
[2011-12-14 13:24:11.967723] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-5
[2011-12-14 13:24:11.967736] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-6
[2011-12-14 13:24:11.967750] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-7
[2011-12-14 13:24:11.967764] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-8
[2011-12-14 13:24:11.967777] E
[glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-9
[2011-12-14 13:24:12.465565] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.17:1013)
[2011-12-14 13:24:12.465623] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.8:1013)
[2011-12-14 13:24:12.465656] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.10:1013)
[2011-12-14 13:24:12.465686] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.11:1013)
[2011-12-14 13:24:12.465716] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.125:1013)
[2011-12-14 13:24:12.633288] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.65:1006)
[2011-12-14 13:24:13.138150] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.1:1013)
[2011-12-14 13:24:13.284665] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.3:1013)
[2011-12-14 13:24:15.790805] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.8:1013)
[2011-12-14 13:24:16.113430] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.125:1013)
[2011-12-14 13:24:16.259040] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.10:1013)
[2011-12-14 13:24:16.392058] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.17:1013)
[2011-12-14 13:24:16.429444] W [socket.c:1494:__socket_proto_state_machine]
0-socket.management: reading from socket failed. Error (Transport endpoint
is not connected), peer (10.1.1.11:1013)
[2011-12-14 13:26:05.787680] W [glusterfsd.c:727:cleanup_and_exit]
(-->/lib64/libc.so.6(clone+0x6d) [0x37c8ed3c2d] (-->/lib64/libpthread.so.0
[0x37c96064a7]
(-->/opt/glusterfs/3.2.5/sbin/glusterd(glusterfs_sigwaiter+0x17c)
[0x40477c]))) 0-: received signum (15), shutting down

-- 

Regards,

Cocl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20111214/4efe1c65/attachment.html>
-------------- next part --------------
[2011-12-14 13:24:10.381798] I [glusterfsd.c:1493:main] 0-/opt/glusterfs/3.2.5/sbin/glusterd: Started running /opt/glusterfs/3.2.5/sbin/glusterd version 3.2.5
[2011-12-14 13:24:10.456111] I [glusterd.c:550:init] 0-management: Using /etc/glusterd as working directory
[2011-12-14 13:24:10.483696] C [rdma.c:3934:rdma_init] 0-rpc-transport/rdma: Failed to get IB devices
[2011-12-14 13:24:10.483812] E [rdma.c:4813:init] 0-rdma.management: Failed to initialize IB Device
[2011-12-14 13:24:10.483828] E [rpc-transport.c:742:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2011-12-14 13:24:10.483841] W [rpcsvc.c:1288:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2011-12-14 13:24:10.492780] I [glusterd.c:88:glusterd_uuid_init] 0-glusterd: retrieved UUID: 18ce0543-24f2-4e8f-a13d-3d5a4a7dff1b
[2011-12-14 13:24:11.967621] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2011-12-14 13:24:11.967665] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2011-12-14 13:24:11.967681] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2011-12-14 13:24:11.967695] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3
[2011-12-14 13:24:11.967709] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-4
[2011-12-14 13:24:11.967723] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-5
[2011-12-14 13:24:11.967736] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-6
[2011-12-14 13:24:11.967750] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-7
[2011-12-14 13:24:11.967764] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-8
[2011-12-14 13:24:11.967777] E [glusterd-store.c:1820:glusterd_store_retrieve_volume] 0-: Unknown key: brick-9
[2011-12-14 13:24:12.318700] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-12-14 13:24:12.325662] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-12-14 13:24:12.333999] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-12-14 13:24:12.342100] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-12-14 13:24:12.348505] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-12-14 13:24:12.355102] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-12-14 13:24:12.361994] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-12-14 13:24:12.368569] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-12-14 13:24:12.377832] I [glusterd-handler.c:3409:glusterd_friend_add] 0-glusterd: connect returned 0
[2011-12-14 13:24:12.383343] I [glusterd-utils.c:1097:glusterd_volume_start_glusterfs] 0-: About to start glusterfs for brick x.x.x.x:/data0
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /etc/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.keepalive-time 10
  6:     option transport.socket.keepalive-interval 2
  7: end-volume
  8: 

+------------------------------------------------------------------------------+
[2011-12-14 13:24:12.462383] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-12-14 13:24:12.462428] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-12-14 13:24:12.462457] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-12-14 13:24:12.462505] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-12-14 13:24:12.462549] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-12-14 13:24:12.462608] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-12-14 13:24:12.462637] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-12-14 13:24:12.462663] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-12-14 13:24:12.462740] I [glusterd-handshake.c:317:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd clnt mgmt, Num (1238433), Version (1)
[2011-12-14 13:24:12.465565] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.17:1013)
[2011-12-14 13:24:12.465623] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.8:1013)
[2011-12-14 13:24:12.465656] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.x.x.x)
[2011-12-14 13:24:12.465686] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.11:1013)
[2011-12-14 13:24:12.465716] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.125:1013)
[2011-12-14 13:24:12.469566] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd
[2011-12-14 13:24:12.469691] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0
[2011-12-14 13:24:12.469850] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64
[2011-12-14 13:24:12.469873] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23
[2011-12-14 13:24:12.469889] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24
[2011-12-14 13:24:12.469904] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67
[2011-12-14 13:24:12.469919] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68
[2011-12-14 13:24:12.469934] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28
[2011-12-14 13:24:12.469950] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27
[2011-12-14 13:24:12.469965] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25
[2011-12-14 13:24:12.469980] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26
[2011-12-14 13:24:12.470508] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.470714] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.474314] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22
[2011-12-14 13:24:12.474424] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0
[2011-12-14 13:24:12.474575] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64
[2011-12-14 13:24:12.474608] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23
[2011-12-14 13:24:12.474625] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24
[2011-12-14 13:24:12.474640] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67
[2011-12-14 13:24:12.474655] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68
[2011-12-14 13:24:12.474670] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28
[2011-12-14 13:24:12.474685] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27
[2011-12-14 13:24:12.474701] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25
[2011-12-14 13:24:12.474716] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26
[2011-12-14 13:24:12.475100] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.475305] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.477877] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 493352a0-912c-4298-b987-a0f039559f45
[2011-12-14 13:24:12.477977] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0
[2011-12-14 13:24:12.478101] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64
[2011-12-14 13:24:12.478122] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23
[2011-12-14 13:24:12.478138] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24
[2011-12-14 13:24:12.478153] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67
[2011-12-14 13:24:12.478167] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68
[2011-12-14 13:24:12.478183] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28
[2011-12-14 13:24:12.478198] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27
[2011-12-14 13:24:12.478213] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25
[2011-12-14 13:24:12.478228] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26
[2011-12-14 13:24:12.478609] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.479016] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.480496] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad
[2011-12-14 13:24:12.480622] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0
[2011-12-14 13:24:12.480746] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64
[2011-12-14 13:24:12.480767] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23
[2011-12-14 13:24:12.480783] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24
[2011-12-14 13:24:12.480806] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67
[2011-12-14 13:24:12.480822] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68
[2011-12-14 13:24:12.480837] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28
[2011-12-14 13:24:12.480852] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27
[2011-12-14 13:24:12.480868] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25
[2011-12-14 13:24:12.480883] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26
[2011-12-14 13:24:12.481550] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.481709] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.484275] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.484546] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.484774] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.484981] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.485041] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.485235] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.485453] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.485681] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.492618] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.492965] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.493040] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.493184] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.493382] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.493430] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.493751] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.493782] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.503875] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.504289] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.504573] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.504718] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.518166] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.518803] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.519241] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.519625] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.522356] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.522430] I [glusterd-pmap.c:237:pmap_registry_bind] 0-pmap: adding brick /data0 on port 24009
[2011-12-14 13:24:12.522699] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.522849] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.523066] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.625851] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672
[2011-12-14 13:24:12.625974] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0
[2011-12-14 13:24:12.626124] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64
[2011-12-14 13:24:12.626146] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23
[2011-12-14 13:24:12.626163] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24
[2011-12-14 13:24:12.626178] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67
[2011-12-14 13:24:12.626193] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68
[2011-12-14 13:24:12.626208] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28
[2011-12-14 13:24:12.626223] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27
[2011-12-14 13:24:12.626239] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25
[2011-12-14 13:24:12.626254] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26
[2011-12-14 13:24:12.626763] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.626805] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.626829] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.626860] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.626884] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.626905] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.626926] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.626947] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.633288] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.65:1006)
[2011-12-14 13:24:12.638589] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.731673] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3
[2011-12-14 13:24:12.731784] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0
[2011-12-14 13:24:12.731930] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64
[2011-12-14 13:24:12.731962] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23
[2011-12-14 13:24:12.731979] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24
[2011-12-14 13:24:12.731994] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67
[2011-12-14 13:24:12.732009] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68
[2011-12-14 13:24:12.732024] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28
[2011-12-14 13:24:12.732039] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27
[2011-12-14 13:24:12.732055] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25
[2011-12-14 13:24:12.732070] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26
[2011-12-14 13:24:12.732607] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.732659] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.732687] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.732716] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.732742] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.732767] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.732792] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.732981] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.758831] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.929723] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884
[2011-12-14 13:24:12.929832] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0
[2011-12-14 13:24:12.929970] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64
[2011-12-14 13:24:12.929992] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23
[2011-12-14 13:24:12.930009] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24
[2011-12-14 13:24:12.930024] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67
[2011-12-14 13:24:12.930039] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68
[2011-12-14 13:24:12.930054] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28
[2011-12-14 13:24:12.930069] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27
[2011-12-14 13:24:12.930084] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25
[2011-12-14 13:24:12.930099] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26
[2011-12-14 13:24:12.930570] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.930644] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.930676] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.930701] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.930726] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.930750] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.930777] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.930804] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:12.979192] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.65011] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36
[2011-12-14 13:24:13.65118] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0
[2011-12-14 13:24:13.65247] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64
[2011-12-14 13:24:13.65269] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23
[2011-12-14 13:24:13.65285] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24
[2011-12-14 13:24:13.65300] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67
[2011-12-14 13:24:13.65314] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68
[2011-12-14 13:24:13.65329] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28
[2011-12-14 13:24:13.65344] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27
[2011-12-14 13:24:13.65360] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25
[2011-12-14 13:24:13.65375] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26
[2011-12-14 13:24:13.65931] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.65973] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.65999] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.66024] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.66048] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.66072] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.66097] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.66120] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.82588] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.138150] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.1:1013)
[2011-12-14 13:24:13.275230] I [glusterd-handler.c:2807:glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c
[2011-12-14 13:24:13.275358] I [glusterd-handler.c:3577:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to x.x.x65 (0), ret: 0
[2011-12-14 13:24:13.275504] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 2a1e14cb-4d97-49a9-9d15-6f0ddc10b672, host: x.x.x64
[2011-12-14 13:24:13.275558] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: b9aa7d78-f7ee-4157-9783-cd1a625722a3, host: x.x.x23
[2011-12-14 13:24:13.275578] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: cb03fb74-19ad-4002-a205-ba7457eb83bd, host: x.x.x24
[2011-12-14 13:24:13.275593] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 1c577861-3754-4bc8-9f15-f3eee3fa8e36, host: x.x.x67
[2011-12-14 13:24:13.275608] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 493352a0-912c-4298-b987-a0f039559f45, host: x.x.x68
[2011-12-14 13:24:13.275623] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: fcc0a7ab-593c-4316-b8cf-7bb3acc22884, host: x.x.x28
[2011-12-14 13:24:13.275638] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 3bd33875-dd3a-4c06-b168-e7273dd5dd22, host: x.x.x27
[2011-12-14 13:24:13.275654] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: c2df4016-02f0-4739-bfe7-0a08c1db581c, host: x.x.x25
[2011-12-14 13:24:13.275669] I [glusterd-sm.c:492:glusterd_ac_send_friend_update] 0-: Added uuid: 218083a1-2087-420a-a8ea-30e8fd2eadad, host: x.x.x26
[2011-12-14 13:24:13.276064] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.276105] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.276130] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.276155] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.276188] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.276262] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.276305] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.276330] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:13.284665] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.3:1013)
[2011-12-14 13:24:13.300639] I [glusterd-rpc-ops.c:722:glusterd3_1_friend_update_cbk] 0-glusterd: Received ACC from uuid: 
[2011-12-14 13:24:15.790805] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.8:1013)
[2011-12-14 13:24:16.113430] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.125:1013)
[2011-12-14 13:24:16.259040] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.x.x.x)
[2011-12-14 13:24:16.392058] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.17:1013)
[2011-12-14 13:24:16.429444] W [socket.c:1494:__socket_proto_state_machine] 0-socket.management: reading from socket failed. Error (Transport endpoint is not connected), peer (1
0.1.1.11:1013)
[2011-12-14 13:26:05.787680] W [glusterfsd.c:727:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x37c8ed3c2d] (-->/lib64/libpthread.so.0 [0x37c96064a7] (-->/opt/glusterfs/3
.2.5/sbin/glusterd(glusterfs_sigwaiter+0x17c) [0x40477c]))) 0-: received signum (15), shutting down