[Gluster-users] 3 node 2 brick server keeps going offline. Runs excruciatingly slow when both bricks online
Kaamesh Kamalaaharan
kaamesh at novocraft.com
Thu Feb 26 00:58:56 UTC 2015
Hi guys,
My glusterfs setup is across 3 servers (gfs1 ,gfs2, hpc1) with 2 bricks (on
gfs1 and gfs2).
The reason i have 3 servers is that i wanted the gluster to stay online
when one server goes down as i frequently lose my gfs2 server. I have no
idea what causes this and im at a dead end.
I reinstalled glusterfs-server on my gfs2 server and i found that my
performance dropped to the point my R/W speed is around 1.5 kB/S on my 10G
LAN . Pressing tab when i ls into a folder takes around 20-30 seconds to
autocomplete, sometimes longer. Previously i had blazing fast speeds. When
i take gluster2 off the network, the speed is back up.
Both my servers are now on the fritz and the bricks keep going offline and
i have no idea what to do. Can anyone help shed some light on this? I have
no idea what i need to provide so ill provide the logs and config.
::EDIT::
As i was typing, gluster went down again and when i brought it back up, the
speed went back to normal. This didnt happen the last 3 or four times i
restarted the servers.. I would still like to know what is going on...
::::::::
gluster volume info
Volume Name: gfsvolume
Type: Replicate
Volume ID: a29bd2fb-b1ef-4481-be10-c2f4faf4059b
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gfs1:/export/sda/brick
Brick2: gfs2:/export/sda/brick
Options Reconfigured:
performance.quick-read: off
network.ping-timeout: 30
network.frame-timeout: 90
performance.cache-max-file-size: 2MB
cluster.server-quorum-type: none
nfs.addr-namelookup: off
nfs.trusted-write: off
performance.write-behind-window-size: 4MB
cluster.data-self-heal-algorithm: diff
performance.cache-refresh-timeout: 60
performance.cache-size: 6442450944
cluster.quorum-type: fixed
auth.allow: 172.*
cluster.quorum-count: 1
cluster.server-quorum-ratio: 50%
/var/log//glusterfs/
[2015-02-26 00:13:36.207408] I
[glusterd-volume-ops.c:481:__glusterd_handle_cli_heal_volume] 0-management:
Received heal vol req for volume gfsvolume
[2015-02-26 00:13:36.211810] E [glusterd-syncop.c:101:gd_collate_errors]
0-: Commit failed on gfs2. Please check log file for details.
[2015-02-26 00:14:17.939391] W [glusterfsd.c:1095:cleanup_and_exit]
(-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f13408f8e6d]
(-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f1340fa6b50] (-->/us
r/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f1342832d55]))) 0-: received
signum (15), shutting down
[2015-02-26 00:14:20.127606] I [glusterfsd.c:1959:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0
(/usr/sbin/glusterd -p /var/run/glusterd.pid)
[2015-02-26 00:14:20.136366] I [glusterd.c:1148:init] 0-management: Using
/var/lib/glusterd as working directory
[2015-02-26 00:14:20.137768] I [socket.c:3561:socket_init]
0-socket.management: SSL support is NOT enabled
[2015-02-26 00:14:20.137797] I [socket.c:3576:socket_init]
0-socket.management: using system polling thread
[2015-02-26 00:14:20.139452] W [rdma.c:4194:__gf_rdma_ctx_create]
0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
[2015-02-26 00:14:20.139483] E [rdma.c:4482:init] 0-rdma.management: Failed
to initialize IB Device
[2015-02-26 00:14:20.139497] E [rpc-transport.c:333:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2015-02-26 00:14:20.139568] W [rpcsvc.c:1483:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2015-02-26 00:14:20.139719] I [socket.c:3561:socket_init]
0-socket.management: SSL support is NOT enabled
[2015-02-26 00:14:20.139741] I [socket.c:3576:socket_init]
0-socket.management: using system polling thread
[2015-02-26 00:14:23.351872] I
[glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 2
[2015-02-26 00:14:23.355502] E
[glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-0
[2015-02-26 00:14:23.355552] E
[glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-1
[2015-02-26 00:14:23.584714] I
[glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect
returned 0
[2015-02-26 00:14:23.588106] I
[glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect
returned 0
[2015-02-26 00:14:23.588223] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:23.588328] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled
[2015-02-26 00:14:23.588347] I [socket.c:3576:socket_init] 0-management:
using system polling thread
[2015-02-26 00:14:23.589290] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:23.589378] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled
[2015-02-26 00:14:23.589397] I [socket.c:3576:socket_init] 0-management:
using system polling thread
[2015-02-26 00:14:23.589885] I [glusterd.c:138:glusterd_uuid_init]
0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option transport.socket.listen-backlog 128
7: option transport.socket.read-fail-log off
8: option transport.socket.keepalive-interval 2
9: option transport.socket.keepalive-time 10
10: option transport-type rdma
11: option working-directory /var/lib/glusterd
12: end-volume
13:
+------------------------------------------------------------------------------+
[2015-02-26 00:14:23.780644] I
[glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0
[2015-02-26 00:14:23.798102] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:23.798242] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled
[2015-02-26 00:14:23.798276] I [socket.c:3576:socket_init] 0-management:
using system polling thread
[2015-02-26 00:14:24.800202] E
[glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management:
Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error:
Permission denied
[2015-02-26 00:14:24.800858] I
[glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered
MOUNTV3 successfully
[2015-02-26 00:14:24.801261] I
[glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered
MOUNTV1 successfully
[2015-02-26 00:14:24.801649] I
[glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered
NFSV3 successfully
[2015-02-26 00:14:24.802028] I
[glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM
v4 successfully
[2015-02-26 00:14:24.802410] I
[glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM
v1 successfully
[2015-02-26 00:14:24.802793] I
[glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL
v3 successfully
[2015-02-26 00:14:24.806276] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:24.806387] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled
[2015-02-26 00:14:24.806406] I [socket.c:3576:socket_init] 0-management:
using system polling thread
[2015-02-26 00:14:25.807341] E
[glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management:
Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No
such file or directory
[2015-02-26 00:14:25.810987] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:25.811100] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled[2015-02-26 00:14:25.811119] I
[socket.c:3576:socket_init] 0-management: using system polling thread
[2015-02-26 00:14:25.811329] I
[glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:25.811370] I
[glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
[2015-02-26 00:14:25.811391] I
[glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my
uuid as Friend
[2015-02-26 00:14:25.811407] I
[glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1
[2015-02-26 00:14:25.811510] I
[glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 2
[2015-02-26 00:14:25.982101] I [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2015-02-26 00:14:25.982172] I [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2015-02-26 00:14:25.982265] I
[glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0
[2015-02-26 00:14:26.003332] I [glusterd-pmap.c:227:pmap_registry_bind]
0-pmap: adding brick /export/sda/brick on port 49153
[2015-02-26 00:14:26.159051] I
[glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:26.159112] I
[glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
[2015-02-26 00:14:26.159130] I
[glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my
uuid as Friend
[2015-02-26 00:14:26.178634] I
[glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:26.178794] I
[glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to gfs2 (0), ret: 0
[2015-02-26 00:14:26.194289] I
[glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
[2015-02-26 00:14:26.194348] I
[glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
[2015-02-26 00:14:26.208631] I
[glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:26.373031] I
[glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:26.373101] I
[glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 2
[2015-02-26 00:14:26.385510] I
[glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:26.385645] I
[glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to hpc1 (0), ret: 0
[2015-02-26 00:14:26.399963] I
[glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
[2015-02-26 00:14:26.400010] I
[glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
[2015-02-26 00:14:26.414448] I
[glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:26.668753] I
[glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:32.167298] W [socket.c:522:__socket_rwv] 0-management:
readv on 172.20.20.22:24007 failed (No data available)
[2015-02-26 00:14:33.590941] E [socket.c:2161:socket_connect_finish]
0-management: connection to 172.20.20.22:24007 failed (Connection refused)
[2015-02-26 00:14:39.060572] W [glusterfsd.c:1095:cleanup_and_exit]
(-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f727130ce6d]
(-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f72719bab50]
(-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f7273246d55]))) 0-:
received signum (15), shutting down
[2015-02-26 00:14:42.990079] I [glusterfsd.c:1959:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0
(/usr/sbin/glusterd -p /var/run/glusterd.pid)
[2015-02-26 00:14:42.994671] I [glusterd.c:1148:init] 0-management: Using
/var/lib/glusterd as working directory
[2015-02-26 00:14:42.996068] I [socket.c:3561:socket_init]
0-socket.management: SSL support is NOT enabled
[2015-02-26 00:14:42.996098] I [socket.c:3576:socket_init]
0-socket.management: using system polling thread
[2015-02-26 00:14:42.996892] W [rdma.c:4194:__gf_rdma_ctx_create]
0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
[2015-02-26 00:14:42.996919] E [rdma.c:4482:init] 0-rdma.management: Failed
to initialize IB Device
[2015-02-26 00:14:42.996932] E [rpc-transport.c:333:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2015-02-26 00:14:42.997004] W [rpcsvc.c:1483:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2015-02-26 00:14:42.997138] I [socket.c:3561:socket_init]
0-socket.management: SSL support is NOT enabled
[2015-02-26 00:14:42.997156] I [socket.c:3576:socket_init]
0-socket.management: using system polling thread
[2015-02-26 00:14:46.319737] I
[glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 2
[2015-02-26 00:14:46.323519] E
[glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-0
[2015-02-26 00:14:46.323567] E
[glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key:
brick-1
[2015-02-26 00:14:46.561399] I
[glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect
returned 0
[2015-02-26 00:14:46.564795] I
[glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect
returned 0
[2015-02-26 00:14:46.564919] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:46.565035] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled
[2015-02-26 00:14:46.565054] I [socket.c:3576:socket_init] 0-management:
using system polling thread
[2015-02-26 00:14:46.566216] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:46.566329] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled
[2015-02-26 00:14:46.566348] I [socket.c:3576:socket_init] 0-management:
using system polling thread
[2015-02-26 00:14:46.566832] I [glusterd.c:138:glusterd_uuid_init]
0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option transport.socket.listen-backlog 128
7: option transport.socket.read-fail-log off
8: option transport.socket.keepalive-interval 2
9: option transport.socket.keepalive-time 10
10: option transport-type rdma
11: option working-directory /var/lib/glusterd
12: end-volume
13:
+------------------------------------------------------------------------------+
[2015-02-26 00:14:46.570634] E [socket.c:2161:socket_connect_finish]
0-management: connection to 172.20.20.22:24007 failed (Connection refused)
[2015-02-26 00:14:46.571244] I
[glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 2
[2015-02-26 00:14:47.001039] I
[glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:47.001183] I
[glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to hpc1 (0), ret: 0
[2015-02-26 00:14:47.015772] I
[glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
[2015-02-26 00:14:47.015820] I
[glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
[2015-02-26 00:14:47.030090] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:47.030178] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled
[2015-02-26 00:14:47.030196] I [socket.c:3576:socket_init] 0-management:
using system polling thread
[2015-02-26 00:14:48.031850] E
[glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management:
Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error:
Permission denied
[2015-02-26 00:14:48.032392] I
[glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered
MOUNTV3 successfully
[2015-02-26 00:14:48.032794] I
[glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered
MOUNTV1 successfully
[2015-02-26 00:14:48.033182] I
[glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered
NFSV3 successfully
[2015-02-26 00:14:48.033571] I
[glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM
v4 successfully
[2015-02-26 00:14:48.033964] I
[glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM
v1 successfully
[2015-02-26 00:14:48.034356] I
[glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL
v3 successfully
[2015-02-26 00:14:48.037872] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:48.037982] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled
[2015-02-26 00:14:48.038001] I [socket.c:3576:socket_init] 0-management:
using system polling thread
[2015-02-26 00:14:49.039002] E
[glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management:
Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No
such file or directory
[2015-02-26 00:14:49.042564] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-02-26 00:14:49.042670] I [socket.c:3561:socket_init] 0-management:
SSL support is NOT enabled
[2015-02-26 00:14:49.042690] I [socket.c:3576:socket_init] 0-management:
using system polling thread
[2015-02-26 00:14:49.042958] I
[glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:49.042998] I
[glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
[2015-02-26 00:14:49.043013] I
[glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my
uuid as Friend
[2015-02-26 00:14:49.043319] I
[glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0
[2015-02-26 00:14:49.204554] I [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2015-02-26 00:14:49.204657] I [socket.c:2238:socket_event_handler]
0-transport: disconnecting now
[2015-02-26 00:14:49.204717] I
[glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:49.814028] I [glusterd-pmap.c:227:pmap_registry_bind]
0-pmap: adding brick /export/sda/brick on port 49153
[2015-02-26 00:14:53.678347] I
[glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management:
using the op-version 2
[2015-02-26 00:14:53.848276] I
[glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:53.848407] I
[glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to gfs2 (0), ret: 0
[2015-02-26 00:14:53.862892] I
[glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
[2015-02-26 00:14:53.862952] I
[glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
[2015-02-26 00:14:53.877685] I
[glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
[2015-02-26 00:14:55.217486] I
[glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0
[2015-02-26 00:14:55.234370] I
[glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c
[2015-02-26 00:14:55.234433] I
[glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
[2015-02-26 00:14:55.234451] I
[glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my
uuid as Friend
[2015-02-26 00:14:55.234468] I
[glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1
[2015-02-26 00:15:14.637613] E
[glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management:
Volume gfsvolume already started
[2015-02-26 00:15:14.637661] E [glusterd-syncop.c:912:gd_stage_op_phase]
0-management: Staging of operation 'Volume Start' failed on localhost :
Volume gfsvolume already started
[2015-02-26 00:24:41.265078] W [socket.c:522:__socket_rwv] 0-management:
readv on 172.20.20.22:24007 failed (No data available)
[2015-02-26 00:24:43.631739] E [socket.c:2161:socket_connect_finish]
0-management: connection to 172.20.20.22:24007 failed (Connection refused)
[2015-02-26 00:24:56.174790] W [socket.c:522:__socket_rwv] 0-management:
readv on /var/run/91bfef953906a83fc14aa435a886ac4d.socket failed (No data
available)
[2015-02-26 00:24:56.175299] I
[glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management:
Disconnected from gfs1:/export/sda/brick
[2015-02-26 00:25:24.076560] W
[glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx
modification failed
[2015-02-26 00:25:24.078109] I
[glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management:
Received status volume req for volume gfsvolume
[2015-02-26 00:25:31.041343] E
[glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management:
Volume gfsvolume already started
[2015-02-26 00:25:31.041404] E [glusterd-syncop.c:912:gd_stage_op_phase]
0-management: Staging of operation 'Volume Start' failed on localhost :
Volume gfsvolume already started
[2015-02-26 00:25:43.389260] W [glusterfsd.c:1095:cleanup_and_exit]
(-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8f8cfa7e6d]
(-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f8f8d655b50]
(-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f8f8eee1d55]))) 0-:
received signum (15), shutting down
[2015-02-26 00:25:45.437325] I [glusterfsd.c:1959:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0
(/usr/sbin/glusterd -p /var/run/glusterd.pid)
[2015-02-26 00:25:45.487307] I [glusterd.c:1148:init] 0-management: Using
/var/lib/glusterd as working directory
[2015-02-26 00:25:45.488734] I [socket.c:3561:socket_init]
0-socket.management: SSL support is NOT enabled
[2015-02-26 00:25:45.488765] I [socket.c:3576:socket_init]
0-socket.management: using system polling thread
[2015-02-26 00:25:45.489557] W [rdma.c:4194:__gf_rdma_ctx_create]
0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
[2015-02-26 00:25:45.489584] E [rdma.c:4482:init] 0-rdma.management: Failed
to initialize IB Device
[2015-02-26 00:25:45.489597] E [rpc-transport.c:333:rpc_transport_load]
0-rpc-transport: 'rdma' initialization failed
[2015-02-26 00:25:45.489668] W [rpcsvc.c:1483:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2015-02-26 00:25:45.489799] I [socket.c:3561:socket_init]
0-socket.management: SSL support is NOT enabled
[2015-02-26 00:25:45.489818] I [socket.c:3576:socket_init]
0-socket.management: using system polling thread
[2015-02-26 00:25:48.706235] I
[glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 2
Please Do let me know if there is anything else i can provide to help.
Apologies if this is a simple fix but i have never found an answer after a
week of searching.
Thank You Kindly,
Kaamesh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150226/61854e00/attachment.html>
More information about the Gluster-users
mailing list