[Gluster-users] 3 node 2 brick server keeps going offline. Runs excruciatingly slow when both bricks online
Kaamesh Kamalaaharan
kaamesh at novocraft.com
Thu Feb 26 01:07:28 UTC 2015
Update
The speed is back down, but when the gfs is mounted via nfs instead of
through the gluster client, i get good speeds
Thank You Kindly,
Kaamesh
Bioinformatician
Novocraft Technologies Sdn Bhd
C-23A-05, 3 Two Square, Section 19, 46300 Petaling Jaya
Selangor Darul Ehsan
Malaysia
Mobile: +60176562635
Ph: +60379600541
Fax: +60379600540
On Thu, Feb 26, 2015 at 8:58 AM, Kaamesh Kamalaaharan <kaamesh at novocraft.com
> wrote:
> Hi guys,
>
> My glusterfs setup is across 3 servers (gfs1 ,gfs2, hpc1) with 2 bricks
> (on gfs1 and gfs2).
>
> The reason i have 3 servers is that i wanted the gluster to stay online
> when one server goes down as i frequently lose my gfs2 server. I have no
> idea what causes this and im at a dead end.
>
> I reinstalled glusterfs-server on my gfs2 server and i found that my
> performance dropped to the point my R/W speed is around 1.5 kB/S on my 10G
> LAN . Pressing tab when i ls into a folder takes around 20-30 seconds to
> autocomplete, sometimes longer. Previously i had blazing fast speeds. When
> i take gluster2 off the network, the speed is back up.
>
> Both my servers are now on the fritz and the bricks keep going offline and
> i have no idea what to do. Can anyone help shed some light on this? I have
> no idea what i need to provide so ill provide the logs and config.
>
>
> ::EDIT::
> As i was typing, gluster went down again and when i brought it back up,
> the speed went back to normal. This didnt happen the last 3 or four times i
> restarted the servers.. I would still like to know what is going on...
> ::::::::
>
>
>
> gluster volume info
>
> Volume Name: gfsvolume
> Type: Replicate
> Volume ID: a29bd2fb-b1ef-4481-be10-c2f4faf4059b
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: gfs1:/export/sda/brick
> Brick2: gfs2:/export/sda/brick
> Options Reconfigured:
> performance.quick-read: off
> network.ping-timeout: 30
> network.frame-timeout: 90
> performance.cache-max-file-size: 2MB
> cluster.server-quorum-type: none
> nfs.addr-namelookup: off
> nfs.trusted-write: off
> performance.write-behind-window-size: 4MB
> cluster.data-self-heal-algorithm: diff
> performance.cache-refresh-timeout: 60
> performance.cache-size: 6442450944
> cluster.quorum-type: fixed
> auth.allow: 172.*
> cluster.quorum-count: 1
> cluster.server-quorum-ratio: 50%
>
> /var/log//glusterfs/
>
>
>
> [2015-02-26 00:13:36.207408] I
> [glusterd-volume-ops.c:481:__glusterd_handle_cli_heal_volume] 0-management:
> Received heal vol req for volume gfsvolume
>
> [2015-02-26 00:13:36.211810] E [glusterd-syncop.c:101:gd_collate_errors]
> 0-: Commit failed on gfs2. Please check log file for details.
> [2015-02-26 00:14:17.939391] W [glusterfsd.c:1095:cleanup_and_exit]
> (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f13408f8e6d]
> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f1340fa6b50] (-->/us
> r/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f1342832d55]))) 0-: received
> signum (15), shutting down
> [2015-02-26 00:14:20.127606] I [glusterfsd.c:1959:main]
> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0
> (/usr/sbin/glusterd -p /var/run/glusterd.pid)
> [2015-02-26 00:14:20.136366] I [glusterd.c:1148:init] 0-management: Using
> /var/lib/glusterd as working directory
> [2015-02-26 00:14:20.137768] I [socket.c:3561:socket_init]
> 0-socket.management: SSL support is NOT enabled
> [2015-02-26 00:14:20.137797] I [socket.c:3576:socket_init]
> 0-socket.management: using system polling thread
> [2015-02-26 00:14:20.139452] W [rdma.c:4194:__gf_rdma_ctx_create]
> 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
> [2015-02-26 00:14:20.139483] E [rdma.c:4482:init] 0-rdma.management:
> Failed to initialize IB Device
> [2015-02-26 00:14:20.139497] E [rpc-transport.c:333:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
> [2015-02-26 00:14:20.139568] W [rpcsvc.c:1483:rpcsvc_transport_create]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2015-02-26 00:14:20.139719] I [socket.c:3561:socket_init]
> 0-socket.management: SSL support is NOT enabled
> [2015-02-26 00:14:20.139741] I [socket.c:3576:socket_init]
> 0-socket.management: using system polling thread
> [2015-02-26 00:14:23.351872] I
> [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved
> op-version: 2
> [2015-02-26 00:14:23.355502] E
> [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-0
> [2015-02-26 00:14:23.355552] E
> [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-1
> [2015-02-26 00:14:23.584714] I
> [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect
> returned 0
> [2015-02-26 00:14:23.588106] I
> [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect
> returned 0
> [2015-02-26 00:14:23.588223] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:23.588328] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled
> [2015-02-26 00:14:23.588347] I [socket.c:3576:socket_init] 0-management:
> using system polling thread
> [2015-02-26 00:14:23.589290] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:23.589378] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled
> [2015-02-26 00:14:23.589397] I [socket.c:3576:socket_init] 0-management:
> using system polling thread
> [2015-02-26 00:14:23.589885] I [glusterd.c:138:glusterd_uuid_init]
> 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070
> Final graph:
>
> +------------------------------------------------------------------------------+
> 1: volume management
> 2: type mgmt/glusterd
> 3: option rpc-auth.auth-glusterfs on
> 4: option rpc-auth.auth-unix on
> 5: option rpc-auth.auth-null on
> 6: option transport.socket.listen-backlog 128
> 7: option transport.socket.read-fail-log off
> 8: option transport.socket.keepalive-interval 2
> 9: option transport.socket.keepalive-time 10
> 10: option transport-type rdma
> 11: option working-directory /var/lib/glusterd
> 12: end-volume
> 13:
>
> +------------------------------------------------------------------------------+
> [2015-02-26 00:14:23.780644] I
> [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
> from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0
> [2015-02-26 00:14:23.798102] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:23.798242] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled
> [2015-02-26 00:14:23.798276] I [socket.c:3576:socket_init] 0-management:
> using system polling thread
> [2015-02-26 00:14:24.800202] E
> [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management:
> Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error:
> Permission denied
> [2015-02-26 00:14:24.800858] I
> [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered
> MOUNTV3 successfully
> [2015-02-26 00:14:24.801261] I
> [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered
> MOUNTV1 successfully
> [2015-02-26 00:14:24.801649] I
> [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered
> NFSV3 successfully
> [2015-02-26 00:14:24.802028] I
> [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM
> v4 successfully
> [2015-02-26 00:14:24.802410] I
> [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM
> v1 successfully
> [2015-02-26 00:14:24.802793] I
> [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL
> v3 successfully
> [2015-02-26 00:14:24.806276] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:24.806387] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled
> [2015-02-26 00:14:24.806406] I [socket.c:3576:socket_init] 0-management:
> using system polling thread
> [2015-02-26 00:14:25.807341] E
> [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management:
> Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No
> such file or directory
> [2015-02-26 00:14:25.810987] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:25.811100] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled[2015-02-26 00:14:25.811119] I
> [socket.c:3576:socket_init] 0-management: using system polling thread
> [2015-02-26 00:14:25.811329] I
> [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd:
> Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c
> [2015-02-26 00:14:25.811370] I
> [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
> uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
> [2015-02-26 00:14:25.811391] I
> [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my
> uuid as Friend
> [2015-02-26 00:14:25.811407] I
> [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
> uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1
> [2015-02-26 00:14:25.811510] I
> [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management:
> using the op-version 2
> [2015-02-26 00:14:25.982101] I [socket.c:2238:socket_event_handler]
> 0-transport: disconnecting now
> [2015-02-26 00:14:25.982172] I [socket.c:2238:socket_event_handler]
> 0-transport: disconnecting now
> [2015-02-26 00:14:25.982265] I
> [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
> from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0
> [2015-02-26 00:14:26.003332] I [glusterd-pmap.c:227:pmap_registry_bind]
> 0-pmap: adding brick /export/sda/brick on port 49153
> [2015-02-26 00:14:26.159051] I
> [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd:
> Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
> [2015-02-26 00:14:26.159112] I
> [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
> uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
> [2015-02-26 00:14:26.159130] I
> [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my
> uuid as Friend
> [2015-02-26 00:14:26.178634] I
> [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd:
> Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c
> [2015-02-26 00:14:26.178794] I
> [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd:
> Responded to gfs2 (0), ret: 0
> [2015-02-26 00:14:26.194289] I
> [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
> 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
> [2015-02-26 00:14:26.194348] I
> [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
> adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
> [2015-02-26 00:14:26.208631] I
> [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
> Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c
> [2015-02-26 00:14:26.373031] I
> [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
> Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
> [2015-02-26 00:14:26.373101] I
> [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management:
> using the op-version 2
> [2015-02-26 00:14:26.385510] I
> [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd:
> Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
> [2015-02-26 00:14:26.385645] I
> [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd:
> Responded to hpc1 (0), ret: 0
> [2015-02-26 00:14:26.399963] I
> [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
> 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
> [2015-02-26 00:14:26.400010] I
> [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
> adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
> [2015-02-26 00:14:26.414448] I
> [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
> Received ACC from uuid: adbb7505-3342-4c6d-be3d-75938633612c
> [2015-02-26 00:14:26.668753] I
> [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
> Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
> [2015-02-26 00:14:32.167298] W [socket.c:522:__socket_rwv] 0-management:
> readv on 172.20.20.22:24007 failed (No data available)
> [2015-02-26 00:14:33.590941] E [socket.c:2161:socket_connect_finish]
> 0-management: connection to 172.20.20.22:24007 failed (Connection refused)
> [2015-02-26 00:14:39.060572] W [glusterfsd.c:1095:cleanup_and_exit]
> (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f727130ce6d]
> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f72719bab50]
> (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f7273246d55]))) 0-:
> received signum (15), shutting down
> [2015-02-26 00:14:42.990079] I [glusterfsd.c:1959:main]
> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0
> (/usr/sbin/glusterd -p /var/run/glusterd.pid)
> [2015-02-26 00:14:42.994671] I [glusterd.c:1148:init] 0-management: Using
> /var/lib/glusterd as working directory
> [2015-02-26 00:14:42.996068] I [socket.c:3561:socket_init]
> 0-socket.management: SSL support is NOT enabled
> [2015-02-26 00:14:42.996098] I [socket.c:3576:socket_init]
> 0-socket.management: using system polling thread
> [2015-02-26 00:14:42.996892] W [rdma.c:4194:__gf_rdma_ctx_create]
> 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
> [2015-02-26 00:14:42.996919] E [rdma.c:4482:init] 0-rdma.management:
> Failed to initialize IB Device
> [2015-02-26 00:14:42.996932] E [rpc-transport.c:333:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
> [2015-02-26 00:14:42.997004] W [rpcsvc.c:1483:rpcsvc_transport_create]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2015-02-26 00:14:42.997138] I [socket.c:3561:socket_init]
> 0-socket.management: SSL support is NOT enabled
> [2015-02-26 00:14:42.997156] I [socket.c:3576:socket_init]
> 0-socket.management: using system polling thread
> [2015-02-26 00:14:46.319737] I
> [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved
> op-version: 2
> [2015-02-26 00:14:46.323519] E
> [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-0
> [2015-02-26 00:14:46.323567] E
> [glusterd-store.c:1979:glusterd_store_retrieve_volume] 0-: Unknown key:
> brick-1
> [2015-02-26 00:14:46.561399] I
> [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect
> returned 0
> [2015-02-26 00:14:46.564795] I
> [glusterd-handler.c:2912:glusterd_friend_add] 0-management: connect
> returned 0
> [2015-02-26 00:14:46.564919] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:46.565035] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled
> [2015-02-26 00:14:46.565054] I [socket.c:3576:socket_init] 0-management:
> using system polling thread
> [2015-02-26 00:14:46.566216] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:46.566329] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled
> [2015-02-26 00:14:46.566348] I [socket.c:3576:socket_init] 0-management:
> using system polling thread
> [2015-02-26 00:14:46.566832] I [glusterd.c:138:glusterd_uuid_init]
> 0-management: retrieved UUID: 49acc9c2-4809-4da5-a6f0-6a3d48314070
> Final graph:
>
> +------------------------------------------------------------------------------+
> 1: volume management
> 2: type mgmt/glusterd
> 3: option rpc-auth.auth-glusterfs on
> 4: option rpc-auth.auth-unix on
> 5: option rpc-auth.auth-null on
> 6: option transport.socket.listen-backlog 128
> 7: option transport.socket.read-fail-log off
> 8: option transport.socket.keepalive-interval 2
> 9: option transport.socket.keepalive-time 10
> 10: option transport-type rdma
> 11: option working-directory /var/lib/glusterd
> 12: end-volume
> 13:
>
> +------------------------------------------------------------------------------+
> [2015-02-26 00:14:46.570634] E [socket.c:2161:socket_connect_finish]
> 0-management: connection to 172.20.20.22:24007 failed (Connection refused)
> [2015-02-26 00:14:46.571244] I
> [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management:
> using the op-version 2
> [2015-02-26 00:14:47.001039] I
> [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd:
> Received probe from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
> [2015-02-26 00:14:47.001183] I
> [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd:
> Responded to hpc1 (0), ret: 0
> [2015-02-26 00:14:47.015772] I
> [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
> 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
> [2015-02-26 00:14:47.015820] I
> [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
> adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
> [2015-02-26 00:14:47.030090] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:47.030178] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled
> [2015-02-26 00:14:47.030196] I [socket.c:3576:socket_init] 0-management:
> using system polling thread
> [2015-02-26 00:14:48.031850] E
> [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management:
> Failed to remove /var/run/ec396f4c7328565d7dafb2c74d51d072.socket error:
> Permission denied
> [2015-02-26 00:14:48.032392] I
> [glusterd-utils.c:4155:glusterd_nfs_pmap_deregister] 0-: De-registered
> MOUNTV3 successfully
> [2015-02-26 00:14:48.032794] I
> [glusterd-utils.c:4160:glusterd_nfs_pmap_deregister] 0-: De-registered
> MOUNTV1 successfully
> [2015-02-26 00:14:48.033182] I
> [glusterd-utils.c:4165:glusterd_nfs_pmap_deregister] 0-: De-registered
> NFSV3 successfully
> [2015-02-26 00:14:48.033571] I
> [glusterd-utils.c:4170:glusterd_nfs_pmap_deregister] 0-: De-registered NLM
> v4 successfully
> [2015-02-26 00:14:48.033964] I
> [glusterd-utils.c:4175:glusterd_nfs_pmap_deregister] 0-: De-registered NLM
> v1 successfully
> [2015-02-26 00:14:48.034356] I
> [glusterd-utils.c:4180:glusterd_nfs_pmap_deregister] 0-: De-registered ACL
> v3 successfully
> [2015-02-26 00:14:48.037872] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:48.037982] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled
> [2015-02-26 00:14:48.038001] I [socket.c:3576:socket_init] 0-management:
> using system polling thread
> [2015-02-26 00:14:49.039002] E
> [glusterd-utils.c:4121:glusterd_nodesvc_unlink_socket_file] 0-management:
> Failed to remove /var/run/35e2381254b245fb5c6f66b8a82d585e.socket error: No
> such file or directory
> [2015-02-26 00:14:49.042564] I [rpc-clnt.c:972:rpc_clnt_connection_init]
> 0-management: setting frame-timeout to 600
> [2015-02-26 00:14:49.042670] I [socket.c:3561:socket_init] 0-management:
> SSL support is NOT enabled
> [2015-02-26 00:14:49.042690] I [socket.c:3576:socket_init] 0-management:
> using system polling thread
> [2015-02-26 00:14:49.042958] I
> [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd:
> Received friend update from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
> [2015-02-26 00:14:49.042998] I
> [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
> uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
> [2015-02-26 00:14:49.043013] I
> [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my
> uuid as Friend
> [2015-02-26 00:14:49.043319] I
> [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
> from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1, port: 0
> [2015-02-26 00:14:49.204554] I [socket.c:2238:socket_event_handler]
> 0-transport: disconnecting now
> [2015-02-26 00:14:49.204657] I [socket.c:2238:socket_event_handler]
> 0-transport: disconnecting now
> [2015-02-26 00:14:49.204717] I
> [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
> Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
> [2015-02-26 00:14:49.814028] I [glusterd-pmap.c:227:pmap_registry_bind]
> 0-pmap: adding brick /export/sda/brick on port 49153
> [2015-02-26 00:14:53.678347] I
> [glusterd-handshake.c:563:__glusterd_mgmt_hndsk_versions_ack] 0-management:
> using the op-version 2
> [2015-02-26 00:14:53.848276] I
> [glusterd-handler.c:2050:__glusterd_handle_incoming_friend_req] 0-glusterd:
> Received probe from uuid: adbb7505-3342-4c6d-be3d-75938633612c
> [2015-02-26 00:14:53.848407] I
> [glusterd-handler.c:3085:glusterd_xfer_friend_add_resp] 0-glusterd:
> Responded to gfs2 (0), ret: 0
> [2015-02-26 00:14:53.862892] I
> [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
> 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, host: hpc1
> [2015-02-26 00:14:53.862952] I
> [glusterd-sm.c:495:glusterd_ac_send_friend_update] 0-: Added uuid:
> adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2
> [2015-02-26 00:14:53.877685] I
> [glusterd-rpc-ops.c:553:__glusterd_friend_update_cbk] 0-management:
> Received ACC from uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae
> [2015-02-26 00:14:55.217486] I
> [glusterd-rpc-ops.c:356:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
> from uuid: adbb7505-3342-4c6d-be3d-75938633612c, host: gfs2, port: 0
> [2015-02-26 00:14:55.234370] I
> [glusterd-handler.c:2212:__glusterd_handle_friend_update] 0-glusterd:
> Received friend update from uuid: adbb7505-3342-4c6d-be3d-75938633612c
> [2015-02-26 00:14:55.234433] I
> [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
> uuid: 49acc9c2-4809-4da5-a6f0-6a3d48314070, hostname:gfs1
> [2015-02-26 00:14:55.234451] I
> [glusterd-handler.c:2266:__glusterd_handle_friend_update] 0-: Received my
> uuid as Friend
> [2015-02-26 00:14:55.234468] I
> [glusterd-handler.c:2257:__glusterd_handle_friend_update] 0-: Received
> uuid: 2b22e5f5-e860-409f-bacb-2ac1c76da7ae, hostname:hpc1
> [2015-02-26 00:15:14.637613] E
> [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management:
> Volume gfsvolume already started
> [2015-02-26 00:15:14.637661] E [glusterd-syncop.c:912:gd_stage_op_phase]
> 0-management: Staging of operation 'Volume Start' failed on localhost :
> Volume gfsvolume already started
> [2015-02-26 00:24:41.265078] W [socket.c:522:__socket_rwv] 0-management:
> readv on 172.20.20.22:24007 failed (No data available)
> [2015-02-26 00:24:43.631739] E [socket.c:2161:socket_connect_finish]
> 0-management: connection to 172.20.20.22:24007 failed (Connection refused)
> [2015-02-26 00:24:56.174790] W [socket.c:522:__socket_rwv] 0-management:
> readv on /var/run/91bfef953906a83fc14aa435a886ac4d.socket failed (No data
> available)
> [2015-02-26 00:24:56.175299] I
> [glusterd-handler.c:3713:__glusterd_brick_rpc_notify] 0-management:
> Disconnected from gfs1:/export/sda/brick
> [2015-02-26 00:25:24.076560] W
> [glusterd-op-sm.c:3404:glusterd_op_modify_op_ctx] 0-management: op_ctx
> modification failed
> [2015-02-26 00:25:24.078109] I
> [glusterd-handler.c:3530:__glusterd_handle_status_volume] 0-management:
> Received status volume req for volume gfsvolume
> [2015-02-26 00:25:31.041343] E
> [glusterd-volume-ops.c:1047:glusterd_op_stage_start_volume] 0-management:
> Volume gfsvolume already started
> [2015-02-26 00:25:31.041404] E [glusterd-syncop.c:912:gd_stage_op_phase]
> 0-management: Staging of operation 'Volume Start' failed on localhost :
> Volume gfsvolume already started
> [2015-02-26 00:25:43.389260] W [glusterfsd.c:1095:cleanup_and_exit]
> (-->/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d) [0x7f8f8cfa7e6d]
> (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50) [0x7f8f8d655b50]
> (-->/usr/sbin/glusterd(glusterfs_sigwaiter+0xd5) [0x7f8f8eee1d55]))) 0-:
> received signum (15), shutting down
> [2015-02-26 00:25:45.437325] I [glusterfsd.c:1959:main]
> 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.5.0
> (/usr/sbin/glusterd -p /var/run/glusterd.pid)
> [2015-02-26 00:25:45.487307] I [glusterd.c:1148:init] 0-management: Using
> /var/lib/glusterd as working directory
> [2015-02-26 00:25:45.488734] I [socket.c:3561:socket_init]
> 0-socket.management: SSL support is NOT enabled
> [2015-02-26 00:25:45.488765] I [socket.c:3576:socket_init]
> 0-socket.management: using system polling thread
> [2015-02-26 00:25:45.489557] W [rdma.c:4194:__gf_rdma_ctx_create]
> 0-rpc-transport/rdma: rdma_cm event channel creation failed (No such device)
> [2015-02-26 00:25:45.489584] E [rdma.c:4482:init] 0-rdma.management:
> Failed to initialize IB Device
> [2015-02-26 00:25:45.489597] E [rpc-transport.c:333:rpc_transport_load]
> 0-rpc-transport: 'rdma' initialization failed
> [2015-02-26 00:25:45.489668] W [rpcsvc.c:1483:rpcsvc_transport_create]
> 0-rpc-service: cannot create listener, initing the transport failed
> [2015-02-26 00:25:45.489799] I [socket.c:3561:socket_init]
> 0-socket.management: SSL support is NOT enabled
> [2015-02-26 00:25:45.489818] I [socket.c:3576:socket_init]
> 0-socket.management: using system polling thread
> [2015-02-26 00:25:48.706235] I
> [glusterd-store.c:1421:glusterd_restore_op_version] 0-glusterd: retrieved
> op-version: 2
>
>
>
>
> Please Do let me know if there is anything else i can provide to help.
> Apologies if this is a simple fix but i have never found an answer after a
> week of searching.
>
> Thank You Kindly,
> Kaamesh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150226/93d1eec7/attachment.html>
More information about the Gluster-users
mailing list