[Gluster-devel] Expanding Volumes
Jonhnny Weslley
jw at jonhnnyweslley.net
Tue Jun 9 13:03:37 UTC 2015
Hi guys,
I trying to create a pool of 4 nodes using centos7 and gluster 3.7 in a
vagrant-based environment for test. First, I create and start a replicated
volume using only 2 nodes (replication 2). After, I mount the volume using
fuse and copy some files. Everything works fine.
Then, I try to expand the volume previously created using the command:
sudo gluster volume add-brick jged 10.10.50.73:/home/vagrant/brick
10.10.50.74:/home/vagrant/brick force
And works again:
sudo gluster volume info
Volume Name: jged
Type: Distributed-Replicate
Volume ID: 862ab9b7-4753-4682-ba44-cbe481b1b7df
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: 10.10.50.71:/home/vagrant/brick
Brick2: 10.10.50.72:/home/vagrant/brick
Brick3: 10.10.50.73:/home/vagrant/brick
Brick4: 10.10.50.74:/home/vagrant/brick
Options Reconfigured:
performance.readdir-ahead: on
But when I try to rebalance the volume (sudo gluster volume rebalance jged
start), the gluster's proccess in the node where the command was executed
dies and dont start again after running 'systemctl start glusterd'. I look
the log file (/var/log/glusterfs/etc-glusterfs-glusterd.vol.log) but I cant
figure out what is wrong! :(
Follow the tail of the log file:
[2015-06-09 12:30:56.197802] I [MSGID: 100030] [glusterfsd.c:2294:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.1
(args: /usr/sbin/glusterd -p /var/run/glusterd.pid)
[2015-06-09 12:30:56.207596] I [glusterd.c:1282:init] 0-management: Maximum
allowed open file descriptors set to 65536
[2015-06-09 12:30:56.207653] I [glusterd.c:1327:init] 0-management: Using
/var/lib/glusterd as working directory
[2015-06-09 12:30:56.211505] E [rpc-transport.c:291:rpc_transport_load]
0-rpc-transport: /usr/lib64/glusterfs/3.7.1/rpc-transport/rdma.so: cannot
open shared object file: No such file or directory
[2015-06-09 12:30:56.211521] W [rpc-transport.c:295:rpc_transport_load]
0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not
valid or not found on this machine
[2015-06-09 12:30:56.211528] W [rpcsvc.c:1595:rpcsvc_transport_create]
0-rpc-service: cannot create listener, initing the transport failed
[2015-06-09 12:30:56.211535] E [glusterd.c:1515:init] 0-management:
creation of 1 listeners failed, continuing with succeeded transport
[2015-06-09 12:30:56.213311] I
[glusterd.c:413:glusterd_check_gsync_present] 0-glusterd: geo-replication
module not installed in the system
[2015-06-09 12:30:56.213454] I
[glusterd-store.c:1986:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 30700
[2015-06-09 12:30:56.213523] I [glusterd.c:154:glusterd_uuid_init]
0-management: retrieved UUID: f264d968-5a14-459b-8f3b-569aa15c3ce2
[2015-06-09 12:30:56.213568] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-glustershd: setting frame-timeout to 600
[2015-06-09 12:30:56.213675] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-nfs: setting frame-timeout to 600
[2015-06-09 12:30:56.213801] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-quotad: setting frame-timeout to 600
[2015-06-09 12:30:56.213896] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-bitd: setting frame-timeout to 600
[2015-06-09 12:30:56.213979] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-scrub: setting frame-timeout to 600
[2015-06-09 12:30:56.214094] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-snapd: setting frame-timeout to 600
[2015-06-09 12:30:56.987649] I
[glusterd-handler.c:3387:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
[2015-06-09 12:30:56.987711] I
[glusterd-handler.c:3387:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
[2015-06-09 12:30:56.987755] I
[glusterd-handler.c:3387:glusterd_friend_add_from_peerinfo] 0-management:
connect returned 0
[2015-06-09 12:30:56.987801] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-06-09 12:30:56.989874] W [socket.c:923:__socket_keepalive] 0-socket:
failed to set TCP_USER_TIMEOUT -1000 on socket 13, Invalid argument
[2015-06-09 12:30:56.989890] E [socket.c:3015:socket_connect] 0-management:
Failed to set keep-alive: Invalid argument
[2015-06-09 12:30:56.990051] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-06-09 12:30:56.992360] W [socket.c:923:__socket_keepalive] 0-socket:
failed to set TCP_USER_TIMEOUT -1000 on socket 14, Invalid argument
[2015-06-09 12:30:56.992419] E [socket.c:3015:socket_connect] 0-management:
Failed to set keep-alive: Invalid argument
[2015-06-09 12:30:56.992629] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
[2015-06-09 12:30:56.994163] W [socket.c:923:__socket_keepalive] 0-socket:
failed to set TCP_USER_TIMEOUT -1000 on socket 15, Invalid argument
[2015-06-09 12:30:56.994177] E [socket.c:3015:socket_connect] 0-management:
Failed to set keep-alive: Invalid argument
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option transport.socket.listen-backlog 128
7: option ping-timeout 30
8: option transport.socket.read-fail-log off
9: option transport.socket.keepalive-interval 2
10: option transport.socket.keepalive-time 10
11: option transport-type rdma
12: option working-directory /var/lib/glusterd
13: end-volume
14:
+------------------------------------------------------------------------------+
[2015-06-09 12:30:57.007380] I
[event-epoll.c:629:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2015-06-09 12:30:57.288783] I
[glusterd-rpc-ops.c:464:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: 41707504-c217-4b7a-8f3a-7150bd040d0a, host: 10.10.50.74, port: 0
[2015-06-09 12:30:57.292480] I
[glusterd-handler.c:2587:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: 41707504-c217-4b7a-8f3a-7150bd040d0a
[2015-06-09 12:30:57.292542] I
[glusterd-handler.c:2630:__glusterd_handle_friend_update] 0-management:
Received my uuid as Friend
[2015-06-09 12:30:57.293810] I [MSGID: 106132]
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already
stopped
[2015-06-09 12:30:57.303585] W [socket.c:3059:socket_connect] 0-nfs: Ignore
failed connection attempt on , (No such file or directory)
[2015-06-09 12:30:57.304172] I [MSGID: 106132]
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: glustershd
already stopped
[2015-06-09 12:30:57.324624] W [socket.c:3059:socket_connect] 0-glustershd:
Ignore failed connection attempt on , (No such file or directory)
[2015-06-09 12:30:57.324721] I [MSGID: 106132]
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already
stopped
[2015-06-09 12:30:57.324767] I [MSGID: 106132]
[glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already
stopped
[2015-06-09 12:30:57.343225] I [rpc-clnt.c:972:rpc_clnt_connection_init]
0-management: setting frame-timeout to 600
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash:
2015-06-09 12:30:57
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.1
/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb2)[0x7f3f46ca2d92]
/lib64/libglusterfs.so.0(gf_print_trace+0x32d)[0x7f3f46cbd9ed]
/lib64/libc.so.6(+0x35a00)[0x7f3f45696a00]
/lib64/libc.so.6(gsignal+0x39)[0x7f3f45696989]
/lib64/libc.so.6(abort+0x148)[0x7f3f45698098]
/lib64/libc.so.6(+0x76197)[0x7f3f456d7197]
/lib64/libc.so.6(__fortify_fail+0x37)[0x7f3f4576ecb7]
/lib64/libc.so.6(+0x10be80)[0x7f3f4576ce80]
/lib64/libc.so.6(+0x10b5bb)[0x7f3f4576c5bb]
/lib64/libc.so.6(__snprintf_chk+0x78)[0x7f3f4576c4d8]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_volume_defrag_restart+0x191)[0x7f3f3bb37931]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_restart_rebalance+0x82)[0x7f3f3bb3daa2]
/usr/lib64/glusterfs/3.7.1/xlator/mgmt/glusterd.so(glusterd_spawn_daemons+0x4f)[0x7f3f3bb3db1f]
/lib64/libglusterfs.so.0(synctask_wrap+0x12)[0x7f3f46cdd482]
/lib64/libc.so.6(+0x47570)[0x7f3f456a8570]
---------
--
Jonhnny Weslley
http://blog.jonhnnyweslley.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20150609/a22812ed/attachment-0001.html>
More information about the Gluster-devel
mailing list