[Gluster-users] glusterd service fails to start from AWS AMI

[Jon Cope]

Hello all.

I have a working replica 2 cluster (4 nodes) up and running happily over Amazon EC2.  My end goal is to create AMIs of each machine and then quickly reproduce the same, but new, cluster from those AMIs.  Essentially, I'd like a cluster "template".

-Assigned original instances' Elastic IPs to new machines to reduce resolution issues.
-Passwordless SSH works on initial boot across all machines
-Node1: has no evident issue.  Starts with glusterd running.
-Node1: 'gluster peer status' returns correct public DNS / hostnames for peer nodes.  Status: (Disconnected)  --since the service is off on them

Since my goal is to create a cluster template, reinstalling gluster for each node, though it'll probably work, isn't an optimal solution.

Thank You

#  Node2: etc-glusterfs-glusterd.vol.log
#  Begins at 'service glusterd start' command entry

[2014-03-04 21:20:30.532138] I [glusterfsd.c:2024:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.4.0.44rhs (/usr/sbin/glusterd --pid-file=/var/run/glusterd.pid)
[2014-03-04 21:20:30.539331] I [glusterd.c:1020:init] 0-management: Using /var/lib/glusterd as working directory
[2014-03-04 21:20:30.542578] I [socket.c:3485:socket_init] 0-socket.management: SSL support is NOT enabled
[2014-03-04 21:20:30.542603] I [socket.c:3500:socket_init] 0-socket.management: using system polling thread
[2014-03-04 21:20:30.543203] C [rdma.c:4099:gf_rdma_init] 0-rpc-transport/rdma: Failed to get IB devices
[2014-03-04 21:20:30.543342] E [rdma.c:4990:init] 0-rdma.management: Failed to initialize IB Device
[2014-03-04 21:20:30.543375] E [rpc-transport.c:320:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2014-03-04 21:20:30.543471] W [rpcsvc.c:1387:rpcsvc_transport_create] 0-rpc-service: cannot create listener, initing the transport failed
[2014-03-04 21:20:37.116571] I [glusterd-store.c:1388:glusterd_restore_op_version] 0-glusterd: retrieved op-version: 2
[2014-03-04 21:20:37.120082] E [glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key: brick-0
[2014-03-04 21:20:37.120118] E [glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key: brick-1
[2014-03-04 21:20:37.120137] E [glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key: brick-2
[2014-03-04 21:20:37.120154] E [glusterd-store.c:1905:glusterd_store_retrieve_volume] 0-: Unknown key: brick-3
[2014-03-04 21:20:37.761785] I [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect returned 0
[2014-03-04 21:20:37.765059] I [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect returned 0
[2014-03-04 21:20:37.767677] I [glusterd-handler.c:2886:glusterd_friend_add] 0-management: connect returned 0
[2014-03-04 21:20:37.767783] I [rpc-clnt.c:974:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2014-03-04 21:20:37.767850] I [socket.c:3485:socket_init] 0-management: SSL support is NOT enabled
[2014-03-04 21:20:37.767866] I [socket.c:3500:socket_init] 0-management: using system polling thread
[2014-03-04 21:20:37.772356] I [rpc-clnt.c:974:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2014-03-04 21:20:37.772441] I [socket.c:3485:socket_init] 0-management: SSL support is NOT enabled
[2014-03-04 21:20:37.772459] I [socket.c:3500:socket_init] 0-management: using system polling thread
[2014-03-04 21:20:37.776131] I [rpc-clnt.c:974:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2014-03-04 21:20:37.776185] I [socket.c:3485:socket_init] 0-management: SSL support is NOT enabled
[2014-03-04 21:20:37.776201] I [socket.c:3500:socket_init] 0-management: using system polling thread
[2014-03-04 21:20:37.780363] E [glusterd-store.c:2548:glusterd_resolve_all_bricks] 0-glusterd: resolve brick failed in restore
[2014-03-04 21:20:37.780395] E [xlator.c:423:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again
[2014-03-04 21:20:37.780410] E [graph.c:292:glusterfs_graph_init] 0-management: initializing translator failed
[2014-03-04 21:20:37.780422] E [graph.c:479:glusterfs_graph_activate] 0-graph: init failed
[2014-03-04 21:20:37.780723] W [glusterfsd.c:1097:cleanup_and_exit] (-->/usr/sbin/glusterd(main+0x6b1) [0x406a91] (-->/usr/sbin/glusterd(glusterfs_volumes_init+0xb7) [0x405247] (-->/usr/sbin/glusterd(glusterfs_process_volfp+0x106) [0x405156]))) 0-: received signum (0), shutting down