[Gluster-users] Problems adding new server to pool

Fri Sep 14 20:57:25 UTC 2012

Hello,

I don't have a great deal of experience yet with Gluster, and I'm having some tremendous difficulties adding an additional (3rd) server to the pool I have set up.  This is running Gluster 3.3.0 on RHEL 6.3, using RPMs:
# rpm -qa | grep gluster
glusterfs-rdma-3.3.0-1.el6.x86_64
glusterfs-server-3.3.0-1.el6.x86_64
glusterfs-3.3.0-1.el6.x86_64
glusterfs-fuse-3.3.0-1.el6.x86_64

When I first set it up, I was able to add the second server without an issue using gluster peer probe vothgluster2, however when I do the same here, it just hangs for a couple minutes, then returns to the prompt without successfully probing the server.

>From the first server:
# gluster volume info

Volume Name: home
Type: Distribute
Volume ID: a6f62691-a876-431a-a941-d3b76e197eeb
Status: Started
Number of Bricks: 4
Transport-type: tcp,rdma
Bricks:
Brick1: vothgluster1:/export/brick1
Brick2: vothgluster1:/export/brick2
Brick3: vothgluster2:/export/brick1
Brick4: vothgluster2:/export/brick2
Options Reconfigured:
nfs.rpc-auth-allow: 128.135.10.*,205.208.111.*,128.135.233.*,128.135.186.*,10.135.10.*,205.208.76.203
nfs.disable: on
nfs.register-with-portmap: on
performance.io-cache: on
nfs.port: 2049
features.quota: off
performance.cache-size: 64MB
performance.read-ahead: off
performance.write-behind-window-size: 1MB
nfs.ports-insecure: on

# gluster peer status
Number of Peers: 2

Hostname: vothgluster2
Uuid: 44033fcb-f6da-4491-b860-d05bb3ce23f7
State: Peer in Cluster (Connected)

Hostname: vothgluster3
Uuid: 924322ef-5e55-427b-a393-90903ccb7d16
State: Establishing Connection (Connected)

>From the second server, gluster volume info reports the same.
# gluster peer status
Number of Peers: 1

Hostname: 205.208.111.14
Uuid: 70ee8eed-224e-4031-8cfe-52004c5ccf47
State: Peer in Cluster (Connected)

>From the new (problematic) server:
# gluster volume info
No volumes present

# gluster peer status
Number of Peers: 1

Hostname: 205.208.111.14
Uuid: 00000000-0000-0000-0000-000000000000
State: Connected to Peer (Connected)

Forward and Reverse DNS lookups between all 3 servers work fine.  Ping works fine.  IB connectivity works fine (using tcp,rdma transport type).

/var/log/glusterfs/etc-glusterfs-glusterd.vol.log on vothgluster3
[2012-09-14 15:16:27.528532] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.3.0
[2012-09-14 15:16:27.530778] I [glusterd.c:807:init] 0-management: Using /var/lib/glusterd as working directory
[2012-09-14 15:16:28.088422] I [glusterd.c:95:glusterd_uuid_init] 0-glusterd: retrieved UUID: 924322ef-5e55-427b-a393-90903ccb7d16
[2012-09-14 15:16:28.097455] I [glusterd.c:265:glusterd_check_gsync_present] 0-glusterd: geo-replication module not installed in the system
Given volfile:
+------------------------------------------------------------------------------+
  1: volume management
  2:     type mgmt/glusterd
  3:     option working-directory /var/lib/glusterd
  4:     option transport-type socket,rdma
  5:     option transport.socket.keepalive-time 10
  6:     option transport.socket.keepalive-interval 2
  7:     option transport.socket.read-fail-log off
  8: end-volume

+------------------------------------------------------------------------------+
[2012-09-14 15:16:32.760625] I [glusterd-handler.c:1753:glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 70ee8eed-224e-4031-8cfe-52004c5ccf47
[2012-09-14 15:16:32.760707] I [glusterd-handler.c:411:glusterd_friend_find] 0-glusterd: Unable to find peer by uuid
[2012-09-14 15:16:32.760929] I [glusterd-handler.c:423:glusterd_friend_find] 0-glusterd: Unable to find hostname: 205.208.111.14
[2012-09-14 15:16:32.760950] I [glusterd-handler.c:1767:glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 205.208.111.14 (24007)
[2012-09-14 15:16:32.766007] I [glusterd-handler.c:2204:glusterd_friend_add] 0-management: connect returned 0
[2012-09-14 15:16:32.766067] I [glusterd-handler.c:1793:glusterd_handle_probe_query] 0-glusterd: Responded to 205.208.111.14, op_ret: 0, op_errno: 0, ret: 0
[2012-09-14 15:16:32.766572] I [glusterd-handshake.c:397:glusterd_set_clnt_mgmt_program] 0-: Using Program glusterd mgmt, Num (1238433), Version (2)
[2012-09-14 15:16:32.766620] I [glusterd-handshake.c:403:glusterd_set_clnt_mgmt_program] 0-: Using Program Peer mgmt, Num (1238437), Version (2)
[2012-09-14 15:18:44.916221] I [glusterd-handler.c:813:glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2012-09-14 15:19:41.020394] I [glusterd-handler.c:860:glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2012-09-14 15:22:40.807457] I [glusterd-handler.c:860:glusterd_handle_cli_get_volume] 0-glusterd: Received get vol req
[2012-09-14 15:22:48.655202] I [glusterd-handler.c:813:glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req

-----

Please let me know any other information to provide.

Thanks in advance,
Tom

--
Tom Indelli
tindelli at uchicago.edu<mailto:tindelli at uchicago.edu>
Systems Administrator
Physical Sciences Division
The University of Chicago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120914/c469c531/attachment.html>