[Gluster-users] Cannot create a simple 2-brick volume.

Mon Nov 14 20:19:56 UTC 2011

[2011-11-13 21:46:09.133729] E
[server-handshake.c:553:server_setvolume] 0-glue-server: Cannot
authenticate client from 127.0.0.1:1023 3.2.4

Can you remove the access 10.* and see if it works first?

On Mon, Nov 14, 2011 at 12:11 PM, Mark Sullivan <marks at mars.ucla.edu> wrote:
> Gluster Community,
>
> I'm having a terrible time just trying to get started with gluster.
> I'm running Centos 5.7 on a few nodes, and have installed gluster 3.2.4 and
> its prereqs from RPMs.
> Yet I'm finding it impossible to create a simple 2-brick distributed volume.
> I keep seeing this error a lot:
>
>   reading from socket failed. Error (Transport endpoint is not connected)
>
> referring to both the localhost and peers.   There is no iptables running
> on any of these machines, and all machines can ssh to each other
> and report that their peers are connected.
>
> I've googled this and other errors I've seen, and many results point
> into this site, but none of the suggestions I've read have helped me.
> The glusterfsd's are running.  The peers are connected.  I've done
> multiple reboots and restarts of daemons.   This is a fresh install.
>
> Details are listed below.
> Can someone please help me out?
> Thanks!
> -Mark Sullivan
> Diviner Lunar Radiometer Experiment
>
> ==========================================================================================================
> ==========================================================================================================
> ==========================================================================================================
>
> On gluster03, creating a volume "glue" which is comprised of
> gluster03:/g1 and gluster04:/g1
>
> gluster volume create glue transport tcp gluster03:/g1 gluster04:/g1
> gluster volume set glue auth.allow 10.*
> gluster volume start glue
>
> The "etc*" log files show this:
>
> [2011-11-13 16:10:22.429786] I
> [glusterd-handler.c:900:glusterd_handle_create_volume] 0-glusterd: Received
> create volume req
> [2011-11-13 16:10:22.430303] I [glusterd-utils.c:243:glusterd_lock]
> 0-glusterd: Cluster lock held by fb1f46cf-a03a-4fcd-b103-735040af3ced
> [2011-11-13 16:10:22.430330] I
> [glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired local
> lock
> [2011-11-13 16:10:22.430777] I
> [glusterd-rpc-ops.c:752:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received
> ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:22.431182] I
> [glusterd-op-sm.c:6543:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req
> to 1 peers
> [2011-11-13 16:10:22.431814] I
> [glusterd-rpc-ops.c:1050:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC
> from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:22.470773] I
> [glusterd-op-sm.c:6660:glusterd_op_ac_send_commit_op] 0-glusterd: Sent op
> req to 1 peers
> [2011-11-13 16:10:22.489143] I
> [glusterd-rpc-ops.c:1236:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC
> from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:22.489566] I
> [glusterd-rpc-ops.c:811:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received
> ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:22.489604] I
> [glusterd-op-sm.c:7077:glusterd_op_txn_complete] 0-glusterd: Cleared local
> lock
> [2011-11-13 16:10:22.492971] W [socket.c:1494:__socket_proto_state_machine]
> 0-socket.management: reading from socket failed. Error (Transport endpoint
> is not connected), peer (127.0.0.1:1023)
> [2011-11-13 16:10:22.611682] I [glusterd-utils.c:243:glusterd_lock]
> 0-glusterd: Cluster lock held by fb1f46cf-a03a-4fcd-b103-735040af3ced
> [2011-11-13 16:10:22.611709] I
> [glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired local
> lock
> [2011-11-13 16:10:22.612096] I
> [glusterd-rpc-ops.c:752:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received
> ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:22.896543] I
> [glusterd-op-sm.c:6543:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req
> to 1 peers
> [2011-11-13 16:10:23.55185] I
> [glusterd-rpc-ops.c:1050:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC
> from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:23.64798] I
> [glusterd-op-sm.c:6660:glusterd_op_ac_send_commit_op] 0-glusterd: Sent op
> req to 1 peers
> [2011-11-13 16:10:23.74209] I
> [glusterd-rpc-ops.c:1236:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC
> from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:23.74527] I
> [glusterd-rpc-ops.c:811:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received
> ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:23.74558] I
> [glusterd-op-sm.c:7077:glusterd_op_txn_complete] 0-glusterd: Cleared local
> lock
> [2011-11-13 16:10:23.79190] W [socket.c:1494:__socket_proto_state_machine]
> 0-socket.management: reading from socket failed. Error (Transport endpoint
> is not connected), peer (127.0.0.1:1020)
> [2011-11-13 16:10:23.198846] I
> [glusterd-handler.c:1078:glusterd_handle_cli_start_volume] 0-glusterd:
> Received start vol reqfor volume glue
> [2011-11-13 16:10:23.198913] I [glusterd-utils.c:243:glusterd_lock]
> 0-glusterd: Cluster lock held by fb1f46cf-a03a-4fcd-b103-735040af3ced
> [2011-11-13 16:10:23.198938] I
> [glusterd-handler.c:420:glusterd_op_txn_begin] 0-glusterd: Acquired local
> lock
> [2011-11-13 16:10:23.199364] I
> [glusterd-rpc-ops.c:752:glusterd3_1_cluster_lock_cbk] 0-glusterd: Received
> ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:23.199819] I
> [glusterd-op-sm.c:6543:glusterd_op_ac_send_stage_op] 0-glusterd: Sent op req
> to 1 peers
> [2011-11-13 16:10:23.200396] I
> [glusterd-rpc-ops.c:1050:glusterd3_1_stage_op_cbk] 0-glusterd: Received ACC
> from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:23.724138] I
> [glusterd-utils.c:1095:glusterd_volume_start_glusterfs] 0-: About to start
> glusterfs for brick gluster03:/g1
> [2011-11-13 16:10:23.989454] I
> [glusterd-op-sm.c:6660:glusterd_op_ac_send_commit_op] 0-glusterd: Sent op
> req to 1 peers
> [2011-11-13 16:10:24.7044] I [glusterd-pmap.c:237:pmap_registry_bind]
> 0-pmap: adding brick /g1 on port 24009
> [2011-11-13 16:10:24.39658] W [socket.c:1494:__socket_proto_state_machine]
> 0-socket.management: reading from socket failed. Error (Transport endpoint
> is not connected), peer (127.0.0.1:1017)
> [2011-11-13 16:10:24.816411] I
> [glusterd-rpc-ops.c:1236:glusterd3_1_commit_op_cbk] 0-glusterd: Received ACC
> from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:24.816940] I
> [glusterd-rpc-ops.c:811:glusterd3_1_cluster_unlock_cbk] 0-glusterd: Received
> ACC from uuid: 7c9ee90c-91a5-45c0-aaf9-8b8a7347b67d
> [2011-11-13 16:10:24.816993] I
> [glusterd-op-sm.c:7077:glusterd_op_txn_complete] 0-glusterd: Cleared local
> lock
> [2011-11-13 16:10:24.818726] W [socket.c:1494:__socket_proto_state_machine]
> 0-socket.management: reading from socket failed. Error (Transport endpoint
> is not connected), peer (127.0.0.1:1019)
> [2011-11-13 16:10:24.859565] W [socket.c:1494:__socket_proto_state_machine]
> 0-socket.management: reading from socket failed. Error (Transport endpoint
> is not connected), peer (10.1.1.24:1019)
>
> ==========================================================================================================
>
> My volume info looks okay, I guess...
>
> gluster volume info
>
> Volume Name: glue
> Type: Distribute
> Status: Started
> Number of Bricks: 2
> Transport-type: tcp
> Bricks:
> Brick1: gluster03:/g1
> Brick2: gluster04:/g1
> Options Reconfigured:
> auth.allow: 10.*
>
> When I mount the volume "glue" on gluster03 using "mount -t nfs
> gluster03:/glue /mnt", the nfs.log shows:
>
> [2011-11-13 16:18:06.83447] I [client3_1-fops.c:2228:client3_1_lookup_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:18:06.83507] I [dht-common.c:478:dht_revalidate_cbk]
> 0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument)
> [2011-11-13 16:18:06.84676] I [client3_1-fops.c:2228:client3_1_lookup_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:18:06.84704] I [dht-common.c:478:dht_revalidate_cbk]
> 0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument)
> [2011-11-13 16:18:06.85687] W [rpc-common.c:64:xdr_to_generic]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
> [0x2ae52ccad6fd]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
> [0x2ae52ccad502]
> (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91)
> [0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed
> [2011-11-13 16:18:06.85723] E [client3_1-fops.c:398:client3_1_stat_cbk]
> 0-glue-client-0: error
> [2011-11-13 16:18:06.85748] I [client3_1-fops.c:411:client3_1_stat_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:18:06.86273] W [rpc-common.c:64:xdr_to_generic]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
> [0x2ae52ccad6fd]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
> [0x2ae52ccad502]
> (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91)
> [0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed
> [2011-11-13 16:18:06.86301] E [client3_1-fops.c:398:client3_1_stat_cbk]
> 0-glue-client-0: error
> [2011-11-13 16:18:06.86324] I [client3_1-fops.c:411:client3_1_stat_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
>
> ==========================================================================================================
>
> When I do "touch /mnt/new", I get "No such file or directory", and nfs.log
> shows:
>
> [2011-11-13 16:18:06.83447] I [client3_1-fops.c:2228:client3_1_lookup_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:18:06.83507] I [dht-common.c:478:dht_revalidate_cbk]
> 0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument)
> [2011-11-13 16:18:06.84676] I [client3_1-fops.c:2228:client3_1_lookup_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:18:06.84704] I [dht-common.c:478:dht_revalidate_cbk]
> 0-glue-dht: subvolume glue-client-0 for / returned -1 (Invalid argument)
> [2011-11-13 16:18:06.85687] W [rpc-common.c:64:xdr_to_generic]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
> [0x2ae52ccad6fd]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
> [0x2ae52ccad502]
> (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91)
> [0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed
> [2011-11-13 16:18:06.85723] E [client3_1-fops.c:398:client3_1_stat_cbk]
> 0-glue-client-0: error
> [2011-11-13 16:18:06.85748] I [client3_1-fops.c:411:client3_1_stat_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:18:06.86273] W [rpc-common.c:64:xdr_to_generic]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
> [0x2ae52ccad6fd]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
> [0x2ae52ccad502]
> (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_stat_cbk+0x91)
> [0x2aaaaaacccb1]))) 0-xdr: XDR decoding failed
> [2011-11-13 16:18:06.86301] E [client3_1-fops.c:398:client3_1_stat_cbk]
> 0-glue-client-0: error
> [2011-11-13 16:18:06.86324] I [client3_1-fops.c:411:client3_1_stat_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:19:48.424842] I [dht-layout.c:192:dht_layout_search]
> 0-glue-dht: no subvolume for hash (value) = 1407928635
> [2011-11-13 16:19:48.425129] I [client3_1-fops.c:2228:client3_1_lookup_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:19:48.425751] I [dht-layout.c:192:dht_layout_search]
> 0-glue-dht: no subvolume for hash (value) = 1407928635
> [2011-11-13 16:19:48.425991] I [client3_1-fops.c:2228:client3_1_lookup_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:19:48.449516] I [dht-layout.c:192:dht_layout_search]
> 0-glue-dht: no subvolume for hash (value) = 1407928635
> [2011-11-13 16:19:48.449662] E [fd.c:465:fd_unref]
> (-->/opt/glusterfs/3.2.4/lib64/libglusterfs.so.0(default_create_cbk+0xb4)
> [0x2ae52ca65cc4]
> (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/debug/io-stats.so(io_stats_create_cbk+0x20c)
> [0x2aaaab76263c]
> (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/nfs/server.so(nfs_fop_create_cbk+0x73)
> [0x2aaaab988a13]))) 0-fd: fd is NULL
> [2011-11-13 16:19:48.449859] W [rpc-common.c:64:xdr_to_generic]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_notify+0x8d)
> [0x2ae52ccad6fd]
> (-->/opt/glusterfs/3.2.4/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa2)
> [0x2ae52ccad502]
> (-->/opt/glusterfs/3.2.4/lib64/glusterfs/3.2.4/xlator/protocol/client.so(client3_1_statfs_cbk+0x7e)
> [0x2aaaaaac806e]))) 0-xdr: XDR decoding failed
> [2011-11-13 16:19:48.449888] E [client3_1-fops.c:624:client3_1_statfs_cbk]
> 0-glue-client-0: error
> [2011-11-13 16:19:48.449912] I [client3_1-fops.c:637:client3_1_statfs_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
> [2011-11-13 16:19:48.450030] I [dht-layout.c:192:dht_layout_search]
> 0-glue-dht: no subvolume for hash (value) = 1407928635
> [2011-11-13 16:19:48.450260] I [client3_1-fops.c:2228:client3_1_lookup_cbk]
> 0-glue-client-0: remote operation failed: Invalid argument
>
> ==========================================================================================================
>
> And from the brick log g1.log, in case this helps:
>
> [2011-11-13 21:46:05.929654] I [glusterfsd.c:1493:main]
> 0-/opt/glusterfs/3.2.4/sbin/glusterfsd: Started Running
> /opt/glusterfs/3.2.4/sbin/glusterfsd version 3.2.4
> [2011-11-13 21:46:05.946509] W [socket.c:419:__socket_keepalive] 0-socket:
> failed to set keep idle on socket 8
> [2011-11-13 21:46:05.946618] W [socket.c:1846:socket_server_event_handler]
> 0-socket.glusterfsd: Failed to set keep-alive: Operation not supported
> [2011-11-13 21:46:06.72770] W [graph.c:291:gf_add_cmdline_options]
> 0-glue-server: adding option 'listen-port' for volume 'glue-server' with
> value '24010'
> [2011-11-13 21:46:06.73873] W [rpc-transport.c:447:validate_volume_options]
> 0-tcp.glue-server: option 'listen-port' is deprecated, preferred is
> 'transport.socket.listen-port', continuing with correction
> [2011-11-13 21:46:06.74204] W [posix.c:4686:init] 0-glue-posix: Posix access
> control list is not supported.
> Given volfile:
> +------------------------------------------------------------------------------+
>  1: volume glue-posix
>  2:     type storage/posix
>  3:     option directory /g1
>  4: end-volume
>  5:
>  6: volume glue-access-control
>  7:     type features/access-control
>  8:     subvolumes glue-posix
>  9: end-volume
> 10:
> 11: volume glue-locks
> 12:     type features/locks
> 13:     subvolumes glue-access-control
> 14: end-volume
> 15:
> 16: volume glue-io-threads
> 17:     type performance/io-threads
> 18:     subvolumes glue-locks
> 19: end-volume
> 20:
> 21: volume glue-marker
> 22:     type features/marker
> 23:     option volume-uuid 2b567c80-ab30-44b2-9b17-e67e6e679096
> 24:     option timestamp-file /etc/glusterd/vols/glue/marker.tstamp
> 25:     option xtime off
> 26:     option quota off
> 27:     subvolumes glue-io-threads
> 28: end-volume
> 29:
> 30: volume /g1
> 31:     type debug/io-stats
> 32:     option latency-measurement off
> 33:     option count-fop-hits off
> 34:     subvolumes glue-marker
> 35: end-volume
> 36:
> 37: volume glue-server
> 38:     type protocol/server
> 39:     option transport-type tcp
> 40:     option auth.addr./g1.allow 10.*
> 41:     subvolumes /g1
> 42: end-volume
>
> +------------------------------------------------------------------------------+
> [2011-11-13 21:46:09.133670] E [authenticate.c:227:gf_authenticate] 0-auth:
> no authentication module is interested in accepting remote-client (null)
> [2011-11-13 21:46:09.133729] E [server-handshake.c:553:server_setvolume]
> 0-glue-server: Cannot authenticate client from 127.0.0.1:1023 3.2.4
> [2011-11-13 21:46:09.389447] I [server-handshake.c:542:server_setvolume]
> 0-glue-server: accepted client from 10.1.1.24:1022 (version: 3.2.4)
>
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>