[Gluster-users] cant mount gluster volume

Mon Nov 19 20:36:36 UTC 2012

I connect on 24009 glusterfs and fail on 27040 glusterd
Steve
________________________________
From: John Mark Walker [johnmark at redhat.com]
Sent: Monday, November 19, 2012 3:31 PM
To: Steve Postma
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] cant mount gluster volume

What happens if you telnet to any of the GlusterFS ports - from another machine and from localhost?

-JM

----- Original Message -----
> I am still unable to mount a new 3.3.1 glusterfs install. I have
> tried from one of the actual machines in the cluster to itself, as
> well as from various other clients. They all seem to be failing in
> the same part of the process.
>
> From the client log, my comments in bold
>
> Initial contact with server, no errors
> [2012-11-19 15:07:44.802826] I [glusterfsd.c:1666:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version
> 3.3.1
> [2012-11-19 15:07:44.811730] I [io-cache.c:1549:check_cache_size_ok]
> 0-gdata-quick-read: Max cache size is 25332322304
> [2012-11-19 15:07:44.811797] I [io-cache.c:1549:check_cache_size_ok]
> 0-gdata-io-cache: Max cache size is 25332322304
> [2012-11-19 15:07:44.815938] I [client.c:2142:notify]
> 0-gdata-client-0: parent translators are ready, attempting connect
> on transport
> [2012-11-19 15:07:44.819667] I [client.c:2142:notify]
> 0-gdata-client-1: parent translators are ready, attempting connect
> on transport
> [2012-11-19 15:07:44.823668] I [client.c:2142:notify]
> 0-gdata-client-2: parent translators are ready, attempting connect
> on transport
> Data from server is returned, looks like proper responses
> Given volfile:
> +------------------------------------------------------------------------------+
> 1: volume gdata-client-0
> 2: type protocol/client
> 3: option remote-host gluster-0-0
> 4: option remote-subvolume /mseas-data-0-0
> 5: option transport-type tcp
> 6: end-volume
> 7:
> 8: volume gdata-client-1
> 9: type protocol/client
> 10: option remote-host gluster-0-1
> 11: option remote-subvolume /mseas-data-0-1
> 12: option transport-type tcp
> 13: end-volume
> 14:
> 15: volume gdata-client-2
> 16: type protocol/client
> 17: option remote-host gluster-data
> 18: option remote-subvolume /data
> 19: option transport-type tcp
> 20: end-volume
> 21:
> 22: volume gdata-dht
> 23: type cluster/distribute
> 24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2
> 25: end-volume
> 26:
> 27: volume gdata-write-behind
> 28: type performance/write-behind
> 29: subvolumes gdata-dht
> 30: end-volume
> 31:
> 32: volume gdata-read-ahead
> 33: type performance/read-ahead
> 34: subvolumes gdata-write-behind
> 35: end-volume
> 36:
> 37: volume gdata-io-cache
> 38: type performance/io-cache
> 39: subvolumes gdata-read-ahead
> 40: end-volume
> 41:
> 42: volume gdata-quick-read
> 43: type performance/quick-read
> 44: subvolumes gdata-io-cache
> 45: end-volume
> 46:
> 47: volume gdata-stat-prefetch
> 48: type performance/stat-prefetch
> 49: subvolumes gdata-quick-read
> 50: end-volume
> 51:
> 52: volume gdata
> 53: type debug/io-stats
> 54: subvolumes gdata-stat-prefetch
> 55: end-volume
> +------------------------------------------------------------------------------+
>
> Port is changed and we can no longer connect.
> [2012-11-19 15:07:44.828213] I [rpc-clnt.c:1657:rpc_clnt_reconfig]
> 0-gdata-client-2: changing port to 24009 (from 0)
> [2012-11-19 15:07:44.828285] I [rpc-clnt.c:1657:rpc_clnt_reconfig]
> 0-gdata-client-0: changing port to 24009 (from 0)
> [2012-11-19 15:07:44.828329] I [rpc-clnt.c:1657:rpc_clnt_reconfig]
> 0-gdata-client-1: changing port to 24009 (from 0)
> [2012-11-19 15:07:48.812899] W
> [client-handshake.c:1819:client_dump_version_cbk] 0-gdata-client-2:
> received RPC status error
> [2012-11-19 15:07:48.812958] W
> [socket.c:1512:__socket_proto_state_machine] 0-gdata-client-2:
> reading from socket failed. Error (Transport endpoint is not
> connected), peer (10.1.1.2:24009)
> [2012-11-19 15:07:48.812985] I [client.c:2090:client_rpc_notify]
> 0-gdata-client-2: disconnected
> [2012-11-19 15:07:48.816284] W
> [client-handshake.c:1819:client_dump_version_cbk] 0-gdata-client-0:
> received RPC status error
> [2012-11-19 15:07:48.816325] W
> [socket.c:1512:__socket_proto_state_machine] 0-gdata-client-0:
> reading from socket failed. Error (Transport endpoint is not
> connected), peer (10.1.1.10:24009)
> [2012-11-19 15:07:48.816347] I [client.c:2090:client_rpc_notify]
> 0-gdata-client-0: disconnected
> [2012-11-19 15:07:48.819636] W
> [client-handshake.c:1819:client_dump_version_cbk] 0-gdata-client-1:
> received RPC status error
> [2012-11-19 15:07:48.819669] W
> [socket.c:1512:__socket_proto_state_machine] 0-gdata-client-1:
> reading from socket failed. Error (Transport endpoint is not
> connected), peer (10.1.1.11:24009)
> [2012-11-19 15:07:48.819688] I [client.c:2090:client_rpc_notify]
> 0-gdata-client-1: disconnected
> [2012-11-19 15:07:48.827123] I [fuse-bridge.c:4191:fuse_graph_setup]
> 0-fuse: switched to graph 0
> [2012-11-19 15:07:48.827368] I [fuse-bridge.c:3376:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13
> kernel 7.13
> [2012-11-19 15:07:48.827492] E [dht-common.c:1372:dht_lookup]
> 0-gdata-dht: Failed to get hashed subvol for /
> [2012-11-19 15:07:48.827607] E [dht-common.c:1372:dht_lookup]
> 0-gdata-dht: Failed to get hashed subvol for /
> [2012-11-19 15:07:48.827626] W [fuse-bridge.c:513:fuse_attr_cbk]
> 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Invalid argument)
> [2012-11-19 15:07:48.827709] E [dht-common.c:1372:dht_lookup]
> 0-gdata-dht: Failed to get hashed subvol for /
> [2012-11-19 15:07:48.827779] E [dht-common.c:1372:dht_lookup]
> 0-gdata-dht: Failed to get hashed subvol for /
> [2012-11-19 15:07:48.827917] E [dht-common.c:1372:dht_lookup]
> 0-gdata-dht: Failed to get hashed subvol for /
> [2012-11-19 15:07:48.828063] E [dht-common.c:1372:dht_lookup]
> 0-gdata-dht: Failed to get hashed subvol for /
> [2012-11-19 15:07:48.841568] I [fuse-bridge.c:4091:fuse_thread_proc]
> 0-fuse: unmounting /gdata
> [2012-11-19 15:07:48.841928] W [glusterfsd.c:831:cleanup_and_exit]
> (-->/lib64/libc.so.6(clone+0x6d) [0x3065ae5ccd]
> (-->/lib64/libpthread.so.0() [0x30666077f1]
> (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d4d]))) 0-:
> received signum (15), shutting down
> [2012-11-19 15:07:48.841965] I [fuse-bridge.c:4648:fini] 0-fuse:
> Unmounting '/gdata'.
>
>
>
> netstat verifies that glusterfs is listening on 24009
> [root at mseas-data<mailto:root at mseas-data> ~]# netstat --tcp --listening --programs
> Active Internet connections (only servers)
> Proto Recv-Q Send-Q Local Address Foreign Address
> State PID/Program name
> tcp 0 0 *:amanda *:*
> LISTEN 13890/xinetd
> tcp 0 0 *:sift-uft *:*
> LISTEN 23324/rpc.rquotad
> tcp 0 0 *:38465 *:*
> LISTEN 26629/glusterfs
> tcp 0 0 *:kamanda *:*
> LISTEN 13890/xinetd
> tcp 0 0 *:nfs *:*
> LISTEN -
> tcp 0 0 *:24007 *:*
> LISTEN 27040/glusterd
> tcp 0 0 *:sunrpc *:*
> LISTEN 2682/portmap
> tcp 0 0 *:51538 *:*
> LISTEN -
> tcp 0 0 *:ssh *:*
> LISTEN 14045/sshd
> tcp 0 0 *:ipcserver *:*
> LISTEN 19498/rpc.statd
> tcp 0 0 localhost.localdomain:smtp *:*
> LISTEN 3268/sendmail
> tcp 0 0 localhost.lo:x11-ssh-offset *:*
> LISTEN 23982/sshd
> tcp 0 0 *:794 *:*
> LISTEN 23511/rpc.mountd
> tcp 0 0 localhost.localdomain:6011 *:*
> LISTEN 24022/sshd
> tcp 0 0 localhost.localdomain:6012 *:*
> LISTEN 9462/sshd
> tcp 0 0 *:24009 *:*
> LISTEN 26625/glusterfsd
> tcp 0 0 *:ssh *:*
> LISTEN 14045/sshd
> tcp 0 0 localhost6.l:x11-ssh-offset *:*
> LISTEN 23982/sshd
> tcp 0 0 localhost6.localdomain:6011 *:*
> LISTEN 24022/sshd
> tcp 0 0 localhost6.localdomain:6012 *:*
> LISTEN 9462/sshd
>
> IPTABLES service has been stopped on all machines.
>
>
>
> Any help you could give me would be greatly appreciated.
>
> Thanks,
> Steve Postma
>
>
> ________________________________
> From: Steve Postma
> Sent: Friday, November 16, 2012 10:51 AM
> To: gluster-users at gluster.org<mailto:gluster-users at gluster.org>
> Subject: cant mount gluster volume
>
> I have a 3 node gluster cluster running:
> gluster --version
> glusterfs 3.3.1 built on Oct 11 2012 22:01:05
>
> It was previously running a 2.? version, uninstalled and installed
> 3.3.1
>
> It appears to be working correctly.
> gluster volume info
>
> Volume Name: gdata
> Type: Distribute
> Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
> Status: Started
> Number of Bricks: 3
> Transport-type: tcp
> Bricks:
> Brick1: gluster-0-0:/mseas-data-0-0
> Brick2: gluster-0-1:/mseas-data-0-1
> Brick3: gluster-data:/data
>
> The client previously
> I have a client with rpm's installed:
> rpm -qa | grep gluster
> glusterfs-fuse-3.3.1-1.el6.x86_64
> glusterfs-3.3.1-1.el6.x86_64
> glusterfs-debuginfo-3.3.1-1.el6.x86_64
>
>
> I have a line in my fstab file of:
> mseas-data:/gdata /gdata glusterfs defaults
> 0 0
> This is copied from backups before client reinstall
>
> I have also tried:
> mount.glusterfs mseas-data:/gdata \gdata
>
>
> From /var/log/glusterfs/gdata.log
>
> cat gdata.log
> [2012-11-16 10:43:36.462998] I [glusterfsd.c:1666:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version
> 3.3.1
> [2012-11-16 10:43:36.471280] I [io-cache.c:1549:check_cache_size_ok]
> 0-gdata-quick-read: Max cache size is 25332322304
> [2012-11-16 10:43:36.471347] I [io-cache.c:1549:check_cache_size_ok]
> 0-gdata-io-cache: Max cache size is 25332322304
> [2012-11-16 10:43:36.474518] I [client.c:2142:notify]
> 0-gdata-client-0: parent translators are ready, attempting connect
> on transport
> [2012-11-16 10:43:36.478166] I [client.c:2142:notify]
> 0-gdata-client-1: parent translators are ready, attempting connect
> on transport
> [2012-11-16 10:43:36.481604] I [client.c:2142:notify]
> 0-gdata-client-2: parent translators are ready, attempting connect
> on transport
> Given volfile:
> +------------------------------------------------------------------------------+
> 1: volume gdata-client-0
> 2: type protocol/client
> 3: option remote-host gluster-0-0
> 4: option remote-subvolume /mseas-data-0-0
> 5: option transport-type tcp
> 6: end-volume
> 7:
> 8: volume gdata-client-1
> 9: type protocol/client
> 10: option remote-host gluster-0-1
> 11: option remote-subvolume /mseas-data-0-1
> 12: option transport-type tcp
> 13: end-volume
> 14:
> 15: volume gdata-client-2
> 16: type protocol/client
> 17: option remote-host gluster-data
> 18: option remote-subvolume /data
> 19: option transport-type tcp
> 20: end-volume
> 21:
> 22: volume gdata-dht
> 23: type cluster/distribute
> 24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2
> 25: end-volume
> 26:
> 27: volume gdata-write-behind
> 28: type performance/write-behind
> 29: subvolumes gdata-dht
> 30: end-volume
> 31:
> 32: volume gdata-read-ahead
> 33: type performance/read-ahead
> 34: subvolumes gdata-write-behind
> 35: end-volume
> 36:
> 37: volume gdata-io-cache
> 38: type performance/io-cache
> 39: subvolumes gdata-read-ahead
> 40: end-volume
> 41:
> 42: volume gdata-quick-read
> 43: type performance/quick-read
> 44: subvolumes gdata-io-cache
> 45: end-volume
> 46:
> 47: volume gdata-stat-prefetch
> 48: type performance/stat-prefetch
> 49: subvolumes gdata-quick-read
> 50: end-volume
> 51:
> 52: volume gdata
> 53: type debug/io-stats
> 54: subvolumes gdata-stat-prefetch
> 55: end-volume
>
> +------------------------------------------------------------------------------+
> [2012-11-16 10:43:36.485221] E [socket.c:1715:socket_connect_finish]
> 0-gdata-client-2: connection to failed (No route to host)
> [2012-11-16 10:43:36.485534] E
> [client-handshake.c:1717:client_query_portmap_cbk] 0-gdata-client-1:
> failed to get the port number for remote subvolume
> [2012-11-16 10:43:36.485607] I [client.c:2090:client_rpc_notify]
> 0-gdata-client-1: disconnected
> [2012-11-16 10:43:36.485626] E
> [client-handshake.c:1717:client_query_portmap_cbk] 0-gdata-client-0:
> failed to get the port number for remote subvolume
> [2012-11-16 10:43:36.485647] I [client.c:2090:client_rpc_notify]
> 0-gdata-client-0: disconnected
> [2012-11-16 10:43:36.492937] I [fuse-bridge.c:4191:fuse_graph_setup]
> 0-fuse: switched to graph 0
> [2012-11-16 10:43:36.493966] I [fuse-bridge.c:3376:fuse_init]
> 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13
> kernel 7.13
> [2012-11-16 10:43:36.494088] E [dht-common.c:1372:dht_lookup]
> 0-gdata-dht: Failed to get hashed subvol for /
> [2012-11-16 10:43:36.494214] E [dht-common.c:1372:dht_lookup]
> 0-gdata-dht: Failed to get hashed subvol for /
> [2012-11-16 10:43:36.494231] W [fuse-bridge.c:513:fuse_attr_cbk]
> 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Invalid argument)
> [2012-11-16 10:43:36.500651] I [fuse-bridge.c:4091:fuse_thread_proc]
> 0-fuse: unmounting /var/log/glusterfs/gdata
> [2012-11-16 10:43:36.501036] W [glusterfsd.c:831:cleanup_and_exit]
> (-->/lib64/libc.so.6(clone+0x6d) [0x3065ae5ccd]
> (-->/lib64/libpthread.so.0() [0x30666077f1]
> (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d4d]))) 0-:
> received signum (15), shutting down
> [2012-11-16 10:43:36.501072] I [fuse-bridge.c:4648:fini] 0-fuse:
> Unmounting '/var/log/glusterfs/gdata'.
>
>
>
>
> It appears I am reaching the file system, but:
> 0-gdata-client-0: failed to get the port number for remote subvolume
> 0-gdata-dht: Failed to get hashed subvol for /
>
> I have googled/troubleshot but am unable to find solution. Your help
> would be Greatly appreciated.
> Thanks,
> Steve Postma
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org<mailto:Gluster-users at gluster.org>
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
________________________________