[Gluster-users] cant mount gluster volume
Steve Postma
SPostma at ztechnet.com
Mon Nov 19 20:29:05 UTC 2012
I am still unable to mount a new 3.3.1 glusterfs install. I have tried from one of the actual machines in the cluster to itself, as well as from various other clients. They all seem to be failing in the same part of the process.
>From the client log, my comments in bold
Initial contact with server, no errors
[2012-11-19 15:07:44.802826] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
[2012-11-19 15:07:44.811730] I [io-cache.c:1549:check_cache_size_ok] 0-gdata-quick-read: Max cache size is 25332322304
[2012-11-19 15:07:44.811797] I [io-cache.c:1549:check_cache_size_ok] 0-gdata-io-cache: Max cache size is 25332322304
[2012-11-19 15:07:44.815938] I [client.c:2142:notify] 0-gdata-client-0: parent translators are ready, attempting connect on transport
[2012-11-19 15:07:44.819667] I [client.c:2142:notify] 0-gdata-client-1: parent translators are ready, attempting connect on transport
[2012-11-19 15:07:44.823668] I [client.c:2142:notify] 0-gdata-client-2: parent translators are ready, attempting connect on transport
Data from server is returned, looks like proper responses
Given volfile:
+------------------------------------------------------------------------------+
1: volume gdata-client-0
2: type protocol/client
3: option remote-host gluster-0-0
4: option remote-subvolume /mseas-data-0-0
5: option transport-type tcp
6: end-volume
7:
8: volume gdata-client-1
9: type protocol/client
10: option remote-host gluster-0-1
11: option remote-subvolume /mseas-data-0-1
12: option transport-type tcp
13: end-volume
14:
15: volume gdata-client-2
16: type protocol/client
17: option remote-host gluster-data
18: option remote-subvolume /data
19: option transport-type tcp
20: end-volume
21:
22: volume gdata-dht
23: type cluster/distribute
24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2
25: end-volume
26:
27: volume gdata-write-behind
28: type performance/write-behind
29: subvolumes gdata-dht
30: end-volume
31:
32: volume gdata-read-ahead
33: type performance/read-ahead
34: subvolumes gdata-write-behind
35: end-volume
36:
37: volume gdata-io-cache
38: type performance/io-cache
39: subvolumes gdata-read-ahead
40: end-volume
41:
42: volume gdata-quick-read
43: type performance/quick-read
44: subvolumes gdata-io-cache
45: end-volume
46:
47: volume gdata-stat-prefetch
48: type performance/stat-prefetch
49: subvolumes gdata-quick-read
50: end-volume
51:
52: volume gdata
53: type debug/io-stats
54: subvolumes gdata-stat-prefetch
55: end-volume
+------------------------------------------------------------------------------+
Port is changed and we can no longer connect.
[2012-11-19 15:07:44.828213] I [rpc-clnt.c:1657:rpc_clnt_reconfig] 0-gdata-client-2: changing port to 24009 (from 0)
[2012-11-19 15:07:44.828285] I [rpc-clnt.c:1657:rpc_clnt_reconfig] 0-gdata-client-0: changing port to 24009 (from 0)
[2012-11-19 15:07:44.828329] I [rpc-clnt.c:1657:rpc_clnt_reconfig] 0-gdata-client-1: changing port to 24009 (from 0)
[2012-11-19 15:07:48.812899] W [client-handshake.c:1819:client_dump_version_cbk] 0-gdata-client-2: received RPC status error
[2012-11-19 15:07:48.812958] W [socket.c:1512:__socket_proto_state_machine] 0-gdata-client-2: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.2:24009)
[2012-11-19 15:07:48.812985] I [client.c:2090:client_rpc_notify] 0-gdata-client-2: disconnected
[2012-11-19 15:07:48.816284] W [client-handshake.c:1819:client_dump_version_cbk] 0-gdata-client-0: received RPC status error
[2012-11-19 15:07:48.816325] W [socket.c:1512:__socket_proto_state_machine] 0-gdata-client-0: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.10:24009)
[2012-11-19 15:07:48.816347] I [client.c:2090:client_rpc_notify] 0-gdata-client-0: disconnected
[2012-11-19 15:07:48.819636] W [client-handshake.c:1819:client_dump_version_cbk] 0-gdata-client-1: received RPC status error
[2012-11-19 15:07:48.819669] W [socket.c:1512:__socket_proto_state_machine] 0-gdata-client-1: reading from socket failed. Error (Transport endpoint is not connected), peer (10.1.1.11:24009)
[2012-11-19 15:07:48.819688] I [client.c:2090:client_rpc_notify] 0-gdata-client-1: disconnected
[2012-11-19 15:07:48.827123] I [fuse-bridge.c:4191:fuse_graph_setup] 0-fuse: switched to graph 0
[2012-11-19 15:07:48.827368] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
[2012-11-19 15:07:48.827492] E [dht-common.c:1372:dht_lookup] 0-gdata-dht: Failed to get hashed subvol for /
[2012-11-19 15:07:48.827607] E [dht-common.c:1372:dht_lookup] 0-gdata-dht: Failed to get hashed subvol for /
[2012-11-19 15:07:48.827626] W [fuse-bridge.c:513:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Invalid argument)
[2012-11-19 15:07:48.827709] E [dht-common.c:1372:dht_lookup] 0-gdata-dht: Failed to get hashed subvol for /
[2012-11-19 15:07:48.827779] E [dht-common.c:1372:dht_lookup] 0-gdata-dht: Failed to get hashed subvol for /
[2012-11-19 15:07:48.827917] E [dht-common.c:1372:dht_lookup] 0-gdata-dht: Failed to get hashed subvol for /
[2012-11-19 15:07:48.828063] E [dht-common.c:1372:dht_lookup] 0-gdata-dht: Failed to get hashed subvol for /
[2012-11-19 15:07:48.841568] I [fuse-bridge.c:4091:fuse_thread_proc] 0-fuse: unmounting /gdata
[2012-11-19 15:07:48.841928] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3065ae5ccd] (-->/lib64/libpthread.so.0() [0x30666077f1] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d4d]))) 0-: received signum (15), shutting down
[2012-11-19 15:07:48.841965] I [fuse-bridge.c:4648:fini] 0-fuse: Unmounting '/gdata'.
netstat verifies that glusterfs is listening on 24009
[root at mseas-data ~]# netstat --tcp --listening --programs
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 *:amanda *:* LISTEN 13890/xinetd
tcp 0 0 *:sift-uft *:* LISTEN 23324/rpc.rquotad
tcp 0 0 *:38465 *:* LISTEN 26629/glusterfs
tcp 0 0 *:kamanda *:* LISTEN 13890/xinetd
tcp 0 0 *:nfs *:* LISTEN -
tcp 0 0 *:24007 *:* LISTEN 27040/glusterd
tcp 0 0 *:sunrpc *:* LISTEN 2682/portmap
tcp 0 0 *:51538 *:* LISTEN -
tcp 0 0 *:ssh *:* LISTEN 14045/sshd
tcp 0 0 *:ipcserver *:* LISTEN 19498/rpc.statd
tcp 0 0 localhost.localdomain:smtp *:* LISTEN 3268/sendmail
tcp 0 0 localhost.lo:x11-ssh-offset *:* LISTEN 23982/sshd
tcp 0 0 *:794 *:* LISTEN 23511/rpc.mountd
tcp 0 0 localhost.localdomain:6011 *:* LISTEN 24022/sshd
tcp 0 0 localhost.localdomain:6012 *:* LISTEN 9462/sshd
tcp 0 0 *:24009 *:* LISTEN 26625/glusterfsd
tcp 0 0 *:ssh *:* LISTEN 14045/sshd
tcp 0 0 localhost6.l:x11-ssh-offset *:* LISTEN 23982/sshd
tcp 0 0 localhost6.localdomain:6011 *:* LISTEN 24022/sshd
tcp 0 0 localhost6.localdomain:6012 *:* LISTEN 9462/sshd
IPTABLES service has been stopped on all machines.
Any help you could give me would be greatly appreciated.
Thanks,
Steve Postma
________________________________
From: Steve Postma
Sent: Friday, November 16, 2012 10:51 AM
To: gluster-users at gluster.org
Subject: cant mount gluster volume
I have a 3 node gluster cluster running:
gluster --version
glusterfs 3.3.1 built on Oct 11 2012 22:01:05
It was previously running a 2.? version, uninstalled and installed 3.3.1
It appears to be working correctly.
gluster volume info
Volume Name: gdata
Type: Distribute
Volume ID: eccc3a90-212d-4563-ae8d-10a77758738d
Status: Started
Number of Bricks: 3
Transport-type: tcp
Bricks:
Brick1: gluster-0-0:/mseas-data-0-0
Brick2: gluster-0-1:/mseas-data-0-1
Brick3: gluster-data:/data
The client previously
I have a client with rpm's installed:
rpm -qa | grep gluster
glusterfs-fuse-3.3.1-1.el6.x86_64
glusterfs-3.3.1-1.el6.x86_64
glusterfs-debuginfo-3.3.1-1.el6.x86_64
I have a line in my fstab file of:
mseas-data:/gdata /gdata glusterfs defaults 0 0
This is copied from backups before client reinstall
I have also tried:
mount.glusterfs mseas-data:/gdata \gdata
>From /var/log/glusterfs/gdata.log
cat gdata.log
[2012-11-16 10:43:36.462998] I [glusterfsd.c:1666:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.3.1
[2012-11-16 10:43:36.471280] I [io-cache.c:1549:check_cache_size_ok] 0-gdata-quick-read: Max cache size is 25332322304
[2012-11-16 10:43:36.471347] I [io-cache.c:1549:check_cache_size_ok] 0-gdata-io-cache: Max cache size is 25332322304
[2012-11-16 10:43:36.474518] I [client.c:2142:notify] 0-gdata-client-0: parent translators are ready, attempting connect on transport
[2012-11-16 10:43:36.478166] I [client.c:2142:notify] 0-gdata-client-1: parent translators are ready, attempting connect on transport
[2012-11-16 10:43:36.481604] I [client.c:2142:notify] 0-gdata-client-2: parent translators are ready, attempting connect on transport
Given volfile:
+------------------------------------------------------------------------------+
1: volume gdata-client-0
2: type protocol/client
3: option remote-host gluster-0-0
4: option remote-subvolume /mseas-data-0-0
5: option transport-type tcp
6: end-volume
7:
8: volume gdata-client-1
9: type protocol/client
10: option remote-host gluster-0-1
11: option remote-subvolume /mseas-data-0-1
12: option transport-type tcp
13: end-volume
14:
15: volume gdata-client-2
16: type protocol/client
17: option remote-host gluster-data
18: option remote-subvolume /data
19: option transport-type tcp
20: end-volume
21:
22: volume gdata-dht
23: type cluster/distribute
24: subvolumes gdata-client-0 gdata-client-1 gdata-client-2
25: end-volume
26:
27: volume gdata-write-behind
28: type performance/write-behind
29: subvolumes gdata-dht
30: end-volume
31:
32: volume gdata-read-ahead
33: type performance/read-ahead
34: subvolumes gdata-write-behind
35: end-volume
36:
37: volume gdata-io-cache
38: type performance/io-cache
39: subvolumes gdata-read-ahead
40: end-volume
41:
42: volume gdata-quick-read
43: type performance/quick-read
44: subvolumes gdata-io-cache
45: end-volume
46:
47: volume gdata-stat-prefetch
48: type performance/stat-prefetch
49: subvolumes gdata-quick-read
50: end-volume
51:
52: volume gdata
53: type debug/io-stats
54: subvolumes gdata-stat-prefetch
55: end-volume
+------------------------------------------------------------------------------+
[2012-11-16 10:43:36.485221] E [socket.c:1715:socket_connect_finish] 0-gdata-client-2: connection to failed (No route to host)
[2012-11-16 10:43:36.485534] E [client-handshake.c:1717:client_query_portmap_cbk] 0-gdata-client-1: failed to get the port number for remote subvolume
[2012-11-16 10:43:36.485607] I [client.c:2090:client_rpc_notify] 0-gdata-client-1: disconnected
[2012-11-16 10:43:36.485626] E [client-handshake.c:1717:client_query_portmap_cbk] 0-gdata-client-0: failed to get the port number for remote subvolume
[2012-11-16 10:43:36.485647] I [client.c:2090:client_rpc_notify] 0-gdata-client-0: disconnected
[2012-11-16 10:43:36.492937] I [fuse-bridge.c:4191:fuse_graph_setup] 0-fuse: switched to graph 0
[2012-11-16 10:43:36.493966] I [fuse-bridge.c:3376:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.13 kernel 7.13
[2012-11-16 10:43:36.494088] E [dht-common.c:1372:dht_lookup] 0-gdata-dht: Failed to get hashed subvol for /
[2012-11-16 10:43:36.494214] E [dht-common.c:1372:dht_lookup] 0-gdata-dht: Failed to get hashed subvol for /
[2012-11-16 10:43:36.494231] W [fuse-bridge.c:513:fuse_attr_cbk] 0-glusterfs-fuse: 2: LOOKUP() / => -1 (Invalid argument)
[2012-11-16 10:43:36.500651] I [fuse-bridge.c:4091:fuse_thread_proc] 0-fuse: unmounting /var/log/glusterfs/gdata
[2012-11-16 10:43:36.501036] W [glusterfsd.c:831:cleanup_and_exit] (-->/lib64/libc.so.6(clone+0x6d) [0x3065ae5ccd] (-->/lib64/libpthread.so.0() [0x30666077f1] (-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xdd) [0x405d4d]))) 0-: received signum (15), shutting down
[2012-11-16 10:43:36.501072] I [fuse-bridge.c:4648:fini] 0-fuse: Unmounting '/var/log/glusterfs/gdata'.
It appears I am reaching the file system, but:
0-gdata-client-0: failed to get the port number for remote subvolume
0-gdata-dht: Failed to get hashed subvol for /
I have googled/troubleshot but am unable to find solution. Your help would be Greatly appreciated.
Thanks,
Steve Postma
More information about the Gluster-users
mailing list