[Gluster-users] can't write to gluster when local disk full
m.c.wilkins at massey.ac.nz
m.c.wilkins at massey.ac.nz
Wed Apr 29 03:34:06 UTC 2009
Hi,
I have three machines in a NUFA arrangment. When the local disk is
full no writes can be made to the gluster disk, even though there is
space on the other disks?! Thanks for any help.
Perhaps I have done something wrong, this is very odd. Here is what
df says:
tur-awc1:~> df -hl
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/sysvg-root
7.7G 3.1G 4.3G 42% /
/dev/sda1 92M 25M 62M 29% /boot
tmpfs 7.9G 0 7.9G 0% /dev/shm
/dev/sdb2 6.3T 3.4T 2.9T 55% /data
/dev/sdb1 5.4T 2.8T 2.7T 51% /export/brick0
glusterfs 26T 12T 15T 46% /awc/data
/dev/mapper/sysvg-newgluster
886G 886G 63M 100% /export/brick-newgluster
glusterfs#/usr/local/glusterfs-2.0.0rc9/etc/glusterfs/glusterfsd.vol
2.8T 887G 1.9T 32% /ng
Looks like plenty of room on the gluster, but my local brick is full,
and I can't write to /ng. Here is the log:
================================================================================
Version : glusterfs 2.0.0git built on Apr 29 2009 11:08:01
TLA Revision : git://git.sv.gnu.org/gluster.git
Starting Time: 2009-04-29 11:35:46
Command line : /usr/local/glusterfs-2.0.0rc9/sbin/glusterfsd -f /usr/local/glusterfs-2.0.0rc9/etc/glusterfs/glusterfsd.vol /ng
PID : 22711
System name : Linux
Nodename : tur-awc1
Kernel Release : 2.6.18-92.1.13.el5
Hardware Identifier: x86_64
Given volfile:
+------------------------------------------------------------------------------+
1: volume posix0
2: type storage/posix
3: option directory /export/brick-newgluster
4: end-volume
5:
6: volume locks0
7: type features/locks
8: subvolumes posix0
9: end-volume
10:
11: volume brick0
12: type performance/io-threads
13: subvolumes locks0
14: end-volume
15:
16: volume server
17: type protocol/server
18: option transport-type tcp
19: option listen-port 16996
20: option auth.addr.brick0.allow *
21: subvolumes brick0
22: end-volume
23:
24: volume tur-awc1-0
25: type protocol/client
26: option transport-type tcp
27: option remote-port 16996
28: option remote-host tur-awc1
29: option remote-subvolume brick0
30: option ping-timeout 120
31: end-volume
32:
33: volume tur-awc2-0
34: type protocol/client
35: option transport-type tcp
36: option remote-port 16996
37: option remote-host tur-awc2
38: option remote-subvolume brick0
39: option ping-timeout 120
40: end-volume
41:
42: volume tur-awc3-0
43: type protocol/client
44: option transport-type tcp
45: option remote-port 16996
46: option remote-host tur-awc3
47: option remote-subvolume brick0
48: option ping-timeout 120
49: end-volume
50:
51: volume nufa
52: type cluster/nufa
53: option local-volume-name tur-awc1-0
54: subvolumes tur-awc1-0 tur-awc2-0 tur-awc3-0
55: end-volume
56:
57: volume writebehind
58: type performance/write-behind
59: option cache-size 1MB
60: subvolumes nufa
61: end-volume
62:
63: # before or after writebehind?
64: volume ra
65: type performance/read-ahead
66: subvolumes writebehind
67: end-volume
68:
69: volume cache
70: type performance/io-cache
71: option cache-size 512MB
72: subvolumes ra
73: end-volume
74:
+------------------------------------------------------------------------------+
2009-04-29 11:35:46 W [xlator.c:555:validate_xlator_volume_options] server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
2009-04-29 11:35:46 N [glusterfsd.c:1152:main] glusterfs: Successfully started
2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc2-0: connection failed (Connection refused)
2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected
2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected
2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc2-0: connection failed (Connection refused)
2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected
2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected
2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc1-0: connection failed (Connection refused)
2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected
2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected
2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc1-0: connection failed (Connection refused)
2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected
2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected
2009-04-29 11:35:46 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc3-0: connection and handshake succeeded
2009-04-29 11:35:46 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc3-0: connection and handshake succeeded
2009-04-29 11:35:51 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.35:1020
2009-04-29 11:35:51 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.35:1019
2009-04-29 11:35:52 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.28:1017
2009-04-29 11:35:52 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.28:1016
2009-04-29 11:35:57 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.129.121:1022
2009-04-29 11:35:57 W [client-protocol.c:6320:client_setvolume_cbk] tur-awc1-0: attaching to the local volume 'brick0'
2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc1-0: connection and handshake succeeded
2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc2-0: connection and handshake succeeded
2009-04-29 11:35:57 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.129.121:1016
2009-04-29 11:35:57 W [client-protocol.c:6320:client_setvolume_cbk] tur-awc1-0: attaching to the local volume 'brick0'
2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc1-0: connection and handshake succeeded
2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc2-0: connection and handshake succeeded
2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
<snip>
2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
2009-04-29 15:18:23 E [fuse-bridge.c:923:fuse_err_cbk] glusterfs-fuse: 4939521: FLUSH() ERR => -1 (No space left on device)
More information about the Gluster-users
mailing list