[Gluster-users] can't write to gluster when local disk full
m.c.wilkins at massey.ac.nz
m.c.wilkins at massey.ac.nz
Wed Apr 29 07:32:18 UTC 2009
OK to answer my own question, the problem was due to me writing a
really big file. there was space on the local brick at the start of
the write, but I filled that brick up during the write. No problem
making new files.
Matt
On Wed, Apr 29, 2009 at 03:34:06PM +1200, m.c.wilkins at massey.ac.nz wrote:
>
>
> Hi,
>
> I have three machines in a NUFA arrangment. When the local disk is
> full no writes can be made to the gluster disk, even though there is
> space on the other disks?! Thanks for any help.
>
> Perhaps I have done something wrong, this is very odd. Here is what
> df says:
>
> tur-awc1:~> df -hl
> Filesystem Size Used Avail Use% Mounted on
> /dev/mapper/sysvg-root
> 7.7G 3.1G 4.3G 42% /
> /dev/sda1 92M 25M 62M 29% /boot
> tmpfs 7.9G 0 7.9G 0% /dev/shm
> /dev/sdb2 6.3T 3.4T 2.9T 55% /data
> /dev/sdb1 5.4T 2.8T 2.7T 51% /export/brick0
> glusterfs 26T 12T 15T 46% /awc/data
> /dev/mapper/sysvg-newgluster
> 886G 886G 63M 100% /export/brick-newgluster
> glusterfs#/usr/local/glusterfs-2.0.0rc9/etc/glusterfs/glusterfsd.vol
> 2.8T 887G 1.9T 32% /ng
>
> Looks like plenty of room on the gluster, but my local brick is full,
> and I can't write to /ng. Here is the log:
>
>
> ================================================================================
> Version : glusterfs 2.0.0git built on Apr 29 2009 11:08:01
> TLA Revision : git://git.sv.gnu.org/gluster.git
> Starting Time: 2009-04-29 11:35:46
> Command line : /usr/local/glusterfs-2.0.0rc9/sbin/glusterfsd -f /usr/local/glusterfs-2.0.0rc9/etc/glusterfs/glusterfsd.vol /ng
> PID : 22711
> System name : Linux
> Nodename : tur-awc1
> Kernel Release : 2.6.18-92.1.13.el5
> Hardware Identifier: x86_64
>
> Given volfile:
> +------------------------------------------------------------------------------+
> 1: volume posix0
> 2: type storage/posix
> 3: option directory /export/brick-newgluster
> 4: end-volume
> 5:
> 6: volume locks0
> 7: type features/locks
> 8: subvolumes posix0
> 9: end-volume
> 10:
> 11: volume brick0
> 12: type performance/io-threads
> 13: subvolumes locks0
> 14: end-volume
> 15:
> 16: volume server
> 17: type protocol/server
> 18: option transport-type tcp
> 19: option listen-port 16996
> 20: option auth.addr.brick0.allow *
> 21: subvolumes brick0
> 22: end-volume
> 23:
> 24: volume tur-awc1-0
> 25: type protocol/client
> 26: option transport-type tcp
> 27: option remote-port 16996
> 28: option remote-host tur-awc1
> 29: option remote-subvolume brick0
> 30: option ping-timeout 120
> 31: end-volume
> 32:
> 33: volume tur-awc2-0
> 34: type protocol/client
> 35: option transport-type tcp
> 36: option remote-port 16996
> 37: option remote-host tur-awc2
> 38: option remote-subvolume brick0
> 39: option ping-timeout 120
> 40: end-volume
> 41:
> 42: volume tur-awc3-0
> 43: type protocol/client
> 44: option transport-type tcp
> 45: option remote-port 16996
> 46: option remote-host tur-awc3
> 47: option remote-subvolume brick0
> 48: option ping-timeout 120
> 49: end-volume
> 50:
> 51: volume nufa
> 52: type cluster/nufa
> 53: option local-volume-name tur-awc1-0
> 54: subvolumes tur-awc1-0 tur-awc2-0 tur-awc3-0
> 55: end-volume
> 56:
> 57: volume writebehind
> 58: type performance/write-behind
> 59: option cache-size 1MB
> 60: subvolumes nufa
> 61: end-volume
> 62:
> 63: # before or after writebehind?
> 64: volume ra
> 65: type performance/read-ahead
> 66: subvolumes writebehind
> 67: end-volume
> 68:
> 69: volume cache
> 70: type performance/io-cache
> 71: option cache-size 512MB
> 72: subvolumes ra
> 73: end-volume
> 74:
>
> +------------------------------------------------------------------------------+
> 2009-04-29 11:35:46 W [xlator.c:555:validate_xlator_volume_options] server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction
> 2009-04-29 11:35:46 N [glusterfsd.c:1152:main] glusterfs: Successfully started
> 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc2-0: connection failed (Connection refused)
> 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected
> 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected
> 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc2-0: connection failed (Connection refused)
> 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected
> 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc2-0: disconnected
> 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc1-0: connection failed (Connection refused)
> 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected
> 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected
> 2009-04-29 11:35:46 E [socket.c:743:socket_connect_finish] tur-awc1-0: connection failed (Connection refused)
> 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected
> 2009-04-29 11:35:46 N [client-protocol.c:7014:notify] tur-awc1-0: disconnected
> 2009-04-29 11:35:46 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc3-0: connection and handshake succeeded
> 2009-04-29 11:35:46 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc3-0: connection and handshake succeeded
> 2009-04-29 11:35:51 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.35:1020
> 2009-04-29 11:35:51 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.35:1019
> 2009-04-29 11:35:52 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.28:1017
> 2009-04-29 11:35:52 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.128.28:1016
> 2009-04-29 11:35:57 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.129.121:1022
> 2009-04-29 11:35:57 W [client-protocol.c:6320:client_setvolume_cbk] tur-awc1-0: attaching to the local volume 'brick0'
> 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc1-0: connection and handshake succeeded
> 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc2-0: connection and handshake succeeded
> 2009-04-29 11:35:57 N [server-protocol.c:7502:mop_setvolume] server: accepted client from 130.123.129.121:1016
> 2009-04-29 11:35:57 W [client-protocol.c:6320:client_setvolume_cbk] tur-awc1-0: attaching to the local volume 'brick0'
> 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc1-0: connection and handshake succeeded
> 2009-04-29 11:35:57 N [client-protocol.c:6327:client_setvolume_cbk] tur-awc2-0: connection and handshake succeeded
> 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 14:35:06 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> <snip>
> 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 15:18:23 W [posix.c:1697:posix_writev] posix0: writev failed: No space left on device
> 2009-04-29 15:18:23 E [fuse-bridge.c:923:fuse_err_cbk] glusterfs-fuse: 4939521: FLUSH() ERR => -1 (No space left on device)
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list