[Gluster-users] Write failure on distributed volume with free space available

[Nux!]

Hello,

Thanks to "partner" on IRC who told me about this (quite big) problem.
Apparently in a distributed setup once a brick fills up you start 
getting write failures. Is there a way to work around this?

I would have thought gluster would check for free space before writing 
to a brick.

It's very easy to test, I created a distributed volume from 2 uneven 
bricks and started to write to it; as one of them got full I started 
getting write failures. This is how I tested:

=======================

[root at localhost gluster1]# for i in `seq 1 20`; do df -h 
/mnt/gluster1/; dd if=/dev/zero of=16_$i bs=16M count=1; done
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M   16M  276M   6% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.115813 s, 145 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M   32M  260M  11% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.140746 s, 119 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M   48M  243M  17% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.0905644 s, 185 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M   64M  227M  22% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.088424 s, 190 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M   80M  211M  28% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.0876373 s, 191 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M   96M  195M  33% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.0890243 s, 188 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  112M  179M  39% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.0853196 s, 197 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  128M  163M  45% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.0923682 s, 182 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  145M  147M  50% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.0861475 s, 195 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  161M  131M  56% /mnt/gluster1
dd: writing `16_10': No space left on device
dd: closing output file `16_10': No space left on device
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  170M  121M  59% /mnt/gluster1
dd: writing `16_11': No space left on device
dd: closing output file `16_11': No space left on device
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  170M  121M  59% /mnt/gluster1
dd: opening `16_12': No space left on device
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  170M  121M  59% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.0842241 s, 199 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  186M  105M  64% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.102602 s, 164 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  202M   89M  70% /mnt/gluster1
dd: opening `16_15': No space left on device
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  202M   89M  70% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.0866302 s, 194 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  219M   73M  76% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.0898677 s, 187 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  235M   57M  81% /mnt/gluster1
dd: opening `16_18': No space left on device
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  235M   57M  81% /mnt/gluster1
1+0 records in
1+0 records out
16777216 bytes (17 MB) copied, 0.126375 s, 133 MB/s
Filesystem            Size  Used Avail Use% Mounted on
192.168.192.5:/test   291M  251M   41M  87% /mnt/gluster1
dd: opening `16_20': No space left on device
=======================



-- 
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro