[Gluster-users] brick out of space, unmounted brick

[Jeff Shaw]

Hello Gluster users,
Before I put Gluster into production, I am wondering how it determines whether a byte can be written, and where I should look in the source code to change these behaviors. My experiences are with glusterfs 3.2.4 on CentOS 6 64-bit.

Suppose I have a Gluster volume made up of four 1 MB bricks, like this

Volume Name: test
Type: Distributed-Replicate
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gluster0-node0:/brick0
Brick2: gluster0-node1:/brick1
Brick3: gluster0-node0:/brick2
Brick4: gluster0-node1:/brick3

The mounted Gluster volume will report that the size of the volume is 2 MB, which creates a false impression that it can hold a 2 MB file. This isn't too bad, since people are used to a file system's maximum file size being smaller than the file system's maximum total size.

Scenario 1: One brick runs out of space first.

Taking this a step further, suppose brick0 is actually 2 MB, and I attempt to copy a file having 2 MB to the Gluster volume. If Gluster chooses to copy the file to brick0 and brick1, then the copy succeeds, although brick1 only stores half the file. When brick0 fails, only half of the file is available for reading. It would be better if Gluster failed to continue writing when one brick in the replication group ran out of space.

Scenario 2: One brick is umounted.

Suppose after Scenario 1 completes, brick0 goes offline. Then, a user attempts to retrieve the 2 MB file. The user receives the file fragment. Because gluster0-node0:/brick0 is unmounted, the file doesn't exist at that location, and so the gateway copies the file fragment from gluster0-node1:/brick1 onto gluster0-node0:/brick0. Then, even worse, the user starts copying files onto the Gluster volume. All the files destined for the first replication group appear under /brick0, even though it's unmounted. This eventually will fill up the root file system.

I think to fix this, when creating a file, Gluster should make sure that the file system that the brick was originally created on is mounted. Also, perhaps bricks should only be able to be created at mount points. A colleague of mine suggested mounting all the Gluster bricks within another file system's path that's read only. 

Gluster's source code is quite large, so if someone could point me to the right files to edit, I'd be happy to change its behavior to match what I expect.

Thanks,
Jeff