[Gluster-users] problem with missing bricks

[Todd Pfaff]

Gluster-user folks,

I'm trying to use gluster in a way that may be a considered an unusual use
case for gluster.  Feel free to let me know if you think what I'm doing
is dumb.  It just feels very comfortable doing this with gluster.
I have been using gluster in other, more orthodox configurations, for
several years.

I have a single system with 45 inexpensive sata drives - it's a self-built
backblaze similar to that documented at this url but with some upgrades
and substitutions:

   http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/

We use this system for disk-to-disk backups only, no primary storage,
nothing mission critical.

For the past two years I have been using this system with linux software
raid, with the drives organized as multiple raid 5/6/10 sets of 5 drives
per set.  This has worked ok but I have suffered enough multiple
simultaneous drive failures to prompt me to explore alternatives to raid.
Yes, I know, that's what I get for using cheap sata drives.

What I'm experimenting with now is creating gluster distributed-replicated
volumes on some of these drives, and maybe all in the future if this works
reasonably well.

At this point I am using 10 of the drives configured as shown here:

   Volume Name: volume1
   Type: Distributed-Replicate
   Status: Started
   Number of Bricks: 5 x 2 = 10
   Transport-type: tcp
   Bricks:
   Brick1: host:/gluster/brick01
   Brick2: host:/gluster/brick06
   Brick3: host:/gluster/brick02
   Brick4: host:/gluster/brick07
   Brick5: host:/gluster/brick03
   Brick6: host:/gluster/brick08
   Brick7: host:/gluster/brick04
   Brick8: host:/gluster/brick09
   Brick9: host:/gluster/brick05
   Brick10: host:/gluster/brick10
   Options Reconfigured:
   auth.allow: 127.0.0.1,10.10.10.10


For the most part this is working fine so far.  The problem I have run
into several times now is that when a drive fails and the system is
rebooted, the volume comes up without that brick.  Gluster happily writes
to the missing brick's mount point, thereby eventually filling up the root
filesystem.  Once the root filesystem is full and processes writing to
gluster space are hung, I can never recover from this state without
rebooting.

Is there any way to avoid this problem of gluster writing to a brick
path that isn't really populated by the intended brick filesystem?
Does gluster not create any sort of signature or meta-data that
indicates whether or not a path is really a gluster brick?
How do others deal with missing bricks?

I realize that ultimately I should get the bricks replaced as soon as
possible but there may be times when I want to continue running for some
time with a "degraded" volume if you will.

Any and all ideas, suggestions, comments, criticisms are welcome.

Cheers and Happy New Year,
Todd