[Gluster-devel] GlusterFS Spare Bricks?

[Anand Babu Periasamy]

> -----Original Message-----
> From: abperiasamy at gmail.com [mailto:abperiasamy at gmail.com] On Behalf Of
> Anand Babu Periasamy
> Sent: Wednesday, April 11, 2012 10:13 AM
> To: 7220022
> Cc: gluster-devel at nongnu.org
> Subject: Re: [Gluster-devel] GlusterFS Spare Bricks?
>
> On Tue, Apr 10, 2012 at 1:39 AM, 7220022 <7220022 at gmail.com> wrote:
> >
> > Are there plans to add provisioning of spare bricks in a replicated (or
> distributed-replicated) configuration? E.g., when a brick in a mirror set
> dies, the system rebuilds it automatically on a spare, similar to how it'd
> done by RAID controllers.
> >
> >
> >
> > Nor would it only improve the practical reliability, especially of large
> clusters, but it'd also make it possible to make better-performing
> clusters
> off less expensive components. For example, instead of having slow RAID5
> bricks on expensive RAID controllers one uses cheap HBA-s and stripes a
> few
> disks per brick in RAID0 - that's faster for writes than RAID 5/6 by an
> order of magnitude (and, by the way, should improve rebuild times in
> Gluster
> many are complaining about.).  A failure of one such striped brick is not
> catastrophic in a mirrored Gluster - but it's better to have spare bricks
> standing by strewn across cluster heads.
> >
> >
> >
> > A more advanced setup at a hardware level involves creating "hybrid
> > disks"
> whereas HDD vdisks are cached by enterprise-class SSD-s.  It works
> beautifully and makes HDD-s amazingly fast for random transactions.  The
> technology's become widely available for many $500 COTS controllers.
> However, it is not widely known that the results with HDD-s in RAID0 under
> SSD cache are 10 to 20 (!!) times better than with RAID 5 or 6.
> >
> >
> >
> > There is no way to use RAID0 in commercial storage, the main reason
> > being
> the absence of hot-spares.  If on the other hand the spares are handled by
> Gluster in a form of (cached hardware-RAID0) pre-fabricated bricks both
> very
> good performance and reasonably sufficient redundancy should be easily
> achieved.
>
> Why not use "gluster volume replace-brick ..." command. You can use
> external
> monitoring/management tools (eg. freeipmi) to detect node failures and
> trigger replace brick through a script. GlusterFS has the mechanism for
> hot
> spare, but the policy should be external.
>
> [AS] That should work, but still it'd be prone to human error.  In our
> experience, if we've not had hotspares (block storage) we'd have surely
> experienced catastrophic failures.  First-off, COTS disks (and
> controllers,
> if we talk GlusterFS nodes) have a break-in period when the bad ones fail
> under load within a few months.  Secondly, a lot of our equipment is in
> remote telco facilities where power, cleanliness or airconditioning can be
> far from ideal - leading to  increasing failure rates about 2 years after
> deployment.  As a rule, we have at least 4 hotspares per two 24-bay
> enclosures, while our sister company with similar use profile does 4-6
> spares per enclosure, as they run older and less uniform equipment.
>
> A node may come back online in 5 mins, GlusterFS should not automatically
> make decisions.
> [AS] Good point, e.g. down for maintenance
>
>  I am thinking if it makes sense to add hot-spare as a standard feature,
> because GlusterFS detects failures.
>
> [AS] Given the reason above it'd be best if the feature could be turned on
> and off.  Before attempting maintenance - turn off.  Maintenance complete
> and node up - the "turn hotspare on" command is issued, but it's queued
> until the reconstruction of the node begins - and takes it into
> consideration (won't attempt to sync to spare bricks in case
> reconstruction
> to other good bricks has already began).
>
> In half the cases, failed disks and controllers fail randomly and
> temporarily (due to dust, bad power etc.)  Most of the time the root cause
> is unknown or is impractical to debug in a live system.  Block storage
> SAN-s
> have more or less standard configuration tools that also take that into
> account.  Here's a brief description in their terminology, which may help
> creating the logic in GlusterFS:
>
> 1. Drives can have the statuses of Online, Unconfigured Good, Unconfigured
> Bad, Spare (LSP, a spare local to the drive group,) Global Spare (GSP,
> across the system) and Foreign.
> 2. vDisks can be Optimal, Degraded and Degraded, Rebuilding
> 3.  In presence of spares, if a drive in a redundant vDisk fails the
> system
> marks the drive as Unconfigured Bad and the vDisk picks up the spare and
> enters the Rebuilding mode.
> 4.  The system won't let you make an Unconfigured Bad drive Online.  But
> you
> can try a "make unconfigured good" command on it.  if successful, and it
> passes initialization and it won't show trouble in SMART - include it in a
> new vDisk, make it a spare, etc.  If it's bad - replace it.
>

Very useful points. Took notes.
-ab