[Gluster-users] Is Gluster the wrong solution for us?

[Franco Broi]

If you are only adding disk space and don't necessarily need to increase
bandwidth, then you wont need to rebalance. It's only a problem if you
are adding clients and your most frequently accessed files are all on
the same brick.

On Thu, 2013-12-12 at 04:49 +0000, Scott Smith wrote: 
> Pretty much, our files are never deleted.  We just keep adding more information.  Think of them as write once, read multiple, delete never.
> 
> -----Original Message-----
> From: Franco Broi [mailto:franco.broi at iongeo.com] 
> Sent: Wednesday, December 11, 2013 7:31 PM
> To: Scott Smith
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] Is Gluster the wrong solution for us?
> 
> 
> How long-lived are your files? We have 400TB and are just about to double that but have decided not to rebalance the data, instead we are hoping that the disks will rebalance naturally through attrition and not waste any valuable time or bandwidth moving data around.
> 
> On Thu, 2013-12-12 at 01:15 +0000, Scott Smith wrote: 
> > We are about to abandon GlusterFS as a solution for our object storage 
> > needs.  I’m hoping to get some feedback to tell me whether we have 
> > missed something and are making the wrong decision.  We’re already a 
> > year into this project after evaluating a number of solutions.  I’d 
> > like not to abandon GlusterFS if we just misunderstand how it works.
> > 
> >  
> > 
> > Our use case is fairly straight forward.  We need to save a bunch of 
> > somewhat large files (1MB-100MB).  For the most part, these files are 
> > write once, read several times.  Our initial store is 80TB, but we 
> > expect to go to roughly 320TB fairly quickly.  After that, we expect 
> > to be adding another 80TB every few months.  We are using some COTS 
> > servers which we add in pairs; each server has 40TB of usable storage.
> > We intend to keep two copies of each file.  We currently run 4TB 
> > bricks
> > 
> >  
> > 
> > In our somewhat limited test environment, GlusterFS seemed to work 
> > well.  And, our initial introduction of GlusterFS into our production 
> > environment went well.  We had our initial 2 server (80TB) cluster 
> > about 50% full and things seemed to be going well.
> > 
> >  
> > 
> > Then we added another pair of servers (for a total of 160TB).  This 
> > went fine until we did the rebalance.  We were running 3.3.1.  We ran 
> > into the handle leak problem (which unfortunately we didn’t know about 
> > beforehand).  We also found that if any of the bricks went offline 
> > while the rebalance was going on, then files were lost or they lost 
> > their permissions.  We still don’t know why some of the bricks went 
> > offline, but they did and we have verified in our test environment 
> > that this is sufficient to cause the corruption problem.
> > 
> >  
> > 
> > The good news is that we think both of these problems got fixed in 
> > 3.4.1.  So why are we leaving?
> > 
> >  
> > 
> > In trying to figure out what was going on with our GlusterFS system 
> > after the disastrous rebalance, we ran across two posts.  The first 
> > one was 
> > http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/.  If we understand it correctly, anytime you add new storage servers to your cluster, you have to do a rebalance and that rebalance will require a minimum of 50% of the data in the cluster to be moved to make the hashing algorithms work.  This means that when we have a 320TB cluster and add another 80TB, we have to move at least 160TB just to get things back into balance.  Our estimate is that that will take months.  It probably won’t finish before we need to add another 80TB.
> > 
> >  
> > 
> > The other post we ran across was
> > http://www.gluster.org/community/documentation/index.php/Planning34/ElasticBrick.  This post seems to confirm our understanding of the rebalance.  It appears to be a discussion of the rebalance problem and a possible solution.  It was apparently discussed for 3.4, but didn’t make the cut.  
> > 
> >  
> > 
> > I’d be happy to find out that we just got it wrong.  Tell me that 
> > rebalancing doesn’t work the way we think.  Or maybe we should 
> > configure things different or something.
> > 
> >  
> > 
> > My problem is that if GlusterFS isn’t good for starting with a small 
> > cluster (80TB) and growing over time to half a petabyte, what is the 
> > use case it is intended for?  Do you really have to start out with the 
> > amount of storage you think you’ll need in the long-run and just fill 
> > it up as you go?  That’s why I’m nervous about our understanding of 
> > the rebalance.  It’s hard to believe it works this way (at least from 
> > our perspective).
> > 
> >  
> > 
> > We have a lot of man hours into writing code and putting 
> > infrastructure in for GlusterFS.  We can likely reuse much of it for 
> > another system.  I would just like to know that we really do 
> > understand the rebalance and that it really works the way I described 
> > it before we start evaluating other object store solutions.
> > 
> >  
> > 
> > Comments?
> > 
> >  
> > 
> > Scott
> > 
> >  
> > 
> >  
> > 
> > 
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
>