[Gluster-users] Is Gluster the wrong solution for us?

[Scott Smith]

To answer Franco's post, my guess is that accesses will be pretty evenly balanced among the bricks.   There may be a slight bias to newer data, but our experience is that it's only a slight bias.

I really appreciate all of the great posts from everyone today.  I will consider what you have all said.  I'll continue to watch to see if there are any more comments.

Thanks again

Scott

-----Original Message-----
From: Franco Broi [mailto:franco.broi at iongeo.com] 
Sent: Wednesday, December 11, 2013 10:09 PM
To: Scott Smith
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Is Gluster the wrong solution for us?

If you are only adding disk space and don't necessarily need to increase bandwidth, then you wont need to rebalance. It's only a problem if you are adding clients and your most frequently accessed files are all on the same brick.

On Thu, 2013-12-12 at 04:49 +0000, Scott Smith wrote: 
> Pretty much, our files are never deleted.  We just keep adding more information.  Think of them as write once, read multiple, delete never.
> 
> -----Original Message-----
> From: Franco Broi [mailto:franco.broi at iongeo.com]
> Sent: Wednesday, December 11, 2013 7:31 PM
> To: Scott Smith
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] Is Gluster the wrong solution for us?
> 
> 
> How long-lived are your files? We have 400TB and are just about to double that but have decided not to rebalance the data, instead we are hoping that the disks will rebalance naturally through attrition and not waste any valuable time or bandwidth moving data around.
> 
> On Thu, 2013-12-12 at 01:15 +0000, Scott Smith wrote: 
> > We are about to abandon GlusterFS as a solution for our object 
> > storage needs.  I’m hoping to get some feedback to tell me whether 
> > we have missed something and are making the wrong decision.  We’re 
> > already a year into this project after evaluating a number of 
> > solutions.  I’d like not to abandon GlusterFS if we just misunderstand how it works.
> > 
> >  
> > 
> > Our use case is fairly straight forward.  We need to save a bunch of 
> > somewhat large files (1MB-100MB).  For the most part, these files 
> > are write once, read several times.  Our initial store is 80TB, but 
> > we expect to go to roughly 320TB fairly quickly.  After that, we 
> > expect to be adding another 80TB every few months.  We are using 
> > some COTS servers which we add in pairs; each server has 40TB of usable storage.
> > We intend to keep two copies of each file.  We currently run 4TB 
> > bricks
> > 
> >  
> > 
> > In our somewhat limited test environment, GlusterFS seemed to work 
> > well.  And, our initial introduction of GlusterFS into our 
> > production environment went well.  We had our initial 2 server 
> > (80TB) cluster about 50% full and things seemed to be going well.
> > 
> >  
> > 
> > Then we added another pair of servers (for a total of 160TB).  This 
> > went fine until we did the rebalance.  We were running 3.3.1.  We 
> > ran into the handle leak problem (which unfortunately we didn’t know 
> > about beforehand).  We also found that if any of the bricks went 
> > offline while the rebalance was going on, then files were lost or 
> > they lost their permissions.  We still don’t know why some of the 
> > bricks went offline, but they did and we have verified in our test 
> > environment that this is sufficient to cause the corruption problem.
> > 
> >  
> > 
> > The good news is that we think both of these problems got fixed in 
> > 3.4.1.  So why are we leaving?
> > 
> >  
> > 
> > In trying to figure out what was going on with our GlusterFS system 
> > after the disastrous rebalance, we ran across two posts.  The first 
> > one was 
> > http://hekafs.org/index.php/2012/03/glusterfs-algorithms-distribution/.  If we understand it correctly, anytime you add new storage servers to your cluster, you have to do a rebalance and that rebalance will require a minimum of 50% of the data in the cluster to be moved to make the hashing algorithms work.  This means that when we have a 320TB cluster and add another 80TB, we have to move at least 160TB just to get things back into balance.  Our estimate is that that will take months.  It probably won’t finish before we need to add another 80TB.
> > 
> >  
> > 
> > The other post we ran across was
> > http://www.gluster.org/community/documentation/index.php/Planning34/ElasticBrick.  This post seems to confirm our understanding of the rebalance.  It appears to be a discussion of the rebalance problem and a possible solution.  It was apparently discussed for 3.4, but didn’t make the cut.  
> > 
> >  
> > 
> > I’d be happy to find out that we just got it wrong.  Tell me that 
> > rebalancing doesn’t work the way we think.  Or maybe we should 
> > configure things different or something.
> > 
> >  
> > 
> > My problem is that if GlusterFS isn’t good for starting with a small 
> > cluster (80TB) and growing over time to half a petabyte, what is the 
> > use case it is intended for?  Do you really have to start out with 
> > the amount of storage you think you’ll need in the long-run and just 
> > fill it up as you go?  That’s why I’m nervous about our 
> > understanding of the rebalance.  It’s hard to believe it works this 
> > way (at least from our perspective).
> > 
> >  
> > 
> > We have a lot of man hours into writing code and putting 
> > infrastructure in for GlusterFS.  We can likely reuse much of it for 
> > another system.  I would just like to know that we really do 
> > understand the rebalance and that it really works the way I 
> > described it before we start evaluating other object store solutions.
> > 
> >  
> > 
> > Comments?
> > 
> >  
> > 
> > Scott
> > 
> >  
> > 
> >  
> > 
> > 
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> 
>