[Gluster-users] Failed rebalance resulting in major problems

Wed Nov 6 20:57:17 UTC 2013

Joe,

You're right-- I probably should have dialed it back a bit! It's
frustrating sometimes when I post about such a major issue and never see
any reply.

In my case, I run into gfid bugs regularly, almost always in situations
where I have copied an entire directory tree into a GlusterFS mount.  There
have been no connectivity issues between nodes, no node restarts, etc, for
months, but once in a while, I get a gfid mismatch and must manually
correct the situation.

I would certainly purchase GlusterFS support if I had any option other than
Red Hat-- they only support Red Hat Storage and that isn't a good fit for
my environment at this time.  If GlusterFS is successful the way it could
be, there will definitely be an opportunity for a firm to support it on
non-RedHat platforms.

FWIW, I've created a Github repo to store my scripts for navigating
GlusterFS issues.  If they remain relevant and the repo gets activity, I'll
go to Gluster Forge.  https://github.com/justindossey/gluster-scripts

On Wed, Nov 6, 2013 at 12:15 PM, Joe Julian <joe at julianfamily.org> wrote:

>  On 11/06/2013 11:52 AM, Justin Dossey wrote:
>
> Shawn,
>
>  I had a very similar experience with a rebalance on 3.3.1, and it took
> weeks to get everything straightened out.  I would be happy to share the
> scripts I wrote to correct the permissions issues if you wish, though I'm
> not sure it would be appropriate to share them directly on this list.
>  Perhaps I should just create a project on Github that is devoted to
> collecting scripts people use to fix their GlusterFS environments!
>
>  After that (awful) experience, I am loath to run further rebalances.
>  I've even spent days evaluating alternatives to GlusterFS, as my
> experience with this list over the last six months indicates that support
> for community users is minimal, even in the face of major bugs such as the
> one with rebalancing and the continuing "gfid different on subvolume" bugs
> with 3.3.2.
>
> I'm one of oldest GlusterFS users around here and one of the biggest
> proponents and even I have been loath to rebalance until 3.4.1.
>
> There are no open bugs for gfid mismatches that I could find. The last
> time someone mentioned that error in IRC it was 2am, I was at a convention,
> and I told the user how to solve that problem (
> http://irclog.perlgeek.de/gluster/2013-06-14#i_7196149 ). It was caused
> by split-brain. If you have a bug, it would be more productive to file it
> rather than make negative comments about a community of people that have no
> requirement to help anybody, but do it anyway just because they're nice
> people.
>
> This is going to sound snarky because it's in text, but I mean this
> sincerely. If community support is not sufficient, you might consider
> purchasing support from a company that provides it professionally.
>
>
>
>  Let me know what you think of the Github thing and I'll proceed
> appropriately.
>
> Even better, put them up on http://forge.gluster.org
>
>
>
>
> On Tue, Nov 5, 2013 at 9:05 PM, Shawn Heisey <gluster at elyograg.org> wrote:
>
>> We recently added storage servers to our gluster install, running 3.3.1
>> on CentOS 6.  It went from 40TB usable (8x2 distribute-replicate) to
>> 80TB usable (16x2).  There was a little bit over 20TB used space on the
>> volume.
>>
>> The add-brick went through without incident, but the rebalance failed
>> after moving 1.5TB of the approximately 10TB that needed to be moved.  A
>> side issue is that it took four days for that 1.5TB to move.  I'm aware
>> that gluster has overhead, and that there's only so much speed you can
>> get out of gigabit, but a 100Mb/s half-duplex link could have copied the
>> data faster if it had been a straight copy.
>>
>> After I discovered that the rebalance had failed, I noticed that there
>> were other problems.  There are a small number of completely lost files
>> (91 that I know about so far), a huge number of permission issues (over
>> 800,000 files changed to 000), and about 32000 files that are throwing
>> read errors via the fuse/nfs mount but seem to be available directly on
>> bricks.  That last category of problem file has the sticky bit set, with
>> almost all of them having ---------T permissions.  The good files on
>> bricks typically have the same permissions, but are readable by root.  I
>> haven't worked out the scripting necessary to automate all the fixing
>> that needs to happen yet.
>>
>> We really need to know what happened.  We do plan to upgrade to 3.4.1,
>> but there were some reasons that we didn't want to upgrade before adding
>> storage.
>>
>> * Upgrading will result in service interruption to our clients, which
>> mount via NFS.  It would likely be just a hiccup, with quick failover,
>> but it's still a service interruption.
>> * We have a pacemaker cluster providing the shared IP address for NFS
>> mounting.  It's running CentOS 6.3.  A "yum upgrade" to upgrade gluster
>> will also upgrade to CentOS 6.4.  The pacemaker in 6.4 is incompatible
>> with the pacemaker in 6.3, which will likely result in
>> longer-than-expected downtime for the shared IP address.
>> * We didn't want to risk potential problems with running gluster 3.3.1
>> on the existing servers and 3.4.1 on the new servers.
>> * We needed the new storage added right away, before we could schedule
>> maintenance to deal with the upgrade issues.
>>
>> Something that would be extremely helpful would be obtaining the
>> services of an expert-level gluster consultant who can look over
>> everything we've done to see if there is anything we've done wrong and
>> how we might avoid problems in the future.  I don't know how much the
>> company can authorize for this, but we obviously want it to be as cheap
>> as possible.  We are in Salt Lake City, UT, USA.  It would be preferable
>> to have the consultant be physically present at our location.
>>
>> I'm working on redacting one bit of identifying info from our rebalance
>> log, then I can put it up on dropbox for everyone to examine.
>>
>> Thanks,
>> Shawn
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>
>
>
>  --
> Justin Dossey
> CTO, PodOmatic
>
>
>
> _______________________________________________
> Gluster-users mailing listGluster-users at gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>

-- 
Justin Dossey
CTO, PodOmatic
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131106/8499f64f/attachment.html>