[Gluster-users] Slow healing times on large cinder and nova volumes

Schmid, Larry lschmid at io.com
Tue Apr 22 04:45:37 UTC 2014


Thanks for your reply.  I will attach logs as soon as I can.

We are on 3.4.1 and I followed this process: http://goo.gl/hFwCcB

which essentially details how to set attributes so the new disk space is identified as the brick, after which full heal commands are issued on the volumes to start replication.

This is the same process I used to successfully heal several other volumes critical to our openstack environment.

In addition, I bumped up the background heal threads to 32 per volume (the exact parameter name escapes me at the moment.)  This has been a try-and-see exercise while I watch disk I/O on the surviving node.  I'm not sure how high this parameter can go before bad things happen.

Thx,
Larry Schmid

IO

M +1.602.316.8639
E   lschmid at io.com | io.com

________________________________________
From: Pranith Kumar Karampuri <pkarampu at redhat.com>
Sent: Monday, April 21, 2014 6:43 PM
To: Schmid, Larry
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Slow healing times on large cinder and nova volumes

Could you attach log files please.
You said the bricks are replaced. In case of brick-replacement, index based self-heal doesn't work so full self-heal needs to be triggered using "gluster volume heal <volname> full". Could you confirm if that command is issued?

Pranith
----- Original Message -----
> From: "Larry Schmid" <lschmid at io.com>
> To: gluster-users at gluster.org
> Sent: Tuesday, April 22, 2014 4:07:39 AM
> Subject: [Gluster-users] Slow healing times on large cinder and nova volumes
>
>
>
> Hi guys,
>
>
>
> x-posted from irc.
>
>
>
> We're having an issue on our prod openstack environment, which is backed by
> gluster using two replicas (I know. I wasn't given a choice.)
>
>
>
> We lost storage on one of the replica servers and so had to replace failed
> bricks. The heal operation on Cinder and Nova volumes is coming up on the
> two-week mark and it seems as if it will never catch up and finish.
>
>
>
> Nova heal info shows a constantly fluctuating list with multiple heals on
> many of the files, as if it's trying to keep up with deltas. It’s at 860GB
> of 1.1TB.
>
>
>
> Cinder doesn't really seem to progress. It's at about 1.9T out of 6T
> utilized, though the total sparse file size totals about 30T. It also has
> done multiple heals on the some files.
>
>
>
> I seem to be down to just watching it spin. Any help or tips?
>
>
>
> Thanks,
>
>
>
> Larry Schmid | Principal Cloud Engineer
>
>
>
> IO
>
>
>
> M +1.602.316.8639 | O +1.602.273.5431
>
> E lschmid at io.com | io.com
>
>
>
>
> Founded in 2007, IO is a worldwide leader in software defined data center
> technology, services and solutions that enable businesses and governments to
> intelligently control their information.
>
> The communication contained in this e-mail is confidential and is intended
> only for the named recipient(s) and may contain information that is
> privileged, proprietary, attorney work product or exempt from disclosure
> under applicable law. If you have received this message in error, or are not
> the named recipient(s), please note that any form of distribution, copying
> or use of this communication or the information in it is strictly prohibited
> and may be unlawful. Please immediately notify the sender of the error, and
> delete this communication including any attached files from your system.
> Thank you for your cooperation.
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users


Founded in 2007, IO is a worldwide leader in software defined data center technology, services and solutions that enable businesses and governments to intelligently control their information.

The communication contained in this e-mail is confidential and is intended only for the named recipient(s) and may contain information that is privileged, proprietary, attorney work product or exempt from disclosure under applicable law. If you have received this message in error, or are not the named recipient(s), please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited and may be unlawful. Please immediately notify the sender of the error, and delete this communication including any attached files from your system. Thank you for your cooperation.



More information about the Gluster-users mailing list