[Gluster-users] Fwd: disperse heal speed up

Serkan Çoban cobanserkan at gmail.com
Fri Aug 5 15:07:40 UTC 2016

Hi again,

I am seeing the above situation in production environment now.
One disk on one of my servers broken. I killed the brick process,
replace the disk, mount it and then I do a gluster v start force.

For a 24 hours period  after replacing disks I see below gluster v
heal info count increased until 200.000

gluster v heal v0 info | grep "Number of entries" | grep -v "Number of
entries: 0"
Number of entries: 205117
Number of entries: 205231

For about 72 hours It decreased to 40K, and it is going very slowly right now.
What I am observing is very very slow heal speed. There is no errors
in brick logs.
There was 900GB data in broken disk and now I see 200GB healed after
96 hours after replacing disk.
There are below warnings in glustershd.log but I think they are harmless.

W [ec_combine.c:866:ec_combine_check] 0-v0-disperse-56: Mismatching
xdata in answers of LOOKUP
W [ec_common.c:116:ec_check_status] 0-v0-disperse-56: Operation failed
on some subvolumes (up=FFFFF, mask=FFFFF, remaining=0, good=FFFF7,
W [ec_common.c:71:ec_heal_report] 0-v0-disperse-56: Heal failed
[invalid argument]

I tried turning on performance.client-io-threads but it did not
changed anything.
For 900GB data It will take nearly 8 days to heal. What can I do?


On Fri, Apr 15, 2016 at 1:28 PM, Serkan Çoban <cobanserkan at gmail.com> wrote:
> 100TB is newly created files when brick is down.I rethink the
> situation and realized that I reformatted all the bricks in case 1 so
> write speed limit is 26*100MB/disk
> In case 2 I just reformatted one brick so write speed limited to
> 100MB/disk...I will repeat the tests using one brick in both cases
> once with reformat, and once with just killing brick process...
> Thanks for reply..
> On Fri, Apr 15, 2016 at 9:27 AM, Xavier Hernandez <xhernandez at datalab.es> wrote:
>> Hi Serkan,
>> sorry for the delay, I'm a bit busy lately.
>> On 13/04/16 13:59, Serkan Çoban wrote:
>>> Hi Xavier,
>>> Can you help me about the below issue? How can I increase the disperse
>>> heal speed?
>> It seems weird. Is there any related message in the logs ?
>> In this particular test, are the 100TB modified files or newly created files
>> while the brick was down ?
>> How many files have been modified ?
>>> Also I would be grateful if you have detailed documentation about disperse
>>> heal,
>>> why heal happens on disperse volume, how it is triggered? Which nodes
>>> participate in heal process? Any client interaction?
>> Heal process is basically the same used for replicate. There are two ways to
>> trigger a self-heal:
>> * when an inconsistency is detected, the client initiates a background
>> self-heal of the inode
>> * the self-heal daemon scans the lists of modified files created by the
>> index xlator when a modification is made while some node is down. All these
>> files are self-healed.
>> Xavi
>>> Serkan
>>> ---------- Forwarded message ----------
>>> From: Serkan Çoban <cobanserkan at gmail.com>
>>> Date: Fri, Apr 8, 2016 at 5:46 PM
>>> Subject: disperse heal speed up
>>> To: Gluster Users <gluster-users at gluster.org>
>>> Hi,
>>> I am testing heal speed of disperse volume and what I see is 5-10MB/s per
>>> node.
>>> I increased disperse.background-heals to 32 and
>>> disperse.heal-wait-qlength to 256, but still no difference.
>>> One thing I noticed is that, when I kill a brick process, reformat it
>>> and restart it heal speed is nearly 20x (200MB/s/node)
>>> But when I kill the brick, then write 100TB data, and start brick
>>> afterwords heal is slow (5-10MB/s/node)
>>> What is the difference between two scenarios? Why one heal is slow and
>>> other is fast? How can I increase disperse heal speed? Should I
>>> increase thread count to 128 or 256? I am on 78x(16+4) disperse volume
>>> and my servers are pretty strong (2x14 cores with 512GB ram, each node
>>> has 26x8TB disks)
>>> Gluster version is 3.7.10.
>>> Thanks,
>>> Serkan

More information about the Gluster-users mailing list