[Gluster-users] Fwd: Self-heal and high load

[Darren]

Hi,

Thanks for the reply, I'll try and answer those questions:

The only increase in load comes from CPU usage. There is no noticeable
increase in network traffic, memory usage doesn't change and dik usage
doesn't increase noticeably.
The test involves creating random files which vary in size but are never
more than 1MB. There's about 100 of them created. Increasing or decreasing
this doesn't seem to have any effect on the self heal process.
I believe the ping timeout is set so low as when a file is written by a
client it needs to be immediately accessible in both data centres by
different clients. I've re-tested using the default timeout value and when
the test is running it hangs once the iptables rules are enforced. This
means the the client isn't writing any files, which means other clients
wouldn't be able to pick up the files that had just been written. It also
means that as no data is written to the brick on either server, when
forcing the self heal daemon to run there are no differences, so it doesn't
do anything and so there's no CPU load increase.

I'm not sure if this would work in our production environment, so may well
have to get our QA guys help me out with more appropriate test of this...

It may well be that we're using Gluster in way that it was never designed
for, in which case we'll have to look at something more suited to this
setup.

Thanks,
Darren


On 14 May 2013 12:12, Vijay Bellur <vbellur at redhat.com> wrote:

> On 05/13/2013 09:13 PM, Darren wrote:
>
>> Hi all,
>>
>> I'm pretty new to Gluster, and the company I work for uses it for
>> storage across 2 data centres. An issue has cropped up fairly recently
>> with regards to the self-heal mechanism.
>>
>> Occasionally the connection between these 2 Gluster servers breaks or
>> drops momentarily. Due to the nature of the business it's highly likely
>> that files have been written during this time. When the self-heal daemon
>> runs it notices a discrepancy and gets the volume up to date. The
>> problem we've been seeing is that this appears to cause the CPU load to
>> increase massively on both servers whilst the healing process takes place.
>>
>> After trying to find out if there were any persistent network issues I
>> tried recreating this on a test system and can now re-produce at will.
>> Our test system set up is made up of 3 VMs, 2 Gluster servers and a
>> client. The process to cause this was:
>> Add in an iptables rule to block one of the Gluster servers from being
>> reached by the other server and the client.
>> Create some random files on the client.
>>
>
> Can you describe the number and sizes of files created as part of this
> step?
>
>
>
>  Flush the iptables rules out so the server is reachable again.
>> Force a self heal to run.
>> Watch as the load on the Gluster servers goes bananas.
>>
>
> How high does your load get to? Is it just the CPU or do you see other
> resources like memory, network being consumed to a greater degree as well?
>
>
>
>> The problem with this is that whilst the self-heal happens one the
>> gluster servers will be inaccessible from the client, meaning no files
>> can be read or written, causing problems for our users.
>>
>> I've been searching for a solution, or at least someone else who has
>> been having the same problem and not found anything. I don't know if
>> this is a bug or config issue (see below for config details). I've tried
>> a variety of different options but none of them have had any effect.
>>
>> Our production set up is as follows:
>> 2 Gluster servers (1 in each DC) replicating to each other
>> We then have multiple other servers that store and retrieve files on
>> Gluster using a local glusterfs mount point.
>> Only 1 data centre is active at any one time
>> The Gluster servers are VMs on a Xen hypervisor.
>> All our systems are CentOS 5
>> Gluster 3.3.1 (I've also tried 3.3.2)
>>
>> gluster02 ~ gluster volume info rmfs
>>
>> Volume Name: volume1
>> Type: Replicate
>> Volume ID: 3fef44e1-e840-452e-b16b-**a9fc698e7dfd
>> Status: Started
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster01:/mnt/store1
>> Brick2: gluster02:/mnt/store1
>> Options Reconfigured:
>> nfs.disable: off
>> auth.allow: 172.30.98.*
>> network.ping-timeout: 5
>>
>
> Setting network.ping-timeout to 5 is generally not recommended. As a
> matter of fact, it would not be advisable to alter the ping timeout from
> the default value.
>
> Regards,
> Vijay
>
>
>> Any help or suggestions would be greatly appreciated. If you need
>> anything else from me, just ask.
>>
>> Thanks,
>>
>> Darren
>>
>>
>>
>> ______________________________**_________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.**org/mailman/listinfo/gluster-**users<http://supercolony.gluster.org/mailman/listinfo/gluster-users>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130515/f57a1a3e/attachment.html>