[Gluster-users] File Corruption when adding bricks to live replica volumes

[Krutika Dhananjay]

Hi, 

So I looked at the glustershd logs and there are no log messages that indicate that there was a reverse heal. 
For instance, see this: 
[2016-01-22 06:13:50.268068] I [MSGID: 108026] [afr-self-heal-common.c:651:afr_log_selfheal] 0-datastore1-replicate-0: Completed data selfheal on 5a3dcdd1-8866-47a5-8432-7b625a2806c3. source=0 sinks=2 

The above implies that the sink was "2" (index of the sink brick - vng - counting from 0), and the source was "0" (vnb in this case), which means the heal happened into the correct brick (last brick). 

Could you do the following: 
1) Disable client-side healing: 
For this, you need to execute 
# gluster volume set datastore1 entry-self-heal off 
# gluster volume set datastore1 data-self-heal off 
# gluster volume set datastore1 metadata-self-heal off 

2) Run your add-brick/remove-brick tests again (you know it best). 
3) Share the resultant glustershd.log from all three machines. 

-Krutika 

----- Original Message -----

> From: "Lindsay Mathieson" <lindsay.mathieson at gmail.com>
> To: "Krutika Dhananjay" <kdhananj at redhat.com>, "gluster-users"
> <gluster-users at gluster.org>
> Sent: Friday, January 22, 2016 11:55:27 AM
> Subject: Re: [Gluster-users] File Corruption when adding bricks to live
> replica volumes

> On 21/01/16 20:03, Krutika Dhananjay wrote:

> > Also could you please attach glustershd.log files and the client logs?
> 

> Ok, I ran the procedure again just to be sure. This time I got 81 shards
> being healed from the other bricks, but still got 2 bricks being "reverse
> healed" from the new brick. A image check on the vm file failed with:

> > ERROR: I/O error in check_refcounts_l2
> 
> > qemu-img: Check failed: Input/output error
> 

> removing the brick resolved the problems.

> glustershd.log from node vng attached (the new brick node), the fuse client
> log was empty.

> thanks,
> --
> Lindsay Mathieson
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160122/49e9e385/attachment.html>