[Gluster-users] Different file in two bricks, no split-brain detected

[Krutika Dhananjay]

Hi, 

Could you disable quick-read, read-ahead and io-cache and run your test again and share the results: 
#gluster volume set <VOLNAME> performance.quick-read off 
#gluster volume set <VOLNAME> performance.read-ahead off 
#gluster volume set <VOLNAME> performance.io-cache off 

-Krutika 
----- Original Message -----

> From: "Klearchos Chaloulos (Nokia - GR/Athens)"
> <klearchos.chaloulos at nokia.com>
> To: "EXT Krutika Dhananjay" <kdhananj at redhat.com>
> Cc: gluster-users at gluster.org
> Sent: Friday, February 5, 2016 8:30:57 PM
> Subject: RE: [Gluster-users] Different file in two bricks, no split-brain
> detected

> Hello,

> I managed to get logs from two occurrences, please see anonymized logs
> attached.

> Occurrence1: The copied file in both bricks had the correct checksum, but the
> client so an erroneous checksum.

> Occurrence2: The copied file in one brick had the correct checksum and in the
> second brick has an erroneous checksum.

> For details check the notes.txt file in the tarballs.

> Do you have any idea what could be causing this behavior?

> Best regards,

> Klearchos

> From: Chaloulos, Klearchos (Nokia - GR/Athens)
> Sent: Monday, February 01, 2016 10:39 AM
> To: 'EXT Krutika Dhananjay' <kdhananj at redhat.com>
> Cc: gluster-users at gluster.org
> Subject: RE: [Gluster-users] Different file in two bricks, no split-brain
> detected

> Hello,

> Sorry for not replying, but lately the issue cannot be reproduced. If we have
> any new occurrences I’ll collect the logs and send them here.

> Klearchos

> From: EXT Krutika Dhananjay [ mailto:kdhananj at redhat.com ]
> Sent: Wednesday, January 27, 2016 7:12 AM
> To: Chaloulos, Klearchos (Nokia - GR/Athens) < klearchos.chaloulos at nokia.com
> >
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] Different file in two bricks, no split-brain
> detected

> Hi,

> Could you share the following pieces of information:

> 1) output of `gluster volume info <VOL>`

> 2) the client/mount logs

> 3) glustershd logs

> -Krutika

> > From: "Klearchos Chaloulos (Nokia - GR/Athens)" <
> > klearchos.chaloulos at nokia.com >
> 
> > To: gluster-users at gluster.org
> 
> > Sent: Tuesday, January 26, 2016 9:57:38 PM
> 
> > Subject: [Gluster-users] Different file in two bricks, no split-brain
> > detected
> 

> > Description of problem:
> 

> > My setup has 5 gluster volumes, and each of them has 2 bricks as backend.
> 

> > When I copy a large file (100MB) in a gluster volume, 9/10 times it works
> > OK.
> > But about 1 in 10 times the resulting md5 is wrong. After checking I found
> > that the file in one brick has the correct md5sum, while the file in the
> > other brick has a wrong md5sum. The size of the two files is the same.
> 

> > By running "cmp -l <correct_file> <wrong_file>"
> 

> > I found that the difference was in 49 bytes. So the files in the two bricks
> > had the same size, but 49 files were different. Interestingly enough I saw
> > the same number of 49 bytes being different at every check that I made.
> 

> > Do you know what might cause this behavior, has anyone seen something like
> > this before? Is this a bug in glusterfs?
> 

> > Version-Release number of selected component (if applicable):
> 

> > glusterfs 3.7.5 built on Nov 19 2015 16:29:59
> 

> > Repository revision: git://git.gluster.com/glusterfs.git
> 

> > Copyright (c) 2006-2011 Gluster Inc. < http://www.gluster.com >
> 

> > GlusterFS comes with ABSOLUTELY NO WARRANTY.
> 

> > You may redistribute copies of GlusterFS under the terms of the GNU General
> > Public License.
> 

> > How reproducible:
> 

> > Not easy to reproduce, about 1 in 10 times in some environments, not
> > reproducible at all in other environments.
> 

> > Steps to Reproduce:
> 

> > 1. scp <100MB file> <path in gluster volume>
> 

> > Actual results:
> 

> > 1. md5sum of destination should be the same as the source
> 

> > 2. If checksum of files is different between the two bricks, the command
> > "gluster volume heal <vol-name> info split-brain" should return that the
> > two
> > bricks are in split-brain.
> 

> > Expected results:
> 

> > 1. 1 in 10 times the destination file has incorrect checksum. Size is the
> > same, but 49 bytes are altered.
> 

> > 2. "gluster volume heal <vol-name> info split-brain" does not return that
> > the
> > bricks are in split-brain, even though the checksum of the file in the two
> > bricks is different. The size of the file is the same in the two bricks.
> > But
> > 49 bytes are altered.
> 

> > Additional info:
> 

> > _______________________________________________
> 
> > Gluster-users mailing list
> 
> > Gluster-users at gluster.org
> 
> > http://www.gluster.org/mailman/listinfo/gluster-users
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160208/1aa4447f/attachment.html>