[Bugs] [Bug 1428673] possible repeatedly recursive healing of same file with background heal not happening when IO is going on

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1428673



--- Comment #5 from Ashish Pandey <aspandey at redhat.com> ---

After applying https://review.gluster.org/16985 

1 - Steps to see data corruption

- Create 2+1 ec volume and mount it on /mnt/vol (brick{0,1,2})
- cd /mnt/vol
- start writing data on a file using "dd if=/dev/urandom of=file
conv=fdatasync"
- kill brick0
- Start brick0 using command line. gluster v start force was not recreating
this    
  issue very often.
- keep observing output of "gluster v heal volname info"
- As soon as heal  info shows 0 entries for all the bricks, stop IO on mount
point (ctrl + C dd)

>> Check md5sum of the file by following steps - 
- kill brick0
- unmount /mnt/vol
- mount volume again
- md5sum file  <<<<

- start brick0 using command line

- kill brick1
- unmount /mnt/vol
- mount volume again
- md5sum file  <<<<

- start brick1 using command line

- kill brick2
- unmount /mnt/vol
- mount volume again
- md5sum file  <<<<

md5sum would be different in above outputs. I have observed almost 6 out of 10
times.


I have also observed one issue related to heal info - 


Dirty bit remains more than 1 on bricks even after healing size and version
completely.( disable self heal)

This is visible with my patch but I think the problem lies somewhere else. 
If we disable server side heal and heal starts due to client side access of
file, heal happens, but dirty bit does not become 1 or 0.
if IO is going on that file. The result of which is that heal info will keep
showing this file as heal required. 

Note, now heal info also checks that if dirty bit is more than one then
something is wrong with file even if all version and size are same.

----------

steps 

- gluster v heal vol disable
- Just try to create issue of data corruption by following above steps
-  getxattr -m. -d -e hex <all brick paths> 
- we can see that after some time version and size would be same . dirty would
be same but would be more than 1.
- gluster v heal info will shows 1 entry per brick

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=fCiUYAf6TI&a=cc_unsubscribe