[Bugs] [Bug 1476205] [EC]: md5sum mismatches every time for a file from the fuse client on EC volume

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1476205



--- Comment #1 from Ashish Pandey <aspandey at redhat.com> ---
Description of problem:
-----------------------

some files are giving different md5sum when executed from different mounts on
the same volume.
Example:
:/master on /mnt/glusterfs type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
:/master on /mnt/master type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
:/slave on /mnt/slave type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
[root at fan ~]# md5sum
/mnt/glusterfs/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
9c0de9e06e88ddba88248e9c8e502f82 
/mnt/glusterfs/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
[root at fan ~]# ls -l
/mnt/glusterfs/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
-rwxrwxr--. 3 42023 13883 13526 Jul 18 05:29
/mnt/glusterfs/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
[root at fan ~]# ls -l
/mnt/master/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
-rwxrwxr--. 3 42023 13883 13526 Jul 18 05:29
/mnt/master/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
[root at fan ~]# md5sum
/mnt/master/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
16f92bf8f96094a214b1d0293c1accda 
/mnt/master/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T

Further Analysis by Ashish shows the following:

Looks like files are corrupted at backend, to confirm mounted the volume on 4
mount points and got the different md5sum. This could be because all mount
points are reading file from different sets of 4 bricks. To confirm this,
killed 2 bricks from each subvolume and again calculated md5sum. This time all
md5sums are same.



[root at fan tmp]# md5sum
/mnt/glusterfs/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
0bc19eac0693355a2e954ba3d66d4ef1 
/mnt/glusterfs/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T

[root at fan tmp]# md5sum
/mnt/master/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
0bc19eac0693355a2e954ba3d66d4ef1 
/mnt/master/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T

[root at fan tmp]# md5sum
/mnt/temp_master/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
0bc19eac0693355a2e954ba3d66d4ef1 
/mnt/temp_master/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T

[root at fan tmp]# md5sum
/mnt/temp-2/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T
0bc19eac0693355a2e954ba3d66d4ef1 
/mnt/temp-2/thread0/level01/level11/level21/level31/level41/level51/level61/level71/596c62e1%%HPJLBVY86T


Steps carried:
==============

The steps carried were in geo-rep setup, however the checksum missmatches on
master. Providing only the steps that is needed to avoid confusion. 

1. Create 6 node Master Cluster
2. Create EC volume on Master (2x(4+2))
3. Mount the volume (Fuse)
4. Using crefi, create the data set. fops carried inorder were:

create, chmod, chown, chgrp, hardlink, symlink, truncate

5. After every fop, calculate the checksum of mount (At least twice)
6. After truncate, checksum never matched

Actual results:
===============

Checksum mismatches for a file (Might be data corruption) every time calculated
from different mount for the same volume. 


Expected results:
=================

Checksum should match all the time from all the different mounts.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.