[Bugs] [Bug 1476205] [EC]: md5sum mismatches every time for a file from the fuse client on EC volume
bugzilla at redhat.com
bugzilla at redhat.com
Fri Jul 28 09:43:54 UTC 2017
https://bugzilla.redhat.com/show_bug.cgi?id=1476205
Ashish Pandey <aspandey at redhat.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Status|NEW |ASSIGNED
--- Comment #2 from Ashish Pandey <aspandey at redhat.com> ---
RCA by Pranith -
Some notes, didn't find the RCA yet.
Only the last fragment of the file is corrupted. On closer inspection, it is
found that the file on 5th brick is the bad-one. It is as if the write didn't
happen on the 5th brick but only happened on the rest of the bricks. So the
next step is to find why was the write got skipped.
Rahul,
For more debugging we need /etc/services file from the machine where you
ran crefi. Could you attach it to the bz?
Pranith
--- Additional comment from Pranith Kumar K on 2017-07-27 21:51:51 EDT ---
(In reply to Pranith Kumar K from comment #10)
> Some notes, didn't find the RCA yet.
>
> Only the last fragment of the file is corrupted. On closer inspection, it is
> found that the file on 5th brick is the bad-one. It is as if the write
> didn't happen on the 5th brick but only happened on the rest of the bricks.
> So the next step is to find why was the write got skipped.
>
> Rahul,
> For more debugging we need /etc/services file from the machine where you
> ran crefi. Could you attach it to the bz?
>
> Pranith
I found it in one of the rhgs machines and the md5sum is matching with what we
have in the bz. I will use that
--- Additional comment from Pranith Kumar K on 2017-07-27 22:12:20 EDT ---
Some notes, still no RC
07:24:01 :) ⚡ md5sum /home/gfs/ec_?/5*
ecb9f6f29ebbb72e3e612b0aec775b36 /home/gfs/ec_0/596c62e1%%HPJLBVY86T
6fe5b21f27a5c9dc72ae3422102327b2 /home/gfs/ec_1/596c62e1%%HPJLBVY86T
6f82c16ed99b901929737ddf11dd920b /home/gfs/ec_2/596c62e1%%HPJLBVY86T
bcac001cf1cac8860e4ae233a35cd7ef /home/gfs/ec_3/596c62e1%%HPJLBVY86T
6921a33bd9c1083aaec582cde9973b61 /home/gfs/ec_4/596c62e1%%HPJLBVY86T <<<----
this is the bad fragment
b9d70081b5b41c9f980e20bf40e1fd13 /home/gfs/ec_5/596c62e1%%HPJLBVY86T
I did the following to see what fragment corresponds to the bad md5sum that is
present in the file we got.
07:24:53 :) ⚡ for i in {12288..14336}; do Crefi/crefi.py --multi -n 1 -b 1 -d 1
--size $i -t text --fop=create /mnt/ec2; done
root at dhcp35-190 - ~
07:26:49 :) ⚡ md5sum /home/gfs/ec_4/level00/* | grep
6921a33bd9c1083aaec582cde9973b61
6921a33bd9c1083aaec582cde9973b61 /home/gfs/ec_4/level00/597a99d1%%4SKACGWCDK
root at dhcp35-190 - ~
07:27:07 :) ⚡ ls -l /mnt/ec2/level00/597a99d1%%4SKACGWCDK
-rw-r--r--. 1 root root 14336 Jul 28 07:26
/mnt/ec2/level00/597a99d1%%4SKACGWCDK
07:39:48 :) ⚡ truncate -s 13526 /mnt/ec2/level00/597a99d1%%4SKACGWCDK
root at dhcp35-190 - ~
07:40:10 :) ⚡ md5sum /home/gfs/ec_?/level00/597a99d1%%4SKACGWCDK
ecb9f6f29ebbb72e3e612b0aec775b36 /home/gfs/ec_0/level00/597a99d1%%4SKACGWCDK
6fe5b21f27a5c9dc72ae3422102327b2 /home/gfs/ec_1/level00/597a99d1%%4SKACGWCDK
6f82c16ed99b901929737ddf11dd920b /home/gfs/ec_2/level00/597a99d1%%4SKACGWCDK
bcac001cf1cac8860e4ae233a35cd7ef /home/gfs/ec_3/level00/597a99d1%%4SKACGWCDK
f039f8c213c0b31b2d16da87de8da6b7
/home/gfs/ec_4/level00/597a99d1%%4SKACGWCDK<------ This fragment is correct.
b9d70081b5b41c9f980e20bf40e1fd13 /home/gfs/ec_5/level00/597a99d1%%4SKACGWCDK
So This means either the write was never wound on the file when this bug
happened or it failed silently.
This will be the next things to look into.
--- Additional comment from Pranith Kumar K on 2017-07-28 01:27:26 EDT ---
Ashish,
Didn't find any evidence for how it failed, but there is a bug in EC. When
writev in truncate fails, then it is not marking that brick as bad. Verified it
in gdb. We may need more help from Rahul to find how write exactly failed. Are
the quota errors contributing to any failures is also something we may need to
check.
I put one of the brick processes in gdb and made it return error without
performing any write:
Thread 7 "glusterfsd" hit Breakpoint 3, server3_3_writev (req=0x7fbf100ca140)
at server-rpc-fops.c:4100
4100 server_state_t *state = NULL;
(gdb) n
...
(gdb)
4119 frame = get_frame_from_request (req);
(gdb)
4120 if (!frame) {
(gdb) p frame=0
$3 = (call_frame_t *) 0x0
(gdb) n
4122 SERVER_REQ_SET_ERROR (req, ret);
(gdb)
4123 goto out;
(gdb)
4172 free (args.xdata.xdata_val);
(gdb)
4174 if (op_errno)
(gdb) c
I got the same scenario mentioned in this bug:
root at dhcp35-190 - ~
10:34:11 :) ⚡ getfattr -d -m. -e hex /home/gfs/ec_?/services
getfattr: Removing leading '/' from absolute path names
# file: home/gfs/ec_0/services
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.ec.size=0x00000000000034d6
trusted.ec.version=0x00000000000000270000000000000027
trusted.gfid=0xeacb572971f5457a9a5706020255f312
# file: home/gfs/ec_1/services
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.ec.size=0x00000000000034d6
trusted.ec.version=0x00000000000000270000000000000027
trusted.gfid=0xeacb572971f5457a9a5706020255f312
# file: home/gfs/ec_2/services
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.ec.size=0x00000000000034d6
trusted.ec.version=0x00000000000000270000000000000027
trusted.gfid=0xeacb572971f5457a9a5706020255f312
# file: home/gfs/ec_3/services
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.ec.size=0x00000000000034d6
trusted.ec.version=0x00000000000000270000000000000027
trusted.gfid=0xeacb572971f5457a9a5706020255f312
# file: home/gfs/ec_4/services
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.ec.size=0x00000000000034d6
trusted.ec.version=0x00000000000000270000000000000027
trusted.gfid=0xeacb572971f5457a9a5706020255f312
# file: home/gfs/ec_5/services
security.selinux=0x756e636f6e66696e65645f753a6f626a6563745f723a686f6d655f726f6f745f743a733000
trusted.ec.config=0x0000080602000200
trusted.ec.dirty=0x00000000000000000000000000000000
trusted.ec.size=0x00000000000034d6
trusted.ec.version=0x00000000000000270000000000000027
trusted.gfid=0xeacb572971f5457a9a5706020255f312
root at dhcp35-190 - ~
10:34:13 :) ⚡ md5sum /home/gfs/ec_?/services
ecb9f6f29ebbb72e3e612b0aec775b36 /home/gfs/ec_0/services
6fe5b21f27a5c9dc72ae3422102327b2 /home/gfs/ec_1/services
6f82c16ed99b901929737ddf11dd920b /home/gfs/ec_2/services
bcac001cf1cac8860e4ae233a35cd7ef /home/gfs/ec_3/services
6921a33bd9c1083aaec582cde9973b61 /home/gfs/ec_4/services <<<<------ same
md5sum as the bad file in Rahul's setup and all xattrs match as if everything
is fine when it is not.
b9d70081b5b41c9f980e20bf40e1fd13 /home/gfs/ec_5/services
Ashish,
Can you send out a patch? In both ec_truncate_open_cbk,
ec_truncate_writev's cbk, we need to update lock->good_mask correctly. I don't
find any other way for the file to end up in the state it did on Rahul's setup.
We just need to confirm this is indeed the case. The only other psychotic case
will be if the brick sends a success without performing the write.
Rahul,
Will it be possible to get a tcpdump of a run where this happens? That will
confirm how this bug happened. The bug we found based on the information you
had on your setup I explained above is a day-1 bug.
Pranith
--
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list