[Bugs] [Bug 1414242] [whql][virtio-block+glusterfs]"Disk Stress" and " Disk Verification" job always failed on win7-32/win2012/win2k8R2 guest

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1414242



--- Comment #3 from Raghavendra G <rgowdapp at redhat.com> ---
Description of problem:
"Disk Stress" and "Disk Verification" jobs always failed on win7-32/win2012
guests with "viostor+glusterfs". And job can pass without glusterfs

Version-Release number of selected component (if applicable):
glusterfs-3.4.0.59rhs-1.el7.x86_64
virtio-win-1.6.8-4.el7.noarch
kernel-3.10.0-84.el7.x86_64
qemu-kvm-rhev-1.5.3-45.el7.x86_64
seabios-1.7.2.2-11.el7.x86_64
windows guest:win7-32/win2012

How reproducible:
100%

Steps to Reproduce:
1.boot guest with:
/usr/libexec/qemu-kvm \
-M pc -m 2G -smp 2 \
-cpu Nehalem \
-usb -device usb-tablet \
-drive
file=gluster://10.66.73.33:24007/gv0/win7-32.raw,format=raw,if=none,id=drive-ide0-0-0,werror=stop,rerror=stop,cache=none
\
-device ide-drive,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 \
-netdev tap,id=hostnet0,script=/etc/qemu-ifup1 -device
e1000,netdev=hostnet0,mac=00:52:4f:75:65:35,id=net0 \
-uuid 4870b5fd-65d1-488a-9333-3c4e095285b0 \
-rtc-td-hack -no-kvm-pit-reinjection \
-chardev socket,id=a111,path=/tmp/monitor-win2012-serial,server,nowait -mon
chardev=a111,mode=readline \
-name win7-32-blk-glusterfs \
-drive
file=gluster://10.66.73.33:24007/gv0/win7-32-data.raw,if=none,media=disk,format=raw,rerror=stop,werror=stop,cache=none,aio=native,id=blk-disk1,serial=win8.1-32-blk
-device virtio-blk-pci,drive=blk-disk1,id=disk1 \
-vnc :1 -vga cirrus \
-monitor stdio \
-cdrom /usr/share/virtio-win/virtio-win.iso \
-global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0

2.submit jobs in hck2.1

Actual results:
job always failed

Expected results:
job can pass 

Additional info:
gluster volume info
Volume Name: gv0
Type: Distribute
Volume ID: 67dac294-691b-47e9-8e31-c99be865428f
Status: Started
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 10.66.73.33:/data/gv0/brick1

Analysis:
The HCK failure is caused by a Gluster bug.  The read-ahead and write-behind
xlators are producing incorrect I/O patterns and lead to data corruption.

The workload is generated by the Windows HCK DiskIO.exe tool:
diskio /d "PCI\VEN_1AF4&DEV_1001&SUBSYS_00021AF4&REV_00\3&13C0B0C5&0&20" /b
32KB /t 00:10:00 /c sv /a /o

Block size: 32 KB
IO depth: 32
Pattern: sequential write+read to verify

The error message from DiskIO.exe is:
T8 Error: Data corruption at block offset 1285390336!
T9 Error: Data corruption at block offset 1285488640!
T4 Error: Data corruption at block offset 1285586944!
T7 Error: Data corruption at block offset 1285455872!
T8 Error: Data corruption at block offset 1286537216!

The offsets and number of corruptions can change between runs.

The following SystemTap output for QEMU shows the write+read for the first
corruption at offset 1285390336:
virtio_blk_handle_write req 0x7f9e81543130 sector 2510528 nsectors 64
[...]
bdrv_aio_multiwrite num_callbacks 1 num_reqs 1
virtio_blk_rw_complete req 0x7f9e81543130 ret 0
[...]
virtio_blk_handle_read req 0x7f9e81973330 sector 2510528 nsectors 64
bdrv_aio_readv bs 0x7f9e7fac7740 sector_num 2510528 nb_sectors 64 opaque
0x7f9e81973330
[...]
virtio_blk_rw_complete req 0x7f9e81973330 ret 0

This matches the workload's I/O pattern:
1. A write completes and is followed by a read.
2. sector 2510528 == offset 1285390336 (512-byte sectors)
3. nsectors 64 == 32 KB block size

So far everything seems normal.

The I/O pattern at the Gluster network protocol level should be similar, but
the Wireshark output reveals the client has produced an invalid I/O pattern
that
causes data corruption:
Frame 546, WRITE, 1285324800, 64 KB
Frame 578, READ, 1285292032, 128 KB
Frame 579, READ, 1285292032, 128 KB
Frame 582, WRITE, 1285390336, 32 KB

The WRITE command in Frame 582 was sent *after* the READ command in Frame 579.

There are also other quirks:
1. An identical read request was sent in Frame 578 and 579.  This is a
duplicate I/O and both are pending at the same time!
2. The write request was sent in Frame 582 while the read requests from Frame
578 and 579 are still pending.  In POSIX I/O issuing reads and writes to the
same sector in parallel is undefined!

I disabled various combinations of xlators and discovered that this bug only
happens when read-ahead and write-behind are enabled together.

Gluster team: please see the files I have attached, they should allow you to
debug this in detail.

-- 
You are receiving this mail because:
You are the assignee for the bug.