[Bugs] [Bug 1503079] New: Running fio on vms results in fsync, error=Input/ output error

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1503079

            Bug ID: 1503079
           Summary: Running fio on vms results in fsync,
                    error=Input/output error
           Product: GlusterFS
           Version: mainline
         Component: sharding
          Severity: high
          Priority: high
          Assignee: bugs at gluster.org
          Reporter: kdhananj at redhat.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org, kdhananj at redhat.com,
                    knarra at redhat.com, rhs-bugs at redhat.com,
                    sabose at redhat.com, sasundar at redhat.com,
                    storage-qa-internal at redhat.com, vbellur at redhat.com
            Blocks: 1501309



+++ This bug was initially created as a clone of Bug #1501309 +++

+++ This bug was initially created as a clone of Bug #1497156 +++

Description of problem:
While running fio workload on the vms created in RHHI setup i see that fio job
gives the below error in json.out file.

json.out file:
=======================
ostname=dhcp35-71.lab.eng.blr.redhat.com, be=0, 64-bit, os=Linux, arch=x86-64,
fio=fio-2.1.7, flags=1
hostname=dhcp35-168.lab.eng.blr.redhat.com, be=0, 64-bit, os=Linux,
arch=x86-64, fio=fio-2.1.7, flags=1
hostname=dhcp35-141.lab.eng.blr.redhat.com, be=0, 64-bit, os=Linux,
arch=x86-64, fio=fio-2.1.7, flags=1
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat,
error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat,
error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat,
error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat,
error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat,
error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> file:filesetup.c:270, func=fstat,
error=Input/output error
<dhcp35-168.lab.eng.blr.redhat.com> {
<dhcp35-168.lab.eng.blr.redhat.com>   "fio version" : "fio-2.1.7",
<dhcp35-168.lab.eng.blr.redhat.com>   "jobs" : [
<dhcp35-168.lab.eng.blr.redhat.com>
<dhcp35-168.lab.eng.blr.redhat.com>   ]
<dhcp35-168.lab.eng.blr.redhat.com> }

Run status group 0 (all jobs):
client <10.70.35.168>: exited with error 1
<dhcp35-71.lab.eng.blr.redhat.com> fio: pid=0, err=5/file:filesetup.c:174,
func=fsync, error=Input/output error
<dhcp35-71.lab.eng.blr.redhat.com> {
<dhcp35-71.lab.eng.blr.redhat.com>   "fio version" : "fio-2.1.7",
<dhcp35-71.lab.eng.blr.redhat.com>   "jobs" : [
<dhcp35-71.lab.eng.blr.redhat.com>
<dhcp35-71.lab.eng.blr.redhat.com>   ]
<dhcp35-71.lab.eng.blr.redhat.com> }

Run status group 0 (all jobs):
client <10.70.35.71>: exited with error 1
<dhcp35-141.lab.eng.blr.redhat.com> fio: pid=0, err=5/file:filesetup.c:174,
func=fsync, error=Input/output error
<dhcp35-141.lab.eng.blr.redhat.com> {
<dhcp35-141.lab.eng.blr.redhat.com>   "fio version" : "fio-2.1.7",
<dhcp35-141.lab.eng.blr.redhat.com>   "jobs" : [
<dhcp35-141.lab.eng.blr.redhat.com>
<dhcp35-141.lab.eng.blr.redhat.com>   ]
<dhcp35-141.lab.eng.blr.redhat.com> }

Run status group 0 (all jobs):
client <10.70.35.141>: exited with error 1
{
  "fio version" : "fio-2.1.7",
  "client_stats" : [
    {
      "jobname" : "workload",
      "groupid" : 0,

Attaching the error which is see on the vm console during fio run.

Version-Release number of selected component (if applicable):
glusterfs-3.8.4-45.el7rhgs.x86_64

How reproducible:
Always

Steps to Reproduce:
1. Create RHHI setup with latest glusterfs & RHV bits
2. Create 3 app vms.
3. start running fio.

Actual results:
Running fio results in fsync, error=input/output error.

Expected results:
fsync should run successfully.

Additional info:

screenshot for the vm console when fio runs is captured .
attaching json.out file 
attaching all the sos reports.

--- Additional comment from Sahina Bose on 2017-10-11 07:25:31 EDT ---

Are using gfapi access for VM disks?

--- Additional comment from RamaKasturi on 2017-10-11 10:52:22 EDT ---

sahina, i was seeing this issue with fuse access.


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1501309
[Bug 1501309] Running fio on vms results in fsync, error=Input/output error
-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.