[Bugs] [Bug 1388292] New: performance.read-ahead on results in processes on client stuck in IO wait

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1388292

            Bug ID: 1388292
           Summary: performance.read-ahead on results in processes on
                    client stuck in IO wait
           Product: GlusterFS
           Version: mainline
         Component: read-ahead
          Severity: medium
          Assignee: bugs at gluster.org
          Reporter: bugzilla at slowb.ro
                CC: bugs at gluster.org, ravishankar at redhat.com,
                    rgowdapp at redhat.com



Created attachment 1213677
  --> https://bugzilla.redhat.com/attachment.cgi?id=1213677&action=edit
Gluster State Dumps 2 minutes apart

# Description of problem:
After updating to v3.7.16 our Replica volume (2 clients, and 2 bricks)
developed issues where processes (PHP, clamav/maldet, and occasionally even the
gluster mount process) on the clients (over FUSE) would enter into an
"un-interruptible kernel wait state"

Apparently the cause of this was "performance.read-ahead"
After turning it to "off" we have not had any other problems for over 1 week. 

# Version-Release number of selected component (if applicable):
v3.7.16

# How reproducible:
It was always reproducible but took a long time (24+hours) before a anything
occurred after updating (from v3.7.13) to mainline.

# Steps to Reproduce:
(Full disclosure I have not replicated with a brand new volume, I don't have
the hardware to test currently)

Notes on Reproduction: We have a total of just under 2.8million files on the 1
volume. I am not sure if you will need to create them all to reproduce the
issue we are having. 

1. Create a Replica volume with 2 bricks
2. gluster volume set volume1 performance.read-ahead on
3. Mount gluster volume via FUSE on client.
4. Create a wordpress site, and set it up.
5. run clamav or maldet  on the files.
5.1 Or if installed via wordpress run `php wp-cron.php`
6. After a while (we saw a range of time from 2-24hours) specific files will
trigger a process to get stuck, and then all following transactions on those
files will fail. (such as rm, or moving the file off the volume)

# Actual results:
Processes running on the clients become stuck and the gluster mount.
lsof would hang when trying to find the open file descriptors, as would trying
to cat /proc/${pid}/cmdline on an un-interruptible process. 

# Expected results:
Processes do not become stuck in an iowait/uninterruptible state. 

# Additional info:

Source of Packages Used, http://ppa.launchpad.net/gluster/glusterfs-3.7/ubuntu 

I have also anonymised the path data of the attached statedumps. But i have
kept the length the same. eg: 'welove.site.com' becomes 'aqwedsaqwedsaqw'. I
hope that is okay.

## Volume Information:

sudo gluster volume info

Volume Name: gv-www
Type: Replicate
Volume ID: 6163f185-d58b-45f2-8bf0-eeb3a18142d9
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: demo-gluster01:/gluster
Brick2: demo-gluster02:/gluster
Options Reconfigured:
performance.read-ahead: off
nfs.disable: true
cluster.self-heal-daemon: enable
features.scrub: Active
features.bitrot: on
diagnostics.client-log-level: INFO
diagnostics.brick-log-level: INFO
performance.readdir-ahead: on

## Brick Information:

### Brick 1:

sudo getfattr -d -m. -ehex /gluster/
getfattr: Removing leading '/' from absolute path names
# file: gluster/
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.gv-www-client-1=0x000000000000000000000000
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0x6163f185d58b45f28bf0eeb3a18142d9

### Brick 2:
sudo getfattr -d -m. -ehex /gluster/
getfattr: Removing leading '/' from absolute path names
# file: gluster/
trusted.gfid=0x00000000000000000000000000000001
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
trusted.glusterfs.volume-id=0x6163f185d58b45f28bf0eeb3a18142d9


## Client Information:
All servers are Ubuntu 14.04.5 LTS

### Client 1:
df -Th
Filesystem             Type            Size  Used Avail Use% Mounted on

demo-gluster01:/gv-www fuse.glusterfs  197G  176G   12G  94% /var/www

Mount in /etc/fstab
demo-gluster01:/gv-www /var/www/ glusterfs defaults,_netdev 0 0 


### Client 2:
df -Th
Filesystem             Type            Size  Used Avail Use% Mounted on
demo-gluster02:/gv-www fuse.glusterfs  197G  176G   12G  94% /var/www

Mount in /etc/fstab
demo-gluster02:/gv-www /var/www/ glusterfs defaults,_netdev 0 0

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.