[Bugs] [Bug 1247221] New: glusterd dies with OOM after a simple find executed on the volume

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1247221

            Bug ID: 1247221
           Summary: glusterd dies with OOM after a simple find executed on
                    the volume
           Product: GlusterFS
           Version: mainline
         Component: glusterd
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: mbienek at ebay.com
                CC: bugs at gluster.org, gluster-bugs at redhat.com



Created attachment 1056643
  --> https://bugzilla.redhat.com/attachment.cgi?id=1056643&action=edit
OOM error,  statedump, some logs from the server

Description of problem:

- When executing a simple 'find . -type f' on a volume with around 600 dirs and
8000 files gluster-server explodes with CPU and memory usage and finally dies
with a OOM. 

9496.724134] Out of memory: Kill process 10376 (glusterfsd) score 565 or
sacrifice child
[ 9496.725518] Killed process 10376 (glusterfsd) total-vm:25838340kB,
anon-rss:1737572kB, file-rss:0kB


Version of GlusterFS package installed:
glusterfs-server_3.7.2-11437551431_amd64 

on Ubuntu Trusty 14.04.2:  
3.13.0-58-generic #97-Ubuntu SMP Wed Jul 8 02:56:15 UTC 2015 x86_64 x86_64
x86_64 GNU/Linux


GlusterFS Cluster Information:
- Number of volumes: 10
- Volume on which the particular issue is seen: 1 
- Type of volumes: Replicated
- Output of gluster volume info
Volume Name: ebayk_kftp
ype: Replicate
  Volume ID: 11c2ee66-a186-4136-b577-f23c9c34c500
  Status: Started
  Number of Bricks: 1 x 3 = 3
  Transport-type: tcp
  Bricks:
  Brick1: glustercg47-1:/data/ebayk_kftp
  Brick2: glustercg47-2:/data/ebayk_kftp
  Brick3: glustercg47-3:/data/ebayk_kftp
  Options Reconfigured:
  nfs.disable: On
  features.quota-deem-statfs: on
  features.inode-quota: on
  features.quota: on
  auth.allow: 10.38.*,10.46.*,10.47.*
  performance.readdir-ahead: on

Output of gluster volume status
Attached

Get the statedump of the volume with the problem
Attached 


Client Information: 
- OS Type: Debian
- Mount type:  glusterfs _netdev,defaults        0       0
- OS Version:  Wheezy 7.8 


Version-Release number of selected component (if applicable):
glusterfs-server_3.7.2-11437551431_amd64.deb on Ubuntu Trusty 14.04.2:
3.13.0-58-generic #97-Ubuntu SMP Wed Jul 8 02:56:15 UTC 2015 x86_64 x86_64
x86_64 GNU/Linux

How reproducible:

Steps to Reproduce:
1. Start the volume 
2. Run 'find . -type f' 
3. After some time 1st gluster node will die because OOM 
4. volume will not go online 

Actual results:
Dies because OOM 

Expected results:


Additional info:

There are 3 gluster nodes running on two esx hosts with SSD disks as a storage
pool. 
The problem happens when there is only 1 CPU and 1GB of RAM configured for
every VM but it also happens when there are 8CPU's an 16 - 32GB of RAM
configured.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.