[Gluster-users] Excessive file descriptor usage

[Pranith Kumar Karampuri]

hi Joel,

I fixed some fd-leaks in open-behind xlator after 3.4 release.

Could you do "gluster volume set <volname> performance.open-behind off" and see if open-fd-count still increases periodically.

Commit information:

commit 8c1304b03542eefbbff82014827fc782c3c3584f
Author: Pranith Kumar K <pkarampu at redhat.com>
Date:   Sat Aug 3 08:27:27 2013 +0530

    performance/open-behind: Fix fd-leaks in unlink, rename
    
    Change-Id: Ia8d4bed7ccd316a83c397b53b9c1b1806024f83e
    BUG: 991622
    Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
    Reviewed-on: http://review.gluster.org/5493
    Tested-by: Gluster Build System <jenkins at build.gluster.com>
    Reviewed-by: Anand Avati <avati at redhat.com>

Pranith

----- Original Message -----
> From: "Joel Stalder" <jstalder at panterranetworks.com>
> To: gluster-users at gluster.org
> Sent: Monday, October 28, 2013 11:06:22 PM
> Subject: [Gluster-users] Excessive file descriptor usage
> 
> 
> 
> 
> 
> Folks,
> 
> 
> 
> I’m using Gluster 3.4 on CentOS 6… very simple two-server, two-brick (replica
> 2) setup. The volume itself has many small files across a reasonably large
> directory tree, though I’m not sure if that plays a role. The FUSE client is
> being used.
> 
> 
> 
> I can live with the performance limitations of small files with Gluster, but
> the problem I’m having is that file descriptor usage on the glusterfs
> servers just continues to grow… not sure when it might actually top off, if
> ever. No rebalance has been or is running. The application running on the
> client servers (two) are not leaving the files open.
> 
> 
> 
> I’ve tuned Linux behavior on the glusterfs servers, via /proc, to accept over
> 1 million per-process file descriptors, but that doesn’t seem to be enough.
> This volume hit the FD max some time ago and had to be recovered… I thought
> it was a fluke so started watching the open FD count and see that it’s
> growing again.
> 
> 
> 
> # gluster volume top users open
> 
> Brick: node-75:/storage/users
> 
> Current open fds: 765651, Max open fds: 1048558, Max openfd time: 2013-10-02
> 22:26:18.327010
> 
> 
> 
> Brick: node-76:/storage/users
> 
> Current open fds: 768936, Max open fds: 768938, Max openfd time: 2013-10-28
> 17:11:04.184964
> 
> 
> 
> Clients:
> 
> 
> 
> # cat /proc/sys/fs/file-nr
> 
> 5100 0 1572870
> 
> 
> 
> # cat /proc/sys/fs/file-nr
> 
> 2550 0 1572870
> 
> 
> 
> 
> 
> Looking for thoughts or suggestions here. Anyone else encountered this? Is
> the recommended solution to just define a ridiculously high per-process and
> global file descriptor max?
> 
> 
> 
> -Joel
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users