[Gluster-users] gluster opens thousands of files on one peer

[Tomasz Chmielewski]

On 09.07.2011 09:41, Anand Avati wrote:
> 
> 
> On Wed, Jul 6, 2011 at 6:58 AM, Tomasz Chmielewski <mangoo at wpkg.org 
> <mailto:mangoo at wpkg.org>> wrote:
> 
> 
>     One of the peers is behaving very slow - load is constantly around
>     6-15, and generally all connected clients work very slow.
> 
>     As soon as I shut that peer down, clients are relatively fast.
> 
> 
>     What I noticed, that on one gluster peer, I only get around 100
>     files opened by the gluster process:
> 
>     # lsof -n|grep -c gluster
>     96
> 
> 
>     On the slow peer, it's hundreds, tens of thousands, and growing
>     constantly:
> 
> 
>     # while true; do lsof -n|grep -c gluster; sleep 10; done
>     169934
>     170181
>     170363
>     170655
> 
> 
>     If I restart glusterd on that peer, it starts to grow again.
> 
> 
> 
>     On both peers, I have a big number of such entries:
> 
>     [2011-07-06 03:22:00.20873] W
>     [server-resolve.c:556:server___resolve] 0-gluster-data-server: pure
>     path resolution for /some/path/.../file (LOOKUP)
> 
> 
>     Where the path is a symbolic link to the gluster mount.
> 
> 
> 
> Can you describe this a little more? What do you mean by those paths are 
> symbolic links to the gluster mount? They are supposed to be files 
> within the gluster mount. Do you have a gluster mount on that peer as 
> well? If so, does unmounting have any effect?

I mean, glusterfs is mounted in /home/glusterfs, with real data in /home/gluster-data.

/some/ is a symbolic link to /home/www; where we have some more symlinks, this time to somewhere inside the gluster mount point, /home/glusterfs.

For example, with a path like:

/some/storage_cache/tatuaze/cache/photos/

we have:

1) symlink:

/some -> /home/www

2) another symlink

/home/www/storage_cache -> /home/glusterfs/storage_cache

So the real path without symlinks is this one:

/home/glusterfs/storage_cache/tatuaze/cache/photos/



> This is strange. There is almost no reason how you can be seeing this 
> error unless the extended attributes were modified outside gluster. Can 
> you get a dump of those attributes on those files with 'getfattr -d -m . 
> -e hex /some/path/../file' and post it here?

# getfattr -d -m . -e hex /home/gluster-data/www/storage_cache/tatuaze/cache/photos/tatuaz/4662/tatuaze-na-przedramieniu-5663_5.jpg
getfattr: Removing leading '/' from absolute path names
# file: home/gluster-data/www/storage_cache/tatuaze/cache/photos/tatuaz/4662/tatuaze-na-przedramieniu-5663_5.jpg
trusted.afr.gluster-data-client-0=0x000000000000000000000000
trusted.afr.gluster-data-client-1=0x000000000000000000000000
trusted.gfid=0xc46429f760cf4a278ae9bc38eec597f9



Note that the issue is somehow gone; I've made "ls -lR" on both backends,


-- 
Tomasz Chmielewski
http://wpkg.org