[Gluster-users] gluster opens thousands of files on one peer

[Anand Avati]

On Wed, Jul 6, 2011 at 6:58 AM, Tomasz Chmielewski <mangoo at wpkg.org> wrote:

>
> One of the peers is behaving very slow - load is constantly around 6-15,
> and generally all connected clients work very slow.
>
> As soon as I shut that peer down, clients are relatively fast.
>
>
> What I noticed, that on one gluster peer, I only get around 100 files
> opened by the gluster process:
>
> # lsof -n|grep -c gluster
> 96
>
>
> On the slow peer, it's hundreds, tens of thousands, and growing constantly:
>
>
> # while true; do lsof -n|grep -c gluster; sleep 10; done
> 169934
> 170181
> 170363
> 170655
>
>
> If I restart glusterd on that peer, it starts to grow again.
>
>
>
> On both peers, I have a big number of such entries:
>
> [2011-07-06 03:22:00.20873] W [server-resolve.c:556:server_**resolve]
> 0-gluster-data-server: pure path resolution for /some/path/.../file (LOOKUP)
>
>
> Where the path is a symbolic link to the gluster mount.
>


Can you describe this a little more? What do you mean by those paths are
symbolic links to the gluster mount? They are supposed to be files within
the gluster mount. Do you have a gluster mount on that peer as well? If so,
does unmounting have any effect?


> The unaffected peer has these entries in the log as well:
>
> [2011-07-06 03:23:30.117245] E [posix.c:3672:do_xattrop]
> 0-gluster-data-posix: getxattr failed on /some/path/.../file while doing
> xattrop: Numerical result out of range
>


This is strange. There is almost no reason how you can be seeing this error
unless the extended attributes were modified outside gluster. Can you get a
dump of those attributes on those files with 'getfattr -d -m . -e hex
/some/path/../file' and post it here?


>


> Why is gluster on one of the peers opening so many files?
>
>
>
It is certainly strange that it's opening more files on only one peer.
Please give as much info about the setup as possible.

Avati
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20110709/9fa6da0c/attachment.html>