[Gluster-devel] Too many open files

[Anand Avati]

Brent,
  can you please send me your spec files? because I am able to 'ls'
without any problems and there is no fd leak observed. I have loaded
just cluster/afr, and previously had loaded all performance xlators on
both server and clietn side together and in both the cases things
worked perfectly fine.

 I'm guessing the encrytpion makefile issue caused a bad build? (things
were changed in libglusterfs). the makefile is committed now though
(along with the -l fix). please do a make uninstall/clean/install
since quit a chunk of changes have gone in the last few days.

avati

On Fri, Apr 06, 2007 at 03:33:30PM -0400, Brent A Nelson wrote:
> glusterfsd dies on both nodes almost immediately (I can ls succesfully 
> once before it dies, but cd in and they're dead).  The glusterfs processes 
> are still running, but I of course have "Transport endpoint is not 
> connected."
> 
> Also, glusterfsd and glusterfs no longer seem to know where to log by 
> default and refuse to start unless I give the -l option on each.
> 
> Thanks,
> 
> Brent
> 
> On Fri, 6 Apr 2007, Anand Avati wrote:
> 
> >Brent,
> > the fix has been committed. can you please check if it works for you?
> >
> > regards,
> > avati
> >
> >On Thu, Apr 05, 2007 at 02:09:26AM -0400, Brent A Nelson wrote:
> >>That's correct.  I had commented out unify when narrowing down the mtime
> >>bug (which turned out to be writebehind) and then decided I had no reason
> >>to put it back in for this two-brick filesystem.  It was mounted without
> >>unify when this issue occurred.
> >>
> >>Thanks,
> >>
> >>Brent
> >>
> >>On Wed, 4 Apr 2007, Anand Avati wrote:
> >>
> >>>Can you confirm that you were NOT using unify int he setup??
> >>>
> >>>regards,
> >>>avati
> >>>
> >>>
> >>>On Thu, Apr 05, 2007 at 01:09:16AM -0400, Brent A Nelson wrote:
> >>>>Awesome!
> >>>>
> >>>>Thanks,
> >>>>
> >>>>Brent
> >>>>
> >>>>On Wed, 4 Apr 2007, Anand Avati wrote:
> >>>>
> >>>>>Brent,
> >>>>>thank you so much for your efforts of sending the output!
> >>>>>from the log it is clear the leak fd's are all for directories. Indeed
> >>>>>there was an issue with releasedir() call reaching all the nodes. The
> >>>>>fix should be committed today to tla.
> >>>>>
> >>>>>Thanks!!
> >>>>>
> >>>>>avati
> >>>>>
> >>>>>
> >>>>>
> >>>>>On Wed, Apr 04, 2007 at 09:18:48PM -0400, Brent A Nelson wrote:
> >>>>>>I avoided restarting, as this issue would take a while to reproduce.
> >>>>>>
> >>>>>>jupiter01 and jupiter02 are mirrors of each other.  All performance
> >>>>>>translators are in use, except for writebehind (due to the mtime bug).
> >>>>>>
> >>>>>>jupiter01:
> >>>>>>ls -l /proc/26466/fd |wc
> >>>>>>65536  655408 7358168
> >>>>>>See attached for ls -l output.
> >>>>>>
> >>>>>>jupiter02:
> >>>>>>ls -l /proc/3651/fd |wc
> >>>>>>ls -l /proc/3651/fd
> >>>>>>total 11
> >>>>>>lrwx------ 1 root root 64 2007-04-04 20:43 0 -> /dev/null
> >>>>>>lrwx------ 1 root root 64 2007-04-04 20:43 1 -> /dev/null
> >>>>>>lrwx------ 1 root root 64 2007-04-04 20:43 10 -> socket:[2565251]
> >>>>>>lrwx------ 1 root root 64 2007-04-04 20:43 2 -> /dev/null
> >>>>>>l-wx------ 1 root root 64 2007-04-04 20:43 3 ->
> >>>>>>/var/log/glusterfs/glusterfsd.log
> >>>>>>lrwx------ 1 root root 64 2007-04-04 20:43 4 -> socket:[2255275]
> >>>>>>lrwx------ 1 root root 64 2007-04-04 20:43 5 -> socket:[2249710]
> >>>>>>lr-x------ 1 root root 64 2007-04-04 20:43 6 -> eventpoll:[2249711]
> >>>>>>lrwx------ 1 root root 64 2007-04-04 20:43 7 -> socket:[2255306]
> >>>>>>lr-x------ 1 root root 64 2007-04-04 20:43 8 ->
> >>>>>>/etc/glusterfs/glusterfs-client.vol
> >>>>>>lr-x------ 1 root root 64 2007-04-04 20:43 9 ->
> >>>>>>/etc/glusterfs/glusterfs-client.vol
> >>>>>>
> >>>>>>Note that it looks like all those extra directories listed on 
> >>>>>>jupiter01
> >>>>>>were locally rsynched from jupiter01's Lustre filesystems onto the
> >>>>>>glusterfs client on jupiter01.  A very large rsync from a different
> >>>>>>machine to jupiter02 didn't go nuts.
> >>>>>>
> >>>>>>Thanks,
> >>>>>>
> >>>>>>Brent
> >>>>>>
> >>>>>>On Wed, 4 Apr 2007, Anand Avati wrote:
> >>>>>>
> >>>>>>>Brent,
> >>>>>>>I hope the system is still in the same state to dig some info out.
> >>>>>>>To verify that it is a file descriptor leak, can you please run this
> >>>>>>>test. On the server, run ps ax and get the PID of glusterfsd. then do
> >>>>>>>an ls -l on /proc/<pid>/fd/ and please mail the output of that. That
> >>>>>>>should give a precise idea of what is happening.
> >>>>>>>If the system has been reset out of the state, please give us the
> >>>>>>>spec file you are using and the commands you ran (of some major jobs
> >>>>>>>like heavy rsync) so that we will try to reproduce the error in our
> >>>>>>>setup.
> >>>>>>>
> >>>>>>>regards,
> >>>>>>>avati
> >>>>>>>
> >>>>>>>
> >>>>>>>On Wed, Apr 04, 2007 at 01:12:33PM -0400, Brent A Nelson wrote:
> >>>>>>>>I put a 2-node GlusterFS mirror into use internally yesterday, as
> >>>>>>>>GlusterFS was looking pretty solid, and I rsynced a whole bunch of
> >>>>>>>>stuff
> >>>>>>>>to it.  Today, however, an ls on any of the three clients gives me:
> >>>>>>>>
> >>>>>>>>ls: /backup: Too many open files
> >>>>>>>>
> >>>>>>>>It looks like glusterfsd hit a limit.  Is this a bug
> >>>>>>>>(glusterfs/glusterfsd
> >>>>>>>>forgetting to close files; essentially, a file descriptor leak), or
> >>>>>>>>do I
> >>>>>>>>just need to increase the limit somewhere?
> >>>>>>>>
> >>>>>>>>Thanks,
> >>>>>>>>
> >>>>>>>>Brent
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>_______________________________________________
> >>>>>>>>Gluster-devel mailing list
> >>>>>>>>Gluster-devel at nongnu.org
> >>>>>>>>http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >>>>>>>>
> >>>>>>>
> >>>>>>>--
> >>>>>>>Shaw's Principle:
> >>>>>>>    Build a system that even a fool can use,
> >>>>>>>    and only a fool will want to use it.
> >>>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>--
> >>>>>Shaw's Principle:
> >>>>>     Build a system that even a fool can use,
> >>>>>     and only a fool will want to use it.
> >>>>>
> >>>>
> >>>
> >>>--
> >>>Shaw's Principle:
> >>>      Build a system that even a fool can use,
> >>>      and only a fool will want to use it.
> >>>
> >>
> >
> >-- 
> >Shaw's Principle:
> >       Build a system that even a fool can use,
> >       and only a fool will want to use it.
> >
> 

-- 
Shaw's Principle:
        Build a system that even a fool can use,
        and only a fool will want to use it.