[Gluster-users] Failure: "Transport endpoint is not connected" (but it is!)

[Benjamin Smith]

Haven't seen the "Transport endpoint" problem again, yet. Instead, a different 
problem surfaced this morning. 

The servers began to "hang", taking 60 seconds or more to return a read. lsof 
showed many open files in the GlusterFS partition, all being read. Traffic 
monitor showed extremely high volume data flow (essentially, 1 Gb) between 
the primary webserver and its glusterfs server twin. 

Shutting down the webserver, glusterfs as client, and glusterfs as server, 
then restarting the whole stack from server to client to apache resulted in a 
system that was responsive - for a while. The reads were nearly all of the 
same few dozen files that I want to have replicated on GlusterFS. 

Based on what I've seen, I guess that: 

1) Glusterfs does some kind of coherency check at every file read. 

2) Glusterfs processes these coherency checks serially, 

3) The coherency checking was backing up. 

Am I out in left field, here? Is there something terrible and fundamental that 
I'm missing, or is GlusterFS + Ethernet + stock Fuse + basic config just not 
going to do all that well with medium-to-large amounts of reads of a few 
hundred small files? (say, 50/second) 

I ended up rolling back glusterfs and went back to a single, local file 
system, and would like to move forward on this... 

On Monday 19 January 2009 11:59:05 pm you wrote:
> 1. whether glusterfsd is running on the server or not, with the
> process state (from ps) if running
> 2. backtrace of coredump using gdb if it has crashed.
>
> we can figure the next step only after having one of the above two
>
> avati
>
> On Tue, Jan 20, 2009 at 1:24 PM, Benjamin Smith
>
> <lists at benjamindsmith.com> wrote:
> > On Monday 19 January 2009 05:05:20 pm you wrote:
> >> Do you have a coredump on the server? was glusterfsd running on the
> >> server at all?
> >
> > No.
> >
> > If it should happen again, what should I do to provide you with what you
> > need?
> >
> > -Ben
> >
> >> <lists at benjamindsmith.com> wrote:
> >> > Late last week, I rolled out GlusterFS on our production cluster.
> >> > Config is very simple, two active servers that are also clients to
> >> > each other. Usage is for a fairly low-volume distribution of file
> >> > settings for an application cluster that are updated perhaps a few
> >> > times per day and read constantly. (pretty much every web page hit) 
> >> > Here are the numbers:
> >>
> >> --
> >> This message has been scanned for viruses and
> >> dangerous content by MailScanner, and is
> >> believed to be clean.
> >
> > --
> > This message has been scanned for viruses and
> > dangerous content by MailScanner, and is
> > believed to be clean.
>
> --
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.



-- 
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.