[Gluster-users] The continuing story ...

[Stephan von Krawczynski]

On Tue, 8 Sep 2009 10:13:17 +1000 (EST)
"Jeff Evans" <jeffe at tricab.com> wrote:

> > - server was ping'able
> > - glusterfsd was disconnected by the client because of missing
> > ping-pong - no login possible
> > - no fs action (no lights on the hd-stack)
> > - no screen (was blank, stayed blank)
> 
> This is very similar to what I have seen many times (even back on
> 1.3), and have also commented on the list.
> 
> It seems that we have quite a few ACK's on this, or similar problems.
> 
> The only thing different in my scenario, is that the console doesn't
> stay blank. When attempting to login I get the last login message, and
> nothing more, no prompt ever. Also, I can see that other processes are
> still listening on sockets etc.. so it seems like the kernel just
> can't grab new FD's.
> 
> I too found the hang happens more easily if a downed node from a
> replicate pair re-joins after some time.
> 
> Following suggestions that this is all kernel related, I have just
> moved up to RHEL 5.4 in the hope that the new kernel will
> help.
> 
> This fix stood out as potentially related for me:
> https://bugzilla.redhat.com/show_bug.cgi?id=44543

This is an ext3 fix, unlikely that we run into a similar effect on reiserfs3,
they are really very different in internals and coding.
 
> We also have a broadcom network card, which had reports of hangs under
> load, the kernel has a patch for that too.

We used tg3 in this setup, but the load was not very high (below 10 MBit on a
1000MBit link). 

> If I still run into the hangs, I'll try xfs.

I doubt that this can be a real solution. My guess is that glusterfsd runs
into some race condition where it locks itself up completely.
It is not funny to debug something the like on a production setup. Best would
be to have debugging output sent from the servers' glusterfsd directly to a
client to save the logs. I would not count on syslog in this case, if it
survives one could use a serial console for syslog output though.
 
> Thanks, Jeff.

-- 
Regards,
Stephan