[Gluster-users] 2.0.6

[Stephan von Krawczynski]

On Fri, 21 Aug 2009 17:42:21 -0500 (CDT)
Anand Avati <avati at gluster.com> wrote:

> Stephan,
>    Please find replies below. I am merging the thread back to the ML.
> 
> > > Stephan, we need some more info. I think we are a lot closer to
> > diagnosing this issue now. The hang is being caused by an io-thread
> > getting hung either as a deadlock inside the glusterfsd process code,
> > or blocked on disk access for an excessively long time. The following
> > details will be _extremely_ useful for us.
> > > 
> > > 1. What is your backend FS, kernel version and distro running on
> > server2? Is the backend FS on a local disk or some kind of SAN or
> > iSCSI?
> > 
> > The backend FS is reiserfs3, kernel version 2.6.30.5, distro openSuSE
> > 11.1.
> > The backend FS resides on a local Areca RAID system. See attached
> > output of
> > former email. 
> > 
> > > 2. Was the glusterfsd on server2 taking 100% cpu at the time of the
> > hang?
> > 
> > I can only try to remember that from the time I took the strace logs.
> > I am not
> > a 100% sure, but from typing and looking I would say the load was very
> > low,
> > probably next to zero. 
> > 
> > > 3. On server2, now that you have killed it with -11, you should be
> > having a core file in /. Can you get the backtrace from all the
> > threads? Please use the following commands -
> > > 
> > > sh# gdb /usr/sbin/glusterfsd -c /core.X
> > > 
> > > and then at the gdb prompt
> > > 
> > > (gdb) thread apply all bt
> > > 
> > > This should output the backtraces from all the threads.
> > 
> > The bad news is this: we were not able to normally shut down the box
> > because
> > the local (exported) fs hung completely. So shutdown did not work. We
> > had to
> > hard-reset it. When examining the box few minutes ago we had to find
> > out that
> > all logs (and likely the core dump) were dumped and lost. I have seen
> > this
> > kind of behaviour before, it is originated from reiserfs3 and not
> > really
> > unusual. This means: we redo the test and hope we can force the
> > problem again.
> > Then we take all possible logs, dmesg, cores away from the server
> > before
> > rebooting it. I am very sorry we lost the important part of
> > information... 
> 
> Stephan,
>    This clearly points that the root cause for the bonnie hangs which you have been facing on every release of 2.0.x is because of the hanging reiserfs export you have. When you have the backend FS which is misbehaving, this is the expected behavior of GlusterFS. Not only will you see this in all versions of GlusterFS, you will face the same hangs even with NFS or even running bonnie directly  on your backend FS. All the IO calls are getting queued and blocked in the IO thread which is touching the disk, and the main FS thread is up responding to ping-pong requests, thus keeping the server "alive".
> All of us on this ML could have spent far fewer cycles if the initial description of the problem included a note which mentioned that one of the server's backend reiserfs3 is known to freeze in the environment before. When someone reports a hang on the glusterfs mountpoint, the first thing we developers do is trying to find code paths for what we call "missing frames" (technically it is a syscall leak, somewhat like a memory leak) and this is a very demanding and time consuming debugging for us. All the information you can provide us will only help us debug the issue faster.

Break!

Sorry to interrupt you, but obviously you misunderstood my statement. It is
_not_ the case that reiserfs3 misbehaves or hangs. In fact reiserfs3 never
hung on a single system (we use it nearly everywhere for the last 5 years or
so. What I was trying to tell you was that if you _have_ an instable box and
hard-reset it you will notice that reiserfs3 handles this case perfectly but
looses things that were not synced to disk _before_ the instability. ext3 btw
blows up the whole fs in these situations, and exactly this is the reason why
we do not use it.
It is perfectly clear to us that glusterfs(d) is the reason for the box
becoming instable and producing a hang even on a local fs (you cannot df on
the exported partition for example).
We will therefore continue with debugging as told before.

-- 
Regards,
Stephan

technical PS: Please do not post emails with lines longer than 998 characters
+ CRLF . This is not supported by SMTP. Read RFC 2821. I split your line half.
Your email client is broken if it lets you send something like that.