[Gluster-users] Replication not working on server hang

[Anand Avati]

> i don't think that this is a hang on the file system itself, another
> user reported the same problem on another file system type and it's
> very unlikely that both file systems had the same problem. Also i have
> the problem with a ext3 file system which is very mature. I will say
> that is more likely that the failure is in the driver, the kernel itself
> or even in glusterfs than in the file system itself. The fact that this
> bug does not appear when using the file system directly or through nfs
> make glusterfs the most obvious candidate.

What we have here (kernel lockups and glusterfs on the same machine)
might not be a co-incidence. There might well be a correlation -- but
by nature of the problem it is not right to treat this as a
cause-effect relation with glusterfs being the cause. At best,
glusterfs could be performing some IO pattern (extended attribute
usage?) which might be triggering some bugs in the kernel. It is just
not right to blame _any_ userspace application for any kind of kernel
lockups or hangs. This is not just my personal opinion, but goes with
the definition of OS kernels - they are running in a protected memory
and communicating only via a secure system call interface and few
other mechanisms. So any hang or lockup in the kernel can only be
caused by a bug in itself, which could possibly be triggered by a
specific user application.

>> It's not a simple problem to solve. But, it should be solved.
>
> right, i'm also a developer and can understand that some developers
> may see some bug reports as a personal attack, but the ones in this
> list (at least me) see glusterfs as a good thing (that why we are
> using it) and we are not attacking the software or the authors, we
> are just reporting annoying situations that we feel should be
> corrected. Bug reports that this ones should be investigated and
> workarounds should be found so glusterfs volumes will be able to
> tolerate this 'other software' faults gracefully

Well nothing has been taken personal so far. We acknowledge that it is
a limitation in today's version of glusterfs that backend FS hangs
resulting via kernel lockups are not handled by a graceful failover --
reason being internally glusterfs still does not translate "backend fs
hang" into a "subvolume down" status. In fact glusterfs does not even
recognize a "backend fs hang" situation as of today. This backend fs
hang can happen only because of a kernel bug. It is true that this
situation is not handled by glusterfs today. We will fix this soon.
What we will be fixing is failing over to other machines when the
backend FS hangs. The reason why this was not a priority (so far
atleast) is because a kernel is a trusted piece of software in the
system, and when you are having a kernel which has a bug in the fs,
you should just upgrade to a newer kernel. Kernel hangs and
application hangs are very different (on which you are probably not
sharing my point of view). What we promise to fix is a way to (as best
as possible) somehow translate a backend FS hang into a "subvolume
down" status and consider that subvolume to be down. After that, you
will _still_ continue to face kernel hangs and lockups and just
glusterfs will stop hanging. Your machines would still remain locked
up.

Avati