[Gluster-users] Replication not working on server hang
wdong.pku at gmail.com
Fri Aug 28 13:46:25 UTC 2009
I'm wondering if there's some way for glusterfs to detect the flaws of
the underlying operating system. I believe there's no bug-free file
systems in the universe, so I believe it is the job of the glusterfs
developer to specify which underlying filesystem is tested and
supported. It's not good to simply say that glusterfs works on all
real-world approximations to an imaginary bug-free posix filesystem.
Stephan von Krawczynski wrote:
> On Fri, 28 Aug 2009 14:28:51 +0200
> David Saez Padros <david at ols.es> wrote:
>> well, that never hapen before when using nfs with the same
>> computers, same disk, etc ... for almost 2 years, so it's more
>> than possible that is glusterfs the one which is triggering this
>> suposed ext3 bug, but appart from this:
> I can assure you that you will never have an agreement on this point on this
> list, this happens to be the only bugfree software in universe according to
> authors ;-)
>> a) documentation says "All operations that do not modify the file
>> or directory are sent to all the subvolumes and the first successful
>> reply is returned to the application", why is blocking then ?
>> it's suposed that the reply from the non blocked server will
>> come first and nothing will block, but clients are blocking on
>> a simple ls operation
> My impression is that you have to imagine the setup as serialized queue on the
> server. If there was one operation with a hang, all future ones will be
> hanging, too.
>> b) server1 (the non blocked one) also has the volumes mounted like
>> any other client, but having option read-subvolume set to the local
>> volume, but it also hangs when it was suposed to read from the local
>> volume, not from the hanged one
> This is exactly my experience. You cannot make it work either way. There seems
> to be some locking across all used servers.
>> c) does not glsuterfs ping the servers periodically to see if they
>> are available or not ? if so, why does not it detect that situation ?
> Well, this ping-pong procedure seems to be only detecting offline servers
> (i.e. network down), but is obviously not able to give hints about being
> operational or not.
> My idea of a solution would be to implement something like a bail-out timeout
> configurable on the client vol file for every brick. This would allow to
> intermix slow and fast servers and it would cope with a situation where some
> clients are far away with a slow connnections and others are nearby with very
> fast connection to the same servers.
> The biggest problem about it probably is not to bail out servers, but to
> re-integrate them. Currently there seems to be no userspace tool to tell a
> client to re-integrate a formerly dead server. Obviously this should not
> happen auto-magically to prevent flapping.
>>>> Glusterfs log only shows lines like this ones:
>>>> [2009-08-28 09:19:28] E [client-protocol.c:292:call_bail] data2: bailing
>>>> out frame LOOKUP(32) frame sent = 2009-08-28 08:49:18. frame-timeout = 1800
>>>> [2009-08-28 09:23:38] E [client-protocol.c:292:call_bail] data2: bailing
>>>> out frame LOOKUP(32) frame sent = 2009-08-28 08:53:28. frame-timeout = 1800
>>>> Once server2 has been rebooted all gluster fs become available
>>>> again on all clients and the hanged df and ls processes terminate,
>>>> but difficult to understand why a replicated share that must survive
>>>> to failure on one server does not.
>>> You are suffering from the problem we talked about few days ago on the list.
>>> If your local fs produces a deadlock somehow on one server glusterfs is
>>> currently unable to cope with the situation and just _waits_ for things to
>>> come. This deadlocks your clients, too, without any need.
>>> Your experience backs my critics on the handling of these situations.
>> Best regards ...
>> David Saez Padros http://www.ols.es
>> On-Line Services 2000 S.L. telf +34 902 50 29 75
More information about the Gluster-users