[Gluster-users] Strange server locks isuess with 2.0.7 - updating

[Marek]

Why You suggest ping-timeout with that high value?
When some brick gets in trouble, mounted fs on client side is unusable (I/O is locked)
and have to wait 120 sec. for timeout and "release fs".
Locked client IO for 120 sec. is not acceptable.


regards,

Stephan von Krawczynski wrote:
> Try setting your ping-timeout way higher, since we use 120 we have almost no
> issues in regular use. Nevertheless we do believe every problem will come back
> when some brick(s) die...
> 
> 
> On Tue, 10 Nov 2009 14:59:07 +0100
> Marek Blaszkowski <mb at kis.p.lodz.pl> wrote:
> 
>> OK,
>> here goes some more details, on a "bad" servers (with strange lockups) we got
>> problems with open/move files. We are unable to open,move or just ls files
>> (file utils just hangs )....
>>
>>
>> Marek wrote:
>>> Hello,
>>> we're testing a simple configuration of glusterfs 2.0.7 with 4 servers 
>>> and 1 client (2+2 bricks each replicated with
>>> a distribute translator, configs below).
>>> Durning our tests (client side copying/moving a lot of small files on 
>>> glusterfs mounted FS) we got a strange
>>> lockups on two of servers (bricks).
>>> I was unable to login (via ssh) to server, on already started terminal 
>>> sessions I couldn't spawn a "top"
>>> process (it just hangs), vmstats exists with floating point exception. 
>>> Other fileutils commands behaves "normal".
>>> There was no any dmesg kernel messages (first guess was a kernel ups or 
>>> other kernel related problems).
>>> This server never had any CPU/memory problems under high loads before. 
>>> Problems starts when we
>>> run glusterfsd on this server. We had to a hard reset malfunction server 
>>> (reboot doesn't work).
>>> After a couple hours of testing another server disconected from a client 
>>> (according to a client debug log).
>>> Scenario was the same:
>>> 1. unable to login to a server, connection was established but sshd on 
>>> server side hang/timeout after entering a user password
>>> 2. on a previous established terminal sessions was unable to run top or 
>>> vmstat utility (vmstats exit with with
>>> floating point exception. Copying/moving files was OK. Load was  0.00, 
>>> 0.00, 0.00
>>>
>>>
>>> What could be wrong? These servers never had problems before (simple 
>>> terminal/proxy servers). Strange locking looks
>>> like related to a kernel VM structures (why top/vmstat behaves odd??) or 
>>> other kernel related problems.
>>>
>>> Server remote1 details: Linux version 2.6.26-1-686 (Debian 
>>> 2.6.26-13lenny2) (dannf at debian.org)
>>> (gcc version 4.1.3 20080704 (prerelease) (Debian 4.1.2-25)) #1 SMP Fri 
>>> Mar 13 18:08:45 UTC 2009
>>> running debian 5.0
>>>
>>> Server remote2 details: Linux version 2.6.22-14-server (buildd at palmer) 
>>> (gcc version 4.1.3 20070929
>>> (prerelease) (Ubuntu 4.1.2-16ubuntu2)) #1 SMP Sun Oct 14 23:34:23 GMT 2007
>>> running ubuntu
>>> both run glusterfsd:
>>>  /usr/local/sbin/glusterfsd -p /var/run/glusterfsd.pid -f 
>>> /usr/local/etc/glusterfs/glusterfs-server.vol
>>>
>>>
>>> Note that both servers runs different os versions and got simillar 
>>> lockup problems, never having problems
>>> before (without glusterfsd).
>>>
>>>
>>> Server gluster config file (the same on 4 servers):
>>> -----------------cut here------------------------
>>> volume brick
>>> type storage/posix
>>> option directory /var/gluster
>>> end-volume
>>>
>>> volume locks
>>> type features/posix-locks
>>> subvolumes brick
>>> end-volume
>>>
>>> volume server
>>> type protocol/server
>>> option transport-type tcp/server
>>> option auth.ip.locks.allow *
>>> option auth.ip.brick-ns.allow *
>>> subvolumes locks
>>> end-volume
>>> -----------------cut here-----------------------
>>>
>>> client gluster config below (please note remote1 and remote4 got 
>>> problems metioned above), gluster client was
>>> start with a command:
>>> glusterfs --log-file=/var/log/gluster-client -f 
>>> /usr/local/etc/glusterfs/glusterfs-client.vol /var/glustertest
>>>
>>>
>>> -----------------client config-cut here-----------------------
>>> volume remote1
>>> type protocol/client
>>> option transport-type tcp/client
>>> option remote-host 192.168.2.184
>>> option ping-timeout 5
>>> option remote-subvolume locks
>>> end-volume
>>>
>>> volume remote2
>>> type protocol/client
>>> option transport-type tcp/client
>>> option remote-host 192.168.2.195
>>> option ping-timeout 5
>>> option remote-subvolume locks
>>> end-volume
>>>
>>> volume remote3
>>> type protocol/client
>>> option transport-type tcp/client
>>> option remote-host 192.168.2.145
>>> option ping-timeout 5
>>> option remote-subvolume locks
>>> end-volume
>>>
>>> volume remote4
>>> type protocol/client
>>> option transport-type tcp/client
>>> option remote-host 192.168.2.193
>>> option ping-timeout 5
>>> option remote-subvolume locks
>>> end-volume
>>>
>>> volume afr1
>>> type cluster/replicate
>>> subvolumes remote1 remote3
>>> end-volume
>>>
>>> volume afr2
>>> type cluster/replicate
>>> subvolumes remote2 remote4
>>> end-volume
>>>
>>>
>>> volume bigfs
>>> type cluster/distribute
>>> subvolumes afr1 afr2
>>> end-volume
>>>
>>> volume writebehind
>>> type performance/write-behind
>>> option flush-behind on
>>> option cache-size 3MB
>>> subvolumes bigfs
>>> end-volume
>>>
>>> volume readahead
>>> type performance/read-ahead
>>> option page-count 16
>>> subvolumes writebehind
>>> end-volume
>>> -----------------cut here--------------------------------------
>>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
> 
>