[Gluster-devel] Major lock-up problem

Wed Jan 9 18:25:34 UTC 2008

Ok, this looks like being a XEN/kernel issue as I can reproduce it without actually "using" the glusterfs, even though it's there and mounted.

I've included my XEN mailing list post here as this problem could well effect anyone else using gluster and XEN and it's a bit nasty in that it becomes more frequent the less memory you have .. so the more XEN instances you add, the more unstable your server becomes.

(and I'm fairly convinced gluster is "the" FS to use with XEN .. especially when the current feature requests are processed)

:)

Regards,
Gareth.

-----------

Posting to XEN list;

Ok, I've been chasing this for many days .. I have a server running 10 instances that periodically freezes .. then sometimes "comes back."

I tried many things to try to spot the problem and finally found it by accident.
It's a little frustrating as typically the Dom0 and One (or two) instances "go" and the rest carry on .. and there is diddley squat when it comes to logging information or error messages.

I'm now using 'watch "cat /proc/meminfo"' in the Dom0.
I watch the Dirty figure increase, and occasionally decrease.

In an instance (this is just an easy way to reproduce it quickly) do;
dd if=/dev/zero of=/tmp/bigfile bs=1M count=1000

Watch the "dirty" rise and at some point you'll see "writeback" cut in.
All looks good.

Give it a few seconds and your "watch" of /proc/meminfo will freeze.
On my system "Dirty" will at this point be reading about "500M" and "writeback" will have gone down to zero.
"xm list" in another session will confirm that you have a major problem. (it will hang)

For some reason PDFLUSH is not working properly !!!
On another shell "sync" and the machine instantly jumps back to life!

I'm running a stock Ubuntu XEN 3.1 kernel.
File back XEN instances, typically 5Gb with 1Gb swap.
Dual / Dual Core 2.8G Xeon (4 in total) with 6Gb RAM.
Twin 500Gb SATA HDD (software RAID1)

To my way of thinking (!) when it runs out of memory, it should force a sync (or similar) and it's not, it's just sitting there. If I wait for the dirty_expire_centisecs timer to expire, I may get some life back, some instances will survive and some will have hung.

Here's a working "meminfo";

MemTotal:       860160 kB
MemFree:         22340 kB
Buffers:         49372 kB
Cached:         498416 kB
SwapCached:      15096 kB
Active:          92452 kB
Inactive:       491840 kB
SwapTotal:     4194288 kB
SwapFree:      4136916 kB
Dirty:            3684 kB
Writeback:           0 kB
AnonPages:       29104 kB
Mapped:          13840 kB
Slab:            45088 kB
SReclaimable:    25304 kB
SUnreclaim:      19784 kB
PageTables:       2440 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   4624368 kB
Committed_AS:   362012 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      3144 kB
VmallocChunk: 34359735183 kB

Here's one where "xm list" hangs, but my "watch" is still updating the /proc/meminfo display;

MemTotal:       860160 kB
MemFree:         13756 kB
Buffers:         53656 kB
Cached:         502420 kB
SwapCached:      14812 kB
Active:          84356 kB
Inactive:       507624 kB
SwapTotal:     4194288 kB
SwapFree:      4136900 kB
Dirty:          213096 kB
Writeback:           0 kB
AnonPages:       28832 kB
Mapped:          13924 kB
Slab:            45988 kB
SReclaimable:    25728 kB
SUnreclaim:      20260 kB
PageTables:       2456 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   4624368 kB
Committed_AS:   361796 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      3144 kB
VmallocChunk: 34359735183 kB

Here's a frozen one;

MemTotal:       860160 kB
MemFree:         15840 kB
Buffers:          2208 kB
Cached:         533048 kB
SwapCached:       7956 kB
Active:          49992 kB
Inactive:       519916 kB
SwapTotal:     4194288 kB
SwapFree:      4136916 kB
Dirty:          505112 kB
Writeback:        3456 kB
AnonPages:       34676 kB
Mapped:          14436 kB
Slab:            64508 kB
SReclaimable:    18624 kB
SUnreclaim:      45884 kB
PageTables:       2588 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:   4624368 kB
Committed_AS:   368064 kB
VmallocTotal: 34359738367 kB
VmallocUsed:      3144 kB
VmallocChunk: 34359735183 kB

Help!!!

Gareth.

--
Managing Director, Encryptec Limited
Tel: 0845 25 77033, Mob: 07853 305393, Int: 00 44 1443205756
Email: gareth at encryptec.net 
Statements made are at all times subject to Encryptec's Terms and Conditions of Business, which are available upon request.

----- Original Message -----
From: "Gareth Bult" <gareth at encryptec.net>
To: "gluster-devel" <gluster-devel at nongnu.org>
Sent: Wednesday, January 9, 2008 3:40:49 PM (GMT) Europe/London
Subject: [Gluster-devel] Major lock-up problem

Hi, 

I've been developing a new system (which is now "live", hence the lack of debug information) and have been experiencing lots of inexplicable lock up and pause problems with lots of different components, and I've been working my way through the systems removing / fixing problems as I go. 

I seem to have a problem with gluster I can't nail down. 

When hitting the server with sustained (typically multi-file) writes, after a while the server goes "D" state. 
If I have io-threads running on the server, only ONE process goes "D" state. 

Trouble is, it stays "D" state and starts to lock up other processes .. a favourite is "vi". 

Funny thing is, the machine is a XEN server (glusterfsd in the Dom0) and the XEN instances NOT using gluster are not affected. 
Some of the instances using the glusterfs are affected, depending on whether io-threads is used on the server. 

If I'm lucky, I kill the IO process and 5 mins later the machine springs back to life. 
If I'm not, I reboot. 

Anyone any ideas? 

glfs7 and tla. 

Gareth. 
_______________________________________________
Gluster-devel mailing list
Gluster-devel at nongnu.org
http://lists.nongnu.org/mailman/listinfo/gluster-devel