[Gluster-devel] more bugs (was Re: io-threads...)
Brent A Nelson
brent at phys.ufl.edu
Sat Apr 28 02:43:41 UTC 2007
A couple of more bugs observed today:
1) stat-prefetch still causes glusterfs to die on occasion. I can
reproduce this with a bunch of clients doing a du of a complex directory
structure; out of 8 clients du'ing simultaneously, one or two will die
before the du finishes (glusterfs dies). This is probably the same thing
I've reported before about stat-prefetch, but I was hoping io-threads
might have been responsible (it wasn't).
2) NFS reexport is somehow triggering a really rapid memory
leak/consumption in glusterfsd, causing it to quickly die. On the NFS
client, I did a du of an Ubuntu Edgy mirror, which worked fine. Then I
did multiple cp -a's of a simple 30MB directory, which causes rapid memory
consumption on the glusterfsd of node1 of an AFR. It soon dies (before it
does the sixth copy), along with the NFS-exported glusterfs client
(running on node1, as well). This occurs in a simple mirror with
storage/posix and protocol/server on the server and protocol/client,
cluster/afr, performance/read-ahead, and performance/write-behind on the
On Fri, 27 Apr 2007, Brent A Nelson wrote:
> Hmm, it looks like io-threads is responsible for more than just mtime
> glitches when used with write-behind. I just found that the problems I had
> with NFS re-export go away when I get rid of io-threads (plus, now that I can
> enable write-behind, the NFS write performance is far better, by at least a
> factor of 5)!
> It looks like I'll be switching off io-threads for now, and turning on all
> the other performance enhancements.
> On Fri, 27 Apr 2007, Brent A Nelson wrote:
>> On Thu, 26 Apr 2007, Anand Avati wrote:
>>> I understand what is happening. It is because I/O threads lets the
>>> mtime overtake the write call. I assume you have loaded io-threads on
>>> server side (or below write-behind on client side).
>> Yes, I have io-threads loaded on the server. This occurs when I load
>> write-behind on the client.
>>> I could provide you a temporary 'ugly' fix just for you if the issue is
>>> critical (until the proper framework comes in 1.4)
>> It would be worthwhile if the temporary fix is acceptable for the 1.3
>> release (otherwise, you'll need a warning included with the release, so
>> that people enabling io-threads and write-behind know what to expect), but
>> don't waste your time if it's just for me. Push on to 1.4 and the real
>> fix; I'll just leave write-behind disabled for now.
>> Many Thanks,
More information about the Gluster-devel