[Gluster-devel] more bugs (was Re: io-threads...)

[Brent A Nelson]

A couple of more bugs observed today:

1) stat-prefetch still causes glusterfs to die on occasion.  I can 
reproduce this with a bunch of clients doing a du of a complex directory 
structure; out of 8 clients du'ing simultaneously, one or two will die 
before the du finishes (glusterfs dies).  This is probably the same thing 
I've reported before about stat-prefetch, but I was hoping io-threads 
might have been responsible (it wasn't).

2) NFS reexport is somehow triggering a really rapid memory 
leak/consumption in glusterfsd, causing it to quickly die.  On the NFS 
client, I did a du of an Ubuntu Edgy mirror, which worked fine.  Then I 
did multiple cp -a's of a simple 30MB directory, which causes rapid memory 
consumption on the glusterfsd of node1 of an AFR.  It soon dies (before it 
does the sixth copy), along with the NFS-exported glusterfs client 
(running on node1, as well).  This occurs in a simple mirror with 
storage/posix and protocol/server on the server and protocol/client, 
cluster/afr, performance/read-ahead, and performance/write-behind on the 
client.

Thanks,

Brent

On Fri, 27 Apr 2007, Brent A Nelson wrote:

> Hmm, it looks like io-threads is responsible for more than just mtime 
> glitches when used with write-behind.  I just found that the problems I had 
> with NFS re-export go away when I get rid of io-threads (plus, now that I can 
> enable write-behind, the NFS write performance is far better, by at least a 
> factor of 5)!
>
> It looks like I'll be switching off io-threads for now, and turning on all 
> the other performance enhancements.
>
> Thanks,
>
> Brent
>
> On Fri, 27 Apr 2007, Brent A Nelson wrote:
>
>> On Thu, 26 Apr 2007, Anand Avati wrote:
>> 
>>> Brent,
>>> I understand what is happening. It is because I/O threads lets the
>>> mtime overtake the write call. I assume you have loaded io-threads on
>>> server side (or below write-behind on client side).
>> 
>> Yes, I have io-threads loaded on the server.  This occurs when I load 
>> write-behind on the client.
>> 
>>> I could provide you a temporary 'ugly' fix just for you if the issue is 
>>> critical (until the proper framework comes in 1.4)
>> 
>> It would be worthwhile if the temporary fix is acceptable for the 1.3 
>> release (otherwise, you'll need a warning included with the release, so 
>> that people enabling io-threads and write-behind know what to expect), but 
>> don't waste your time if it's just for me.  Push on to 1.4 and the real 
>> fix; I'll just leave write-behind disabled for now.
>> 
>> Many Thanks,
>> 
>> Brent
>> 
>