[Gluster-devel] Re: more bugs (was Re: io-threads...)
Brent A Nelson
brent at phys.ufl.edu
Sat Apr 28 22:18:27 UTC 2007
Adding to the list, there is still an mtime bug when using write-behind
even without io-threads. It occurs on (big clue!) files with sizes evenly
divisable by 4096 in my AFR/unified setup:
-rw-r--r-- 1 root root 4096 2007-04-28 16:01 /scratch/usr/src/linux-headers-2.6.15-28/include/net/genetlink.h
-rwxr-xr-x 1 root root 12288 2007-04-28 15:54 /scratch/usr/bin/last
-rwxr-xr-x 1 root root 12288 2007-04-28 15:54 /scratch/usr/bin/aseqnet
-rw-r--r-- 1 root root 528384 2007-04-28 15:54 /scratch/usr/share/command-not-found/programs.d/all-universe.db
-rw-r--r-- 1 root root 12288 2007-04-28 15:54 /scratch/usr/share/command-not-found/programs.d/all-multiverse.db
-rw-r--r-- 1 root root 12288 2007-04-28 15:54 /scratch/usr/share/command-not-found/programs.d/i386-restricted.db
-rw-r--r-- 1 root root 135168 2007-04-28 15:54 /scratch/usr/share/command-not-found/programs.d/i386-multiverse.db
-rw-r--r-- 1 root root 12288 2007-04-28 15:54 /scratch/usr/share/command-not-found/programs.d/all-restricted.db
-rw-r--r-- 1 root root 135168 2007-04-28 15:54 /scratch/usr/share/command-not-found/programs.d/all-main.db
-rw-r--r-- 1 root root 4198400 2007-04-28 15:54 /scratch/usr/share/command-not-found/programs.d/i386-universe.db
-rw-r--r-- 1 root root 1052672 2007-04-28 15:54 /scratch/usr/share/command-not-found/programs.d/i386-main.db
-rw-r--r-- 1 root root 65536 2007-04-28 15:53 /scratch/usr/share/samba/valid.dat
-rw-r--r-- 1 root root 61440 2007-04-28 15:57 /scratch/usr/lib/python2.5/distutils/command/wininst-7.1.exe
-rw-r--r-- 1 root root 61440 2007-04-28 15:57 /scratch/usr/lib/python2.5/distutils/command/wininst-6.exe
-rw-r--r-- 1 root root 61440 2007-04-28 15:55 /scratch/usr/lib/python2.4/distutils/command/wininst-7.1.exe
-rw-r--r-- 1 root root 61440 2007-04-28 15:55 /scratch/usr/lib/python2.4/distutils/command/wininst-6.exe
-rw-r--r-- 1 root root 4096 2007-04-28 15:57 /scratch/usr/lib/gettext/msgfmt.net.exe
-rw-r--r-- 1 root root 8192 2007-04-28 15:57 /scratch/usr/lib/GNU.Gettext.dll
-rw-r--r-- 1 root root 143360 2007-04-28 15:56 /scratch/usr/lib/libgc.so.1.0.2
These files have wrong mtimes on both nodes in the AFR, not just one or
the other. It resulted from a simple "cp -a" of my /usr directory.
Thanks,
Brent
On Fri, 27 Apr 2007, Brent A Nelson wrote:
> A couple of more bugs observed today:
>
> 1) stat-prefetch still causes glusterfs to die on occasion. I can reproduce
> this with a bunch of clients doing a du of a complex directory structure; out
> of 8 clients du'ing simultaneously, one or two will die before the du
> finishes (glusterfs dies). This is probably the same thing I've reported
> before about stat-prefetch, but I was hoping io-threads might have been
> responsible (it wasn't).
>
> 2) NFS reexport is somehow triggering a really rapid memory leak/consumption
> in glusterfsd, causing it to quickly die. On the NFS client, I did a du of
> an Ubuntu Edgy mirror, which worked fine. Then I did multiple cp -a's of a
> simple 30MB directory, which causes rapid memory consumption on the
> glusterfsd of node1 of an AFR. It soon dies (before it does the sixth copy),
> along with the NFS-exported glusterfs client (running on node1, as well).
> This occurs in a simple mirror with storage/posix and protocol/server on the
> server and protocol/client, cluster/afr, performance/read-ahead, and
> performance/write-behind on the client.
>
> Thanks,
>
> Brent
>
> On Fri, 27 Apr 2007, Brent A Nelson wrote:
>
>> Hmm, it looks like io-threads is responsible for more than just mtime
>> glitches when used with write-behind. I just found that the problems I had
>> with NFS re-export go away when I get rid of io-threads (plus, now that I
>> can enable write-behind, the NFS write performance is far better, by at
>> least a factor of 5)!
>>
>> It looks like I'll be switching off io-threads for now, and turning on all
>> the other performance enhancements.
>>
>> Thanks,
>>
>> Brent
>>
>> On Fri, 27 Apr 2007, Brent A Nelson wrote:
>>
>>> On Thu, 26 Apr 2007, Anand Avati wrote:
>>>
>>>> Brent,
>>>> I understand what is happening. It is because I/O threads lets the
>>>> mtime overtake the write call. I assume you have loaded io-threads on
>>>> server side (or below write-behind on client side).
>>>
>>> Yes, I have io-threads loaded on the server. This occurs when I load
>>> write-behind on the client.
>>>
>>>> I could provide you a temporary 'ugly' fix just for you if the issue is
>>>> critical (until the proper framework comes in 1.4)
>>>
>>> It would be worthwhile if the temporary fix is acceptable for the 1.3
>>> release (otherwise, you'll need a warning included with the release, so
>>> that people enabling io-threads and write-behind know what to expect), but
>>> don't waste your time if it's just for me. Push on to 1.4 and the real
>>> fix; I'll just leave write-behind disabled for now.
>>>
>>> Many Thanks,
>>>
>>> Brent
>>>
>>
>
More information about the Gluster-devel
mailing list