[Gluster-devel] memory leaks
Brent A Nelson
brent at phys.ufl.edu
Mon Mar 19 23:45:41 UTC 2007
I just wanted to confirm that, yes, this does indeed fix the problem.
Many Thanks!
Brent
On Sun, 18 Mar 2007, Anand Avati wrote:
> Brent,
> The 'leak' has been plugged in the read-ahead translator of the
> latest tla patch. the problem was an integer overflow issue where the
> 'size' parameter for the flush function was 32bit but when the file
> was bigger than 4GB, the code tried to flush a region from 0 to >4GB.
> changing the function prototype to have a 64bit size fixed it.
>
> The 'leak' was actually not a 'leak' because when the file was closed,
> everything was flushed and free'd.
>
> thanks!
> avati
>
>
> On Mon, Mar 12, 2007 at 02:15:31PM -0400, Brent A Nelson wrote:
>> Attached are my present spec files for both servers (which are also my
>> clients); both machines use the same files.
>>
>> To trigger the memory leak, all I need to do is a "dd if=/phys/blah0
>> of=/dev/null bs=1M", where /phys/blah0 was created with "dd if=/dev/zero
>> of=/phys/blah0 bs=10M count=1024". It seems fine for a moment, but then
>> glusterfs starts growing and growing until it can't claim any more RAM
>> and becomes useless (can't ls the mountpoint, dd doesn't complete)...
>>
>> Both machines are running Ubuntu Edgy + Fuse 2.6.3.
>>
>> Thanks,
>>
>> Brent
>>
>> On Fri, 9 Mar 2007, Anand Avati wrote:
>>
>>> Brent,
>>> we are unable to get the memory leak in read-ahead (we found one and
>>> fixed, but your mail indicateds there is another). can you please mail
>>> your exact volume spec files and the command line to initiate the I/O
>>> which reproduces the leak?
>>>
>>> regards,
>>> avati
>>>
>>>
>>> On Fri, Mar 09, 2007 at 07:25:03PM -0500, Brent A Nelson wrote:
>>>> I just confirmed it on client-side, as well. glusterfs was stable for a
>>>> minute, and then it suddenly started increasing in size very rapidly.
>>>>
>>>> Thanks,
>>>>
>>>> Brent
>>>>
>>>
>
>> volume disk0
>> type storage/posix # POSIX FS translator
>> option directory /share0 # Export this directory
>> end-volume
>> volume disk1
>> type storage/posix # POSIX FS translator
>> option directory /share1 # Export this directory
>> end-volume
>> volume disk2
>> type storage/posix # POSIX FS translator
>> option directory /share2 # Export this directory
>> end-volume
>> volume disk3
>> type storage/posix # POSIX FS translator
>> option directory /share3 # Export this directory
>> end-volume
>> volume disk4
>> type storage/posix # POSIX FS translator
>> option directory /share4 # Export this directory
>> end-volume
>> volume disk5
>> type storage/posix # POSIX FS translator
>> option directory /share5 # Export this directory
>> end-volume
>> volume disk6
>> type storage/posix # POSIX FS translator
>> option directory /share6 # Export this directory
>> end-volume
>> volume disk7
>> type storage/posix # POSIX FS translator
>> option directory /share7 # Export this directory
>> end-volume
>> volume disk8
>> type storage/posix # POSIX FS translator
>> option directory /share8 # Export this directory
>> end-volume
>> volume disk9
>> type storage/posix # POSIX FS translator
>> option directory /share9 # Export this directory
>> end-volume
>> volume disk10
>> type storage/posix # POSIX FS translator
>> option directory /share10 # Export this directory
>> end-volume
>> volume disk11
>> type storage/posix # POSIX FS translator
>> option directory /share11 # Export this directory
>> end-volume
>> volume disk12
>> type storage/posix # POSIX FS translator
>> option directory /share12 # Export this directory
>> end-volume
>> volume disk13
>> type storage/posix # POSIX FS translator
>> option directory /share13 # Export this directory
>> end-volume
>> volume disk14
>> type storage/posix # POSIX FS translator
>> option directory /share14 # Export this directory
>> end-volume
>> volume disk15
>> type storage/posix # POSIX FS translator
>> option directory /share15 # Export this directory
>> end-volume
>>
>> volume share0
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk0
>> end-volume
>>
>> volume share1
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk1
>> end-volume
>>
>> volume share2
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk2
>> end-volume
>>
>> volume share3
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk3
>> end-volume
>>
>> volume share4
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk4
>> end-volume
>>
>> volume share5
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk5
>> end-volume
>>
>> volume share6
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk6
>> end-volume
>>
>> volume share7
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk7
>> end-volume
>>
>> volume share8
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk8
>> end-volume
>>
>> volume share9
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk9
>> end-volume
>>
>> volume share10
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk10
>> end-volume
>>
>> volume share11
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk11
>> end-volume
>>
>> volume share12
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk12
>> end-volume
>>
>> volume share13
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk13
>> end-volume
>>
>> volume share14
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk14
>> end-volume
>>
>> volume share15
>> type performance/io-threads
>> option thread-count 8
>> subvolumes disk15
>> end-volume
>>
>> volume server
>> type protocol/server
>> option transport-type tcp/server # For TCP/IP transport
>> # option transport-type ib-sdp/server # For Infiniband transport
>> # option bind-address 192.168.1.10 # Default is to listen on all interfaces
>> # option listen-port 6996 # Default is 6996
>> option client-volume-filename /etc/glusterfs/glusterfs-client.vol
>> subvolumes share0 share1 share2 share3 share4 share5 share6 share7 share8 share9 share10 share11 share12 share13 share14 share15
>> # NOTE: Access to any volume through protocol/server is denied by
>> # default. You need to explicitly grant access through # "auth"
>> # option.
>> option auth.ip.share0.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share1.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share2.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share3.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share4.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share5.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share6.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share7.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share8.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share9.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share10.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share11.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share12.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share13.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share14.allow 128.227.64.*,128.227.89.*
>> option auth.ip.share15.allow 128.227.64.*,128.227.89.*
>> end-volume
>
>> volume share0-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share0 # name of the remote volume
>> end-volume
>> volume share0-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share0 # name of the remote volume
>> end-volume
>> volume share1-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share1 # name of the remote volume
>> end-volume
>> volume share1-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share1 # name of the remote volume
>> end-volume
>> volume share2-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share2 # name of the remote volume
>> end-volume
>> volume share2-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share2 # name of the remote volume
>> end-volume
>> volume share3-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share3 # name of the remote volume
>> end-volume
>> volume share3-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share3 # name of the remote volume
>> end-volume
>> volume share4-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share4 # name of the remote volume
>> end-volume
>> volume share4-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share4 # name of the remote volume
>> end-volume
>> volume share5-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share5 # name of the remote volume
>> end-volume
>> volume share5-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share5 # name of the remote volume
>> end-volume
>> volume share6-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share6 # name of the remote volume
>> end-volume
>> volume share6-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share6 # name of the remote volume
>> end-volume
>> volume share7-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share7 # name of the remote volume
>> end-volume
>> volume share7-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share7 # name of the remote volume
>> end-volume
>> volume share8-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share8 # name of the remote volume
>> end-volume
>> volume share8-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share8 # name of the remote volume
>> end-volume
>> volume share9-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share9 # name of the remote volume
>> end-volume
>> volume share9-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share9 # name of the remote volume
>> end-volume
>> volume share10-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share10 # name of the remote volume
>> end-volume
>> volume share10-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share10 # name of the remote volume
>> end-volume
>> volume share11-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share11 # name of the remote volume
>> end-volume
>> volume share11-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share11 # name of the remote volume
>> end-volume
>> volume share12-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share12 # name of the remote volume
>> end-volume
>> volume share12-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share12 # name of the remote volume
>> end-volume
>> volume share13-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share13 # name of the remote volume
>> end-volume
>> volume share13-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share13 # name of the remote volume
>> end-volume
>> volume share14-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share14 # name of the remote volume
>> end-volume
>> volume share14-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share14 # name of the remote volume
>> end-volume
>> volume share15-0
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.241 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share15 # name of the remote volume
>> end-volume
>> volume share15-1
>> type protocol/client
>> option transport-type tcp/client # for TCP/IP transport
>> # option transport-type ib-sdp/client # for Infiniband transport
>> option remote-host 128.227.64.242 # IP address of the remote brick
>> # option remote-port 6996 # default server port is 6996
>> option remote-subvolume share15 # name of the remote volume
>> end-volume
>>
>> volume mirror0
>> type cluster/afr
>> subvolumes share0-0 share0-1
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror1
>> type cluster/afr
>> subvolumes share1-1 share1-0
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror2
>> type cluster/afr
>> subvolumes share2-0 share2-1
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror3
>> type cluster/afr
>> subvolumes share3-1 share3-0
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror4
>> type cluster/afr
>> subvolumes share4-0 share4-1
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror5
>> type cluster/afr
>> subvolumes share5-1 share5-0
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror6
>> type cluster/afr
>> subvolumes share6-0 share6-1
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror7
>> type cluster/afr
>> subvolumes share7-1 share7-0
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror8
>> type cluster/afr
>> subvolumes share8-0 share8-1
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror9
>> type cluster/afr
>> subvolumes share9-1 share9-0
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror10
>> type cluster/afr
>> subvolumes share10-0 share10-1
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror11
>> type cluster/afr
>> subvolumes share11-1 share11-0
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror12
>> type cluster/afr
>> subvolumes share12-0 share12-1
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror13
>> type cluster/afr
>> subvolumes share13-1 share13-0
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror14
>> type cluster/afr
>> subvolumes share14-0 share14-1
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror15
>> type cluster/afr
>> subvolumes share15-1 share15-0
>> option replicate *:2 # Do not leave space before or after "," and ":"
>> end-volume
>>
>> volume mirrors
>> type cluster/unify
>> subvolumes mirror0 mirror1 mirror2 mirror3 mirror4 mirror5 mirror6 mirror7 mirror8 mirror9 mirror10 mirror11 mirror12 mirror13 mirror14 mirror15
>> # option scheduler rr
>> option scheduler alu
>> option alu.order disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage
>> option alu.disk-usage.entry-threshold 2GB
>> option alu.disk-usage.exit-threshold 60MB
>> option alu.limits.min-free-disk 5GB
>> option alu.stat-refresh.interval 10sec
>> end-volume
>>
>> volume statprefetch
>> type performance/stat-prefetch
>> option cache-seconds 2
>> subvolumes mirrors
>> end-volume
>>
>> volume writebehind
>> type performance/write-behind
>> option aggregate-size 131072 # in bytes
>> subvolumes statprefetch
>> end-volume
>>
>> volume readahead
>> type performance/read-ahead
>> option page-size 65536 ### in bytes
>> option page-count 16 ### memory cache size is page-count x page-size per file
>> subvolumes writebehind
>> end-volume
>
>
> --
> Shaw's Principle:
> Build a system that even a fool can use,
> and only a fool will want to use it.
>
More information about the Gluster-devel
mailing list