[Gluster-devel] memory leaks

Brent A Nelson brent at phys.ufl.edu
Mon Mar 19 23:45:41 UTC 2007


I just wanted to confirm that, yes, this does indeed fix the problem.

Many Thanks!

Brent

On Sun, 18 Mar 2007, Anand Avati wrote:

> Brent,
>  The 'leak' has been plugged in the read-ahead translator of the
> latest tla patch. the problem was an integer overflow issue where the
> 'size' parameter for the flush function was 32bit but when the file
> was bigger than 4GB, the code tried to flush a region from 0 to >4GB.
> changing the function prototype to have a 64bit size fixed it.
>
> The 'leak' was actually not a 'leak' because when the file was closed,
> everything was flushed and free'd.
>
> thanks!
> avati
>
>
> On Mon, Mar 12, 2007 at 02:15:31PM -0400, Brent A Nelson wrote:
>> Attached are my present spec files for both servers (which are also my
>> clients); both machines use the same files.
>>
>> To trigger the memory leak, all I need to do is a "dd if=/phys/blah0
>> of=/dev/null bs=1M", where /phys/blah0 was created with "dd if=/dev/zero
>> of=/phys/blah0 bs=10M count=1024".  It seems fine for a moment, but then
>> glusterfs starts growing and growing until it can't claim any more RAM
>> and becomes useless (can't ls the mountpoint, dd doesn't complete)...
>>
>> Both machines are running Ubuntu Edgy + Fuse 2.6.3.
>>
>> Thanks,
>>
>> Brent
>>
>> On Fri, 9 Mar 2007, Anand Avati wrote:
>>
>>> Brent,
>>> we are unable to get the memory leak in read-ahead (we found one and
>>> fixed, but your mail indicateds there is another). can you please mail
>>> your exact volume spec files and the command line to initiate the I/O
>>> which reproduces the leak?
>>>
>>> regards,
>>> avati
>>>
>>>
>>> On Fri, Mar 09, 2007 at 07:25:03PM -0500, Brent A Nelson wrote:
>>>> I just confirmed it on client-side, as well.  glusterfs was stable for a
>>>> minute, and then it suddenly started increasing in size very rapidly.
>>>>
>>>> Thanks,
>>>>
>>>> Brent
>>>>
>>>
>
>> volume disk0
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share0             # Export this directory
>> end-volume
>> volume disk1
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share1             # Export this directory
>> end-volume
>> volume disk2
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share2             # Export this directory
>> end-volume
>> volume disk3
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share3             # Export this directory
>> end-volume
>> volume disk4
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share4             # Export this directory
>> end-volume
>> volume disk5
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share5             # Export this directory
>> end-volume
>> volume disk6
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share6             # Export this directory
>> end-volume
>> volume disk7
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share7             # Export this directory
>> end-volume
>> volume disk8
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share8             # Export this directory
>> end-volume
>> volume disk9
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share9             # Export this directory
>> end-volume
>> volume disk10
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share10            # Export this directory
>> end-volume
>> volume disk11
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share11            # Export this directory
>> end-volume
>> volume disk12
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share12            # Export this directory
>> end-volume
>> volume disk13
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share13            # Export this directory
>> end-volume
>> volume disk14
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share14            # Export this directory
>> end-volume
>> volume disk15
>>   type storage/posix                   # POSIX FS translator
>>   option directory /share15            # Export this directory
>> end-volume
>>
>> volume share0
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk0
>> end-volume
>>
>> volume share1
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk1
>> end-volume
>>
>> volume share2
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk2
>> end-volume
>>
>> volume share3
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk3
>> end-volume
>>
>> volume share4
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk4
>> end-volume
>>
>> volume share5
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk5
>> end-volume
>>
>> volume share6
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk6
>> end-volume
>>
>> volume share7
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk7
>> end-volume
>>
>> volume share8
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk8
>> end-volume
>>
>> volume share9
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk9
>> end-volume
>>
>> volume share10
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk10
>> end-volume
>>
>> volume share11
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk11
>> end-volume
>>
>> volume share12
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk12
>> end-volume
>>
>> volume share13
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk13
>> end-volume
>>
>> volume share14
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk14
>> end-volume
>>
>> volume share15
>>   type performance/io-threads
>>   option thread-count 8
>>   subvolumes disk15
>> end-volume
>>
>> volume server
>>   type protocol/server
>>   option transport-type tcp/server     # For TCP/IP transport
>> # option transport-type ib-sdp/server  # For Infiniband transport
>> # option bind-address 192.168.1.10     # Default is to listen on all interfaces
>> # option listen-port 6996               # Default is 6996
>>   option client-volume-filename /etc/glusterfs/glusterfs-client.vol
>>   subvolumes share0 share1 share2 share3 share4 share5 share6 share7 share8 share9 share10 share11 share12 share13 share14 share15
>> # NOTE: Access to any volume through protocol/server is denied by
>> # default. You need to explicitly grant access through # "auth"
>> # option.
>>   option auth.ip.share0.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share1.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share2.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share3.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share4.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share5.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share6.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share7.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share8.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share9.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share10.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share11.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share12.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share13.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share14.allow 128.227.64.*,128.227.89.*
>>   option auth.ip.share15.allow 128.227.64.*,128.227.89.*
>> end-volume
>
>> volume share0-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share0       # name of the remote volume
>> end-volume
>> volume share0-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share0       # name of the remote volume
>> end-volume
>> volume share1-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share1       # name of the remote volume
>> end-volume
>> volume share1-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share1       # name of the remote volume
>> end-volume
>> volume share2-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share2       # name of the remote volume
>> end-volume
>> volume share2-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share2       # name of the remote volume
>> end-volume
>> volume share3-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share3       # name of the remote volume
>> end-volume
>> volume share3-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share3       # name of the remote volume
>> end-volume
>> volume share4-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share4       # name of the remote volume
>> end-volume
>> volume share4-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share4       # name of the remote volume
>> end-volume
>> volume share5-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share5       # name of the remote volume
>> end-volume
>> volume share5-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share5       # name of the remote volume
>> end-volume
>> volume share6-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share6       # name of the remote volume
>> end-volume
>> volume share6-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share6       # name of the remote volume
>> end-volume
>> volume share7-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share7       # name of the remote volume
>> end-volume
>> volume share7-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share7       # name of the remote volume
>> end-volume
>> volume share8-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share8       # name of the remote volume
>> end-volume
>> volume share8-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share8       # name of the remote volume
>> end-volume
>> volume share9-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share9       # name of the remote volume
>> end-volume
>> volume share9-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share9       # name of the remote volume
>> end-volume
>> volume share10-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share10      # name of the remote volume
>> end-volume
>> volume share10-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share10      # name of the remote volume
>> end-volume
>> volume share11-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share11      # name of the remote volume
>> end-volume
>> volume share11-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share11      # name of the remote volume
>> end-volume
>> volume share12-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share12      # name of the remote volume
>> end-volume
>> volume share12-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share12      # name of the remote volume
>> end-volume
>> volume share13-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share13      # name of the remote volume
>> end-volume
>> volume share13-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share13      # name of the remote volume
>> end-volume
>> volume share14-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share14      # name of the remote volume
>> end-volume
>> volume share14-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share14      # name of the remote volume
>> end-volume
>> volume share15-0
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.241    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share15      # name of the remote volume
>> end-volume
>> volume share15-1
>>   type protocol/client
>>   option transport-type tcp/client     # for TCP/IP transport
>> # option transport-type ib-sdp/client  # for Infiniband transport
>>   option remote-host 128.227.64.242    # IP address of the remote brick
>> # option remote-port 6996              # default server port is 6996
>>   option remote-subvolume share15      # name of the remote volume
>> end-volume
>>
>> volume mirror0
>>   type cluster/afr
>>   subvolumes share0-0 share0-1
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror1
>>   type cluster/afr
>>   subvolumes share1-1 share1-0
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror2
>>   type cluster/afr
>>   subvolumes share2-0 share2-1
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror3
>>   type cluster/afr
>>   subvolumes share3-1 share3-0
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror4
>>   type cluster/afr
>>   subvolumes share4-0 share4-1
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror5
>>   type cluster/afr
>>   subvolumes share5-1 share5-0
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror6
>>   type cluster/afr
>>   subvolumes share6-0 share6-1
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror7
>>   type cluster/afr
>>   subvolumes share7-1 share7-0
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror8
>>   type cluster/afr
>>   subvolumes share8-0 share8-1
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror9
>>   type cluster/afr
>>   subvolumes share9-1 share9-0
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror10
>>   type cluster/afr
>>   subvolumes share10-0 share10-1
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror11
>>   type cluster/afr
>>   subvolumes share11-1 share11-0
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror12
>>   type cluster/afr
>>   subvolumes share12-0 share12-1
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror13
>>   type cluster/afr
>>   subvolumes share13-1 share13-0
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror14
>>   type cluster/afr
>>   subvolumes share14-0 share14-1
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>> volume mirror15
>>   type cluster/afr
>>   subvolumes share15-1 share15-0
>>   option replicate *:2     # Do not leave space before or after "," and ":"
>> end-volume
>>
>> volume mirrors
>>   type cluster/unify
>>   subvolumes mirror0 mirror1 mirror2 mirror3 mirror4 mirror5 mirror6 mirror7 mirror8 mirror9 mirror10 mirror11 mirror12 mirror13 mirror14 mirror15
>> #  option scheduler rr
>>   option scheduler alu
>>   option alu.order disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage
>>   option alu.disk-usage.entry-threshold 2GB
>>   option alu.disk-usage.exit-threshold 60MB
>>   option alu.limits.min-free-disk 5GB
>>   option alu.stat-refresh.interval 10sec
>> end-volume
>>
>> volume statprefetch
>>   type performance/stat-prefetch
>>   option cache-seconds 2
>>   subvolumes mirrors
>> end-volume
>>
>> volume writebehind
>>   type performance/write-behind
>>   option aggregate-size 131072 # in bytes
>>   subvolumes statprefetch
>> end-volume
>>
>> volume readahead
>>   type performance/read-ahead
>>   option page-size 65536 ### in bytes
>>   option page-count 16 ### memory cache size is page-count x page-size per file
>>   subvolumes writebehind
>> end-volume
>
>
> -- 
> Shaw's Principle:
>        Build a system that even a fool can use,
>        and only a fool will want to use it.
>





More information about the Gluster-devel mailing list