[Gluster-devel] Potential Memory Leak?

[Karl Bernard]

Hello Krishna,

I have 5 servers running the client and 4 servers running the brick 
server. In the config I was testing, only 3 of the brick servers are used.

I have scripts running on the 5 servers that open images of 5k to 20k 
and create thumbnails for those images of about 4k. All files are 
written in a hash directory structure.

After reading and creating a lot of files (1 million for example),  I 
can see that the memory usage for the glusterfsd have grown substancially.

Software versions:
glusterfs-1.3.4
fuse-2.7.0-glfs4

<<-- glusterfs-server.vol -->>
volume brick-posix
        type storage/posix
        option directory /data/glusterfs/dataspace
end-volume

volume brick-ns
        type storage/posix
        option directory /data/glusterfs/namespace
end-volume

volume brick
  type performance/io-threads
  option thread-count 2
  option cache-size 32MB
  subvolumes brick-posix
end-volume

volume server
        type protocol/server
        option transport-type tcp/server
        subvolumes brick brick-ns
        option auth.ip.brick.allow 172.16.93.*
        option auth.ip.brick-ns.allow 172.16.93.*
end-volume
<<-- end of glusterfs-server.vol -->>

<<-- start client.sharedbig.vol -->>
volume sxx01-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host sxx01b
 option remote-subvolume brick-ns
end-volume

volume sxx02-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host sxx02b
 option remote-subvolume brick-ns
end-volume

volume sxx03-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host sxx03b
 option remote-subvolume brick-ns
end-volume

volume sxx04-ns
 type protocol/client
 option transport-type tcp/client
 option remote-host sxx04b
 option remote-subvolume brick-ns
end-volume

volume sxx01
 type protocol/client
 option transport-type tcp/client
 option remote-host sxx01b
 option remote-subvolume brick
end-volume

volume sxx02
 type protocol/client
 option transport-type tcp/client
 option remote-host sxx02b
 option remote-subvolume brick
end-volume

volume sxx03
 type protocol/client
 option transport-type tcp/client
 option remote-host sxx03b
 option remote-subvolume brick
end-volume

volume sxx04
 type protocol/client
 option transport-type tcp/client
 option remote-host sxx04b
 option remote-subvolume brick
end-volume

volume afr3-4
  type cluster/afr
  subvolumes sxx03 sxx04
  option replicate *:2 
end-volume

volume afr2-4ns
  type cluster/afr
  subvolumes sxx02-ns sxx04-ns
  option replicate *:2          
end-volume

volume unify                    
  type cluster/unify
  subvolumes afr3-4
  option namespace afr2-4ns
  option scheduler rr
end-volume

## Add writebehind feature      
volume writebehind
  type performance/write-behind
  option aggregate-size 128kB
  subvolumes unify
end-volume

## Add readahead feature
volume readahead
  type performance/read-ahead
  option page-size 256kB     #
  option page-count 16       # cache per file  = (page-count x page-size)
  subvolumes writebehind
end-volume

<< -- end of client.sharedbig.vol -->>


Krishna Srinivas wrote:
> Hi Karl,
>
> Do you see the memory usage go up everytime you run the script?
>
> Can you give us the config details, spec files and script? That will
> help us to find out where the leak might be happening.
>
> Thanks
> Krishna
>
> On 10/21/07, Karl Bernard <karl at vomba.com> wrote:
>   
>> Hello,
>>
>> I've been testing luster in a development environment with 4 servers
>> acting as both client and servers.
>>
>> In the last 2 days, I've run maintenance script that perform a lot of
>> read and write operations.
>>
>> I've noticed that the glusterfsd was using 57mb of memory when initially
>> started and the memory usage grew to using 85% of server memory on one
>> of the host:
>>
>>   PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+
>> COMMAND
>> 23219 root      17   0 1027m 861m  756 D    2 85.3 315:29.43 glusterfsd
>>
>> I also saw the performance slowly degrade and saw a huge jump in speed
>> after I restarted the deamon on all the bricks.
>>
>> I'm unsure what to monitor to help debug the potential problem. Is there
>> something I can do to help improve gluster and fix that problem?
>>
>> Karl
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>>