[Gluster-users] Memory usage high on server sides
Raghavendra Bhat
raghavendrabhat at gluster.com
Thu Apr 22 04:38:11 UTC 2010
Hi Chris,
http://patches.gluster.com/patch/3151/
Can you please apply this patch and see if this works for you?
Thanks
Regards,
Raghavendra Bhat
> Tejas,
>
> We still have hundreds of GBs to copy, and have not put the new file
> system into the test. So far the clients works all fine. I mean the
> commands like ls, mkdir, touch, and etc.
>
> Thanks again for your time.
>
> regards,
>
> Chris
>
> On Wed, 2010-04-14 at 23:04 -0600, Tejas N. Bhise wrote:
> > Chris,
> >
> > By the way, after the copy is done, how is the system responding to
> > regular access ? In the sense, was the problem with copy also
> > carried forward as more trouble seen with subsequent access of
> > data over glusterfs ?
> >
> > Regards,
> > Tejas.
> >
> > ----- Original Message -----
> > From: "Chris Jin" <chris at pikicentral.com>
> > To: "Tejas N. Bhise" <tejas at gluster.com>
> > Cc: "gluster-users" <gluster-users at gluster.org>
> > Sent: Thursday, April 15, 2010 9:48:42 AM
> > Subject: Re: [Gluster-users] Memory usage high on server sides
> >
> > Hi Tejas,
> >
> > > Problems you saw -
> > >
> > > 1) High memory usage on client where gluster volume is mounted
> >
> > Memory usage for clients is 0% after copying.
> > $ps auxf
> > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
> COMMAND
> > root 19692 1.3 0.0 262148 6980 ? Ssl Apr12
> > 61:33 /sbin/glusterfs --log-level=NORMAL
> > --volfile=/u2/git/modules/shared/glusterfs/clients/r2/c2.vol
> /gfs/r2/f2
> >
> > > 2) High memory usage on server
> > Yes.
> > $ps auxf
> > USER PID %CPU %MEM VSZ RSS TTY STAT START TIME
> COMMAND
> > root 26472 2.2 29.1 718100 600260 ? Ssl Apr09 184:09
> > glusterfsd -f /etc/glusterfs/servers/r2/f1.vol
> > root 26485 1.8 39.8 887744 821384 ? Ssl Apr09 157:16
> > glusterfsd -f /etc/glusterfs/servers/r2/f2.vol
> >
> > > 3) 2 days to copy 300 GB data
> > More than 700GB. There are two folders. The first one is copied to
> > server 1 and server 2, and the second one is copied to server 2 and
> > server 3. The vol files are below.
> >
> > > About the config, can you provide the following for both old and
> new systems -
> > >
> > > 1) OS and kernel level on gluster servers and clients
> > Debian Kernel 2.6.18-6-amd64
> >
> > $uname -a
> > Linux fs2 2.6.18-6-amd64 #1 SMP Tue Aug 19 04:30:56 UTC 2008 x86_64
> > GNU/Linux
> >
> > > 2) volume file from servers and clients
> >
> > #####Server Vol file (f1.vol)
> > # The same settings for f2.vol and f3.vol, just different dirs and
> ports
> > # f1 f3 for Server 1, f1 f2 for Server 2, f2 f3 for Server 3
> > volume posix1
> > type storage/posix
> > option directory /gfs/r2/f1
> > end-volume
> >
> > volume locks1
> > type features/locks
> > subvolumes posix1
> > end-volume
> >
> > volume brick1
> > type performance/io-threads
> > option thread-count 8
> > subvolumes locks1
> > end-volume
> >
> > volume server-tcp
> > type protocol/server
> > option transport-type tcp
> > option auth.addr.brick1.allow 192.168.0.*
> > option transport.socket.listen-port 6991
> > option transport.socket.nodelay on
> > subvolumes brick1
> > end-volume
> >
> > #####Client Vol file (c1.vol)
> > # The same settings for c2.vol and c3.vol
> > # s2 s3 for c2, s3 s1 for c3
> > volume s1
> > type protocol/client
> > option transport-type tcp
> > option remote-host 192.168.0.31
> > option transport.socket.nodelay on
> > option transport.remote-port 6991
> > option remote-subvolume brick1
> > end-volume
> >
> > volume s2
> > type protocol/client
> > option transport-type tcp
> > option remote-host 192.168.0.32
> > option transport.socket.nodelay on
> > option transport.remote-port 6991
> > option remote-subvolume brick1
> > end-volume
> >
> > volume mirror
> > type cluster/replicate
> > option data-self-heal off
> > option metadata-self-heal off
> > option entry-self-heal off
> > subvolumes s1 s2
> > end-volume
> >
> > volume writebehind
> > type performance/write-behind
> > option cache-size 100MB
> > option flush-behind off
> > subvolumes mirror
> > end-volume
> >
> > volume iocache
> > type performance/io-cache
> > option cache-size `grep 'MemTotal' /proc/meminfo | awk '{print
> $2 *
> > 0.2 / 1024}' | cut -f1 -d.`MB
> > option cache-timeout 1
> > subvolumes writebehind
> > end-volume
> >
> > volume quickread
> > type performance/quick-read
> > option cache-timeout 1
> > option max-file-size 256Kb
> > subvolumes iocache
> > end-volume
> >
> > volume statprefetch
> > type performance/stat-prefetch
> > subvolumes quickread
> > end-volume
> >
> >
> > > 3) Filesystem type of backend gluster subvolumes
> > ext3
> >
> > > 4) How close to full the backend subvolumes are
> > New 2T hard disks for each server.
> >
> > > 5) The exact copy command .. did you mount the volumes from
> > > old and new system on a single machine and did cp or used rsync
> > > or some other method ? If something more than just a cp, please
> > > send the exact command line you used.
> > The old file system uses DRBD and NFS.
> > The exact command is
> > sudo cp -R -v -p -P /nfsmounts/nfs3/photo .
> >
> > > 6) How many files/directories ( tentative ) in that 300GB data (
> would help in
> > > trying to reproduce inhouse with a smaller test bed ).
> > I cannot tell, but the file sizes are between 1KB to 200KB, average
> > around 20KB.
> >
> > > 7) Was there other load on the new or old system ?
> > The old systems are still used for web servers.
> > The new systems are on the same servers but different hard disks.
> >
> > > 8) Any other patterns you noticed.
> > There is once that one client tried to connect one server with
> external
> > IP address.
> > Using distribute translator across all three mirrors will make
> system
> > twice slower than using three mounted folders.
> >
> > Is this information enough?
> >
> > Please take a look.
> >
> > Regards,
> >
> > Chris
> >
> >
> >
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list