[Gluster-users] Rsync

Hiren Joshi josh at moonfruit.com
Wed Sep 23 13:01:49 UTC 2009


Bellow.

I've found that I get a performance hit if I add read cache or
whitebehind.

Server conf:

##Open vols

volume posix1
  type storage/posix
  option directory /gluster/export1
end-volume

volume posix2
  type storage/posix
  option directory /gluster/export2
end-volume

volume posix3
  type storage/posix
  option directory /gluster/export3
end-volume

volume posix4
  type storage/posix
  option directory /gluster/export4
end-volume

volume posix5
  type storage/posix
  option directory /gluster/export5
end-volume

volume posix6
  type storage/posix
  option directory /gluster/export6
end-volume
## Add the ability to lock files etc

volume locks1
  type features/locks
  subvolumes posix1
end-volume

volume locks2
  type features/locks
  subvolumes posix2
end-volume

volume locks3
  type features/locks
  subvolumes posix3
end-volume

volume locks4
  type features/locks
  subvolumes posix4
end-volume

volume locks5
  type features/locks
  subvolumes posix5
end-volume

volume locks6
  type features/locks
  subvolumes posix6
end-volume
## Preformance translators

volume brick1
  type performance/io-threads
  option thread-count 8
  subvolumes locks1
end-volume

volume brick2
  type performance/io-threads
  option thread-count 8
  subvolumes locks2
end-volume

volume brick3
  type performance/io-threads
  option thread-count 8
  subvolumes locks3
end-volume

volume brick4
  type performance/io-threads
  option thread-count 8
  subvolumes locks4
end-volume

volume brick5
  type performance/io-threads
  option thread-count 8
  subvolumes locks5
end-volume

volume brick6
  type performance/io-threads
  option thread-count 8
  subvolumes locks6
end-volume
##export the lot

#volume brick6
#  type debug/trace
#  subvolumes trace_brick6
#end-volume

volume server
  type protocol/server
  option transport-type tcp/server
  option auth.addr.brick1.allow *
  option auth.addr.brick2.allow *
  option auth.addr.brick3.allow *
  option auth.addr.brick4.allow *
  option auth.addr.brick5.allow *
  option auth.addr.brick6.allow *
  subvolumes brick1 brick2 brick3 brick4 brick5 brick6
end-volume 


Vol file:
##Clent config
##import all the briks from all the mirrors and mirror them

volume glust1a_1
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1a
  option ping-timeout 30
  option remote-subvolume brick1
end-volume

volume glust1b_1
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1b
  option ping-timeout 30
  option remote-subvolume brick1
end-volume

volume mirror1_1
  type cluster/replicate
  subvolumes glust1a_1 glust1b_1
end-volume

volume glust1a_2
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1a
  option ping-timeout 30
  option remote-subvolume brick2
end-volume

volume glust1b_2
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1b
  option ping-timeout 30
  option remote-subvolume brick2
end-volume

volume mirror1_2
  type cluster/replicate
  subvolumes glust1a_2 glust1b_2
end-volume

volume glust1a_3
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1a
  option ping-timeout 30
  option remote-subvolume brick3
end-volume

volume glust1b_3
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1b
  option ping-timeout 30
  option remote-subvolume brick3
end-volume

volume mirror1_3
  type cluster/replicate
  subvolumes glust1a_3 glust1b_3
end-volume

volume glust1a_4
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1a
  option ping-timeout 30
  option remote-subvolume brick4
end-volume

volume glust1b_4
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1b
  option ping-timeout 30
  option remote-subvolume brick4
end-volume

volume mirror1_4
  type cluster/replicate
  subvolumes glust1a_4 glust1b_4
end-volume

volume glust1a_5
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1a
  option ping-timeout 30
  option remote-subvolume brick5
end-volume

volume glust1b_5
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1b
  option ping-timeout 30
  option remote-subvolume brick5
end-volume

volume mirror1_5
  type cluster/replicate
  subvolumes glust1a_5 glust1b_5
end-volume

volume glust1a_6
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1a
  option ping-timeout 30
  option remote-subvolume brick6
end-volume

volume glust1b_6
  type protocol/client
  option transport-type tcp/client
  option remote-host glust1b
  option ping-timeout 30
  option remote-subvolume brick6
end-volume

volume mirror1_6
  type cluster/replicate
  subvolumes glust1a_6 glust1b_6
end-volume

##place it all into a hash

volume dist
  type cluster/distribute
  subvolumes mirror1_1 mirror1_2 mirror1_3 mirror1_4 mirror1_5 mirror1_6
end-volume


> -----Original Message-----
> From: Pavan Vilas Sondur [mailto:pavan at gluster.com] 
> Sent: 23 September 2009 13:38
> To: Hiren Joshi
> Cc: gluster-users at gluster.org
> Subject: Re: Rsync
> 
> Can you send us the volfiles and logfiles as well?
> 
> Pavan
> 
> On 23/09/09 10:08 +0100, Hiren Joshi wrote:
> > 2.0.6-1.el5, it's a 6 brick distributed system which is 
> mirrored over 2
> > machines (each brick has a mirror on another machine, these 
> mirrors are
> > put in the hash).
> > 
> > An update, after running the rsync for a day, I killed it 
> and remounted
> > all the disks (the underlying filesystem, not the gluster) 
> with noatime,
> > the rsync completed in about 600 minutes. I'm now going to 
> try one level
> > up (about 1,000,000,000 dirs).
> > 
> > > -----Original Message-----
> > > From: Pavan Vilas Sondur [mailto:pavan at gluster.com] 
> > > Sent: 23 September 2009 07:55
> > > To: Hiren Joshi
> > > Cc: gluster-users at gluster.org
> > > Subject: Re: Rsync
> > > 
> > > Hi Hiren,
> > > What glusterfs version are you using? Can you send us the 
> > > volfiles and the log files.
> > > 
> > > Pavan
> > > 
> > > On 22/09/09 16:01 +0100, Hiren Joshi wrote:
> > > > I forgot to mention, the mount is mounted with 
> direct-io, would this
> > > > make a difference? 
> > > > 
> > > > > -----Original Message-----
> > > > > From: gluster-users-bounces at gluster.org 
> > > > > [mailto:gluster-users-bounces at gluster.org] On Behalf Of 
> > > Hiren Joshi
> > > > > Sent: 22 September 2009 11:40
> > > > > To: gluster-users at gluster.org
> > > > > Subject: [Gluster-users] Rsync
> > > > > 
> > > > > Hello all,
> > > > >  
> > > > > I'm getting what I think is bizarre behaviour.... I have 
> > > about 400G to
> > > > > rsync (rsync -av) onto a gluster share, the data is 
> in a directory
> > > > > structure which has about 1000 directories per parent and 
> > > about 1000
> > > > > directories in each of them.
> > > > >  
> > > > > When I try to rsync an end leaf directory (this has about 4 
> > > > > dirs and 100
> > > > > files in each) the operation takes about 10 seconds. When I 
> > > > > go one level
> > > > > above (1000 dirs with about 4 dirs in each with about 100 
> > > > > files in each)
> > > > > the operation takes about 10 minutes.
> > > > >  
> > > > > Now, if I then go one level above that (that's 1000 dirs with 
> > > > > 1000 dirs
> > > > > in each with about 4 dirs in each with about 100 files in 
> > > each) the
> > > > > operation takes days! Top shows glusterfsd takes 300-600% 
> > > cpu usage
> > > > > (2X4core), I have about 48G of memory (usage is 0% as 
> expected).
> > > > >  
> > > > > Has anyone seen anything like this? How can I speed it up?
> > > > >  
> > > > > Thanks,
> > > > >  
> > > > > Josh.
> > > > > 
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> > > 
> 



More information about the Gluster-users mailing list