[Gluster-users] Rsync
Hiren Joshi
josh at moonfruit.com
Wed Sep 23 13:01:49 UTC 2009
Bellow.
I've found that I get a performance hit if I add read cache or
whitebehind.
Server conf:
##Open vols
volume posix1
type storage/posix
option directory /gluster/export1
end-volume
volume posix2
type storage/posix
option directory /gluster/export2
end-volume
volume posix3
type storage/posix
option directory /gluster/export3
end-volume
volume posix4
type storage/posix
option directory /gluster/export4
end-volume
volume posix5
type storage/posix
option directory /gluster/export5
end-volume
volume posix6
type storage/posix
option directory /gluster/export6
end-volume
## Add the ability to lock files etc
volume locks1
type features/locks
subvolumes posix1
end-volume
volume locks2
type features/locks
subvolumes posix2
end-volume
volume locks3
type features/locks
subvolumes posix3
end-volume
volume locks4
type features/locks
subvolumes posix4
end-volume
volume locks5
type features/locks
subvolumes posix5
end-volume
volume locks6
type features/locks
subvolumes posix6
end-volume
## Preformance translators
volume brick1
type performance/io-threads
option thread-count 8
subvolumes locks1
end-volume
volume brick2
type performance/io-threads
option thread-count 8
subvolumes locks2
end-volume
volume brick3
type performance/io-threads
option thread-count 8
subvolumes locks3
end-volume
volume brick4
type performance/io-threads
option thread-count 8
subvolumes locks4
end-volume
volume brick5
type performance/io-threads
option thread-count 8
subvolumes locks5
end-volume
volume brick6
type performance/io-threads
option thread-count 8
subvolumes locks6
end-volume
##export the lot
#volume brick6
# type debug/trace
# subvolumes trace_brick6
#end-volume
volume server
type protocol/server
option transport-type tcp/server
option auth.addr.brick1.allow *
option auth.addr.brick2.allow *
option auth.addr.brick3.allow *
option auth.addr.brick4.allow *
option auth.addr.brick5.allow *
option auth.addr.brick6.allow *
subvolumes brick1 brick2 brick3 brick4 brick5 brick6
end-volume
Vol file:
##Clent config
##import all the briks from all the mirrors and mirror them
volume glust1a_1
type protocol/client
option transport-type tcp/client
option remote-host glust1a
option ping-timeout 30
option remote-subvolume brick1
end-volume
volume glust1b_1
type protocol/client
option transport-type tcp/client
option remote-host glust1b
option ping-timeout 30
option remote-subvolume brick1
end-volume
volume mirror1_1
type cluster/replicate
subvolumes glust1a_1 glust1b_1
end-volume
volume glust1a_2
type protocol/client
option transport-type tcp/client
option remote-host glust1a
option ping-timeout 30
option remote-subvolume brick2
end-volume
volume glust1b_2
type protocol/client
option transport-type tcp/client
option remote-host glust1b
option ping-timeout 30
option remote-subvolume brick2
end-volume
volume mirror1_2
type cluster/replicate
subvolumes glust1a_2 glust1b_2
end-volume
volume glust1a_3
type protocol/client
option transport-type tcp/client
option remote-host glust1a
option ping-timeout 30
option remote-subvolume brick3
end-volume
volume glust1b_3
type protocol/client
option transport-type tcp/client
option remote-host glust1b
option ping-timeout 30
option remote-subvolume brick3
end-volume
volume mirror1_3
type cluster/replicate
subvolumes glust1a_3 glust1b_3
end-volume
volume glust1a_4
type protocol/client
option transport-type tcp/client
option remote-host glust1a
option ping-timeout 30
option remote-subvolume brick4
end-volume
volume glust1b_4
type protocol/client
option transport-type tcp/client
option remote-host glust1b
option ping-timeout 30
option remote-subvolume brick4
end-volume
volume mirror1_4
type cluster/replicate
subvolumes glust1a_4 glust1b_4
end-volume
volume glust1a_5
type protocol/client
option transport-type tcp/client
option remote-host glust1a
option ping-timeout 30
option remote-subvolume brick5
end-volume
volume glust1b_5
type protocol/client
option transport-type tcp/client
option remote-host glust1b
option ping-timeout 30
option remote-subvolume brick5
end-volume
volume mirror1_5
type cluster/replicate
subvolumes glust1a_5 glust1b_5
end-volume
volume glust1a_6
type protocol/client
option transport-type tcp/client
option remote-host glust1a
option ping-timeout 30
option remote-subvolume brick6
end-volume
volume glust1b_6
type protocol/client
option transport-type tcp/client
option remote-host glust1b
option ping-timeout 30
option remote-subvolume brick6
end-volume
volume mirror1_6
type cluster/replicate
subvolumes glust1a_6 glust1b_6
end-volume
##place it all into a hash
volume dist
type cluster/distribute
subvolumes mirror1_1 mirror1_2 mirror1_3 mirror1_4 mirror1_5 mirror1_6
end-volume
> -----Original Message-----
> From: Pavan Vilas Sondur [mailto:pavan at gluster.com]
> Sent: 23 September 2009 13:38
> To: Hiren Joshi
> Cc: gluster-users at gluster.org
> Subject: Re: Rsync
>
> Can you send us the volfiles and logfiles as well?
>
> Pavan
>
> On 23/09/09 10:08 +0100, Hiren Joshi wrote:
> > 2.0.6-1.el5, it's a 6 brick distributed system which is
> mirrored over 2
> > machines (each brick has a mirror on another machine, these
> mirrors are
> > put in the hash).
> >
> > An update, after running the rsync for a day, I killed it
> and remounted
> > all the disks (the underlying filesystem, not the gluster)
> with noatime,
> > the rsync completed in about 600 minutes. I'm now going to
> try one level
> > up (about 1,000,000,000 dirs).
> >
> > > -----Original Message-----
> > > From: Pavan Vilas Sondur [mailto:pavan at gluster.com]
> > > Sent: 23 September 2009 07:55
> > > To: Hiren Joshi
> > > Cc: gluster-users at gluster.org
> > > Subject: Re: Rsync
> > >
> > > Hi Hiren,
> > > What glusterfs version are you using? Can you send us the
> > > volfiles and the log files.
> > >
> > > Pavan
> > >
> > > On 22/09/09 16:01 +0100, Hiren Joshi wrote:
> > > > I forgot to mention, the mount is mounted with
> direct-io, would this
> > > > make a difference?
> > > >
> > > > > -----Original Message-----
> > > > > From: gluster-users-bounces at gluster.org
> > > > > [mailto:gluster-users-bounces at gluster.org] On Behalf Of
> > > Hiren Joshi
> > > > > Sent: 22 September 2009 11:40
> > > > > To: gluster-users at gluster.org
> > > > > Subject: [Gluster-users] Rsync
> > > > >
> > > > > Hello all,
> > > > >
> > > > > I'm getting what I think is bizarre behaviour.... I have
> > > about 400G to
> > > > > rsync (rsync -av) onto a gluster share, the data is
> in a directory
> > > > > structure which has about 1000 directories per parent and
> > > about 1000
> > > > > directories in each of them.
> > > > >
> > > > > When I try to rsync an end leaf directory (this has about 4
> > > > > dirs and 100
> > > > > files in each) the operation takes about 10 seconds. When I
> > > > > go one level
> > > > > above (1000 dirs with about 4 dirs in each with about 100
> > > > > files in each)
> > > > > the operation takes about 10 minutes.
> > > > >
> > > > > Now, if I then go one level above that (that's 1000 dirs with
> > > > > 1000 dirs
> > > > > in each with about 4 dirs in each with about 100 files in
> > > each) the
> > > > > operation takes days! Top shows glusterfsd takes 300-600%
> > > cpu usage
> > > > > (2X4core), I have about 48G of memory (usage is 0% as
> expected).
> > > > >
> > > > > Has anyone seen anything like this? How can I speed it up?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Josh.
> > > > >
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> > >
>
More information about the Gluster-users
mailing list