[Gluster-users] Rsync
Hiren Joshi
josh at moonfruit.com
Wed Sep 23 16:16:56 UTC 2009
Also, there's nothing happening in the log files beyond the initial
mount so it looks like the filesystem aspect of it is fine.
> -----Original Message-----
> From: gluster-users-bounces at gluster.org
> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Hiren Joshi
> Sent: 23 September 2009 14:02
> To: Pavan Vilas Sondur
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] Rsync
>
> Bellow.
>
> I've found that I get a performance hit if I add read cache or
> whitebehind.
>
> Server conf:
>
> ##Open vols
>
> volume posix1
> type storage/posix
> option directory /gluster/export1
> end-volume
>
> volume posix2
> type storage/posix
> option directory /gluster/export2
> end-volume
>
> volume posix3
> type storage/posix
> option directory /gluster/export3
> end-volume
>
> volume posix4
> type storage/posix
> option directory /gluster/export4
> end-volume
>
> volume posix5
> type storage/posix
> option directory /gluster/export5
> end-volume
>
> volume posix6
> type storage/posix
> option directory /gluster/export6
> end-volume
> ## Add the ability to lock files etc
>
> volume locks1
> type features/locks
> subvolumes posix1
> end-volume
>
> volume locks2
> type features/locks
> subvolumes posix2
> end-volume
>
> volume locks3
> type features/locks
> subvolumes posix3
> end-volume
>
> volume locks4
> type features/locks
> subvolumes posix4
> end-volume
>
> volume locks5
> type features/locks
> subvolumes posix5
> end-volume
>
> volume locks6
> type features/locks
> subvolumes posix6
> end-volume
> ## Preformance translators
>
> volume brick1
> type performance/io-threads
> option thread-count 8
> subvolumes locks1
> end-volume
>
> volume brick2
> type performance/io-threads
> option thread-count 8
> subvolumes locks2
> end-volume
>
> volume brick3
> type performance/io-threads
> option thread-count 8
> subvolumes locks3
> end-volume
>
> volume brick4
> type performance/io-threads
> option thread-count 8
> subvolumes locks4
> end-volume
>
> volume brick5
> type performance/io-threads
> option thread-count 8
> subvolumes locks5
> end-volume
>
> volume brick6
> type performance/io-threads
> option thread-count 8
> subvolumes locks6
> end-volume
> ##export the lot
>
> #volume brick6
> # type debug/trace
> # subvolumes trace_brick6
> #end-volume
>
> volume server
> type protocol/server
> option transport-type tcp/server
> option auth.addr.brick1.allow *
> option auth.addr.brick2.allow *
> option auth.addr.brick3.allow *
> option auth.addr.brick4.allow *
> option auth.addr.brick5.allow *
> option auth.addr.brick6.allow *
> subvolumes brick1 brick2 brick3 brick4 brick5 brick6
> end-volume
>
>
> Vol file:
> ##Clent config
> ##import all the briks from all the mirrors and mirror them
>
> volume glust1a_1
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1a
> option ping-timeout 30
> option remote-subvolume brick1
> end-volume
>
> volume glust1b_1
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1b
> option ping-timeout 30
> option remote-subvolume brick1
> end-volume
>
> volume mirror1_1
> type cluster/replicate
> subvolumes glust1a_1 glust1b_1
> end-volume
>
> volume glust1a_2
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1a
> option ping-timeout 30
> option remote-subvolume brick2
> end-volume
>
> volume glust1b_2
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1b
> option ping-timeout 30
> option remote-subvolume brick2
> end-volume
>
> volume mirror1_2
> type cluster/replicate
> subvolumes glust1a_2 glust1b_2
> end-volume
>
> volume glust1a_3
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1a
> option ping-timeout 30
> option remote-subvolume brick3
> end-volume
>
> volume glust1b_3
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1b
> option ping-timeout 30
> option remote-subvolume brick3
> end-volume
>
> volume mirror1_3
> type cluster/replicate
> subvolumes glust1a_3 glust1b_3
> end-volume
>
> volume glust1a_4
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1a
> option ping-timeout 30
> option remote-subvolume brick4
> end-volume
>
> volume glust1b_4
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1b
> option ping-timeout 30
> option remote-subvolume brick4
> end-volume
>
> volume mirror1_4
> type cluster/replicate
> subvolumes glust1a_4 glust1b_4
> end-volume
>
> volume glust1a_5
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1a
> option ping-timeout 30
> option remote-subvolume brick5
> end-volume
>
> volume glust1b_5
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1b
> option ping-timeout 30
> option remote-subvolume brick5
> end-volume
>
> volume mirror1_5
> type cluster/replicate
> subvolumes glust1a_5 glust1b_5
> end-volume
>
> volume glust1a_6
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1a
> option ping-timeout 30
> option remote-subvolume brick6
> end-volume
>
> volume glust1b_6
> type protocol/client
> option transport-type tcp/client
> option remote-host glust1b
> option ping-timeout 30
> option remote-subvolume brick6
> end-volume
>
> volume mirror1_6
> type cluster/replicate
> subvolumes glust1a_6 glust1b_6
> end-volume
>
> ##place it all into a hash
>
> volume dist
> type cluster/distribute
> subvolumes mirror1_1 mirror1_2 mirror1_3 mirror1_4
> mirror1_5 mirror1_6
> end-volume
>
>
> > -----Original Message-----
> > From: Pavan Vilas Sondur [mailto:pavan at gluster.com]
> > Sent: 23 September 2009 13:38
> > To: Hiren Joshi
> > Cc: gluster-users at gluster.org
> > Subject: Re: Rsync
> >
> > Can you send us the volfiles and logfiles as well?
> >
> > Pavan
> >
> > On 23/09/09 10:08 +0100, Hiren Joshi wrote:
> > > 2.0.6-1.el5, it's a 6 brick distributed system which is
> > mirrored over 2
> > > machines (each brick has a mirror on another machine, these
> > mirrors are
> > > put in the hash).
> > >
> > > An update, after running the rsync for a day, I killed it
> > and remounted
> > > all the disks (the underlying filesystem, not the gluster)
> > with noatime,
> > > the rsync completed in about 600 minutes. I'm now going to
> > try one level
> > > up (about 1,000,000,000 dirs).
> > >
> > > > -----Original Message-----
> > > > From: Pavan Vilas Sondur [mailto:pavan at gluster.com]
> > > > Sent: 23 September 2009 07:55
> > > > To: Hiren Joshi
> > > > Cc: gluster-users at gluster.org
> > > > Subject: Re: Rsync
> > > >
> > > > Hi Hiren,
> > > > What glusterfs version are you using? Can you send us the
> > > > volfiles and the log files.
> > > >
> > > > Pavan
> > > >
> > > > On 22/09/09 16:01 +0100, Hiren Joshi wrote:
> > > > > I forgot to mention, the mount is mounted with
> > direct-io, would this
> > > > > make a difference?
> > > > >
> > > > > > -----Original Message-----
> > > > > > From: gluster-users-bounces at gluster.org
> > > > > > [mailto:gluster-users-bounces at gluster.org] On Behalf Of
> > > > Hiren Joshi
> > > > > > Sent: 22 September 2009 11:40
> > > > > > To: gluster-users at gluster.org
> > > > > > Subject: [Gluster-users] Rsync
> > > > > >
> > > > > > Hello all,
> > > > > >
> > > > > > I'm getting what I think is bizarre behaviour.... I have
> > > > about 400G to
> > > > > > rsync (rsync -av) onto a gluster share, the data is
> > in a directory
> > > > > > structure which has about 1000 directories per parent and
> > > > about 1000
> > > > > > directories in each of them.
> > > > > >
> > > > > > When I try to rsync an end leaf directory (this has about 4
> > > > > > dirs and 100
> > > > > > files in each) the operation takes about 10 seconds. When I
> > > > > > go one level
> > > > > > above (1000 dirs with about 4 dirs in each with about 100
> > > > > > files in each)
> > > > > > the operation takes about 10 minutes.
> > > > > >
> > > > > > Now, if I then go one level above that (that's 1000
> dirs with
> > > > > > 1000 dirs
> > > > > > in each with about 4 dirs in each with about 100 files in
> > > > each) the
> > > > > > operation takes days! Top shows glusterfsd takes 300-600%
> > > > cpu usage
> > > > > > (2X4core), I have about 48G of memory (usage is 0% as
> > expected).
> > > > > >
> > > > > > Has anyone seen anything like this? How can I speed it up?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > Josh.
> > > > > >
> > > > > _______________________________________________
> > > > > Gluster-users mailing list
> > > > > Gluster-users at gluster.org
> > > > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> > > >
> >
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
More information about the Gluster-users
mailing list