[Gluster-users] Script and tips for parallelizing rsync

[Alan Orth]

That's good to hear, Dan.

Yeah, since I published that I've heard a variety of similar solutions
(like with GNU parallel).  It's really good to get the information out
there, because you never know you need to run rsync with multiple
threads until you need it!

I think the more solutions we have, the better.  Mine makes sense to me,
because I went through the whole ordeal and built it step by step, but I
can see how other methods would make more sense to other people.  That's
also why I attempted to explain all the rsync and xargs options on my
blog post!

Regards,

Alan

On 07/10/2014 04:27 AM, Dan Mons wrote:
> We do something similar for our nightly backups (100TB between two
> Gluster setups).
> 
> Each of our 6 Gluster nodes gets a set of top level folders
> (representing each department in the org), and within each we thread
> based on folders in the top level of each major section.  That nets us
> around 200+ rsync threads, which makes the nightly sync happen a lot
> faster.
> 
> I played around with parallel rsync, but could never make it work the
> way I wanted.  Just doing a simple "ls -d * | while read DIR ; do
> rsync /$DIR/ remote:/$DIR/ & done" works out far better.
> 
> -Dan
> 
> ----------------
> Dan Mons
> Unbreaker of broken things
> Cutting Edge
> http://cuttingedge.com.au
> 
> 
> On 9 July 2014 21:42, Alan Orth <alan.orth at gmail.com> wrote:
>> Hi,
>>
>> I recently had a RAID failure on one of my Gluster replicas; luckily my
>> replica was ok, and I could re-sync all the data to the bad node's
>> bricks.  I used rsync to pre-seed the brick data, rather than having
>> Gluster's self-heal daemon try to figure it out.
>>
>> It turns out I had way more files than I realized, which exposed some
>> problems with "traditional" rsync invocation.  I found some clever ways
>> to optimize the transfer and speed up the process, and wrote up my
>> experiences on my blog:
>>
>> http://mjanja.co.ke/2014/07/parallelizing-rsync/
>>
>> Hope this helps someone!
>>
>> --
>> Alan Orth
>> alan.orth at gmail.com
>> http://alaninkenya.org
>> http://mjanja.co.ke
>> "I have always wished for my computer to be as easy to use as my telephone; my wish has come true because I can no longer figure out how to use my telephone." -Bjarne Stroustrup, inventor of C++
>> GPG public key ID: 0x8cb0d0acb5cd81ec209c6cdfbd1a0e09c2f836c0
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users


-- 
Alan Orth
alan.orth at gmail.com
http://alaninkenya.org
http://mjanja.co.ke
"I have always wished for my computer to be as easy to use as my
telephone; my wish has come true because I can no longer figure out how
to use my telephone." -Bjarne Stroustrup, inventor of C++
GPG public key ID: 0x8cb0d0acb5cd81ec209c6cdfbd1a0e09c2f836c0

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140710/05a2ee4f/attachment.sig>