[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Backup Restore (Was: Re: Getting started with Tarsnap)

On 03/07/14 13:52, Scott Wheeler wrote:
> I ended up cleaning up and gemifying our little tool for running parallel extracts:
> 
> https://github.com/directededge/redsnapper
> 
> http://rubygems.org/gems/redsnapper
> 
> Doing this will use 25 parallel clients (from testing 25 seemed to be about the place where the benefits maxed out) each extracting up to 50 files to extract an archive:
> 
> sudo redsnapper ARCHIVE
> 
> We see about a 5x speedup doing things that way.  It’s slower than the normal tool for small archives since it has to fetch the file list first.

Thanks for publishing this.  One suggestion -- I'd change the "up to 50 files"
to "up to 1000".  Otherwise with lots of files you end up spawning extra
processes which each download the same set of metadata.

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid