Re: Backup Restore (Was: Re: Getting started with Tarsnap)

[Scott Wheeler <scott@directededge.com>]

On Feb 15, 2014, at 12:19 AM, Colin Percival <cperciva@tarsnap.com> wrote:

> On 02/14/14 15:00, Scott Wheeler wrote:
>> My point was rather that Tarsnap’s slowness isn’t intrinsic to it being a internet-based backup.  Even for an off-site network backup, it’s extremely slow.
> 
> Yep.  And I'm working on it.  In the mean time, the "parallel extracts"
> workaround is useful in most cases.

I ended up cleaning up and gemifying our little tool for running parallel extracts:

https://github.com/directededge/redsnapper

http://rubygems.org/gems/redsnapper

Doing this will use 25 parallel clients (from testing 25 seemed to be about the place where the benefits maxed out) each extracting up to 50 files to extract an archive:

sudo redsnapper ARCHIVE

We see about a 5x speedup doing things that way.  It’s slower than the normal tool for small archives since it has to fetch the file list first.

-Scott

--
Scott Wheeler | Co-founder | Directed Edge | www.directededge.com | @directededge