Re: Tarsnap performance issues

[Colin Percival <cperciva@tarsnap.com>]

On 04/08/17 10:43, Victor Boctor wrote:
> - List archives for a server takes a very long time (order hours) due to a
> high number of archives.  The numbers are high because we had a bug in
> archive trimming which caused accumulation of a large number of archives.
> But I recall even with a smaller number of archives this was a slow
> command.  I know that was caused by some earlier design decisions, but is
> there a have a new approach that can be applied for archives going forward
> (even if these are the only ones that shows up on new list-archives
> command), or we just have to script around tarsnap and avoid using such
> command.

Not fixed yet.

> - When deleting archives in batches, if one fails, the whole command fail.
> If the goal is to delete an archive and it doesn’t exist, then I would
> rather get a warning than a fatal error.  Can this be added even if it
> comes with a new command line switch.

The --keep-going option (new in 1.0.37) does this.

> - The daily tarsnap backups takes hours to complete, if we need to run an
> adhoc backup during such time on a specific one of the web apps being
> backed up, tarsnap fails because it runs concurrently.  If we have a large
> N of such apps that have their db and attachments, is there a way to have
> them no conflict with each other.  So adhoc backup for app C succeeds while
> daily backup is working on app M.

If you want deduplication between your different apps, no -- one archive might
be uploading the same data another archive would need to add.

But if you know that your apps aren't going to share any data in common (or
you don't care about storing it multiple times): Register a separate "machine"
for each app, and do backups for each using the different keys and different
tarsnap cache directories.  As far as tarsnap is concerned, different keys
correspond to completely different archival spaces, so there's no problem with
doing multiple backups at once using different keys.

> - If you are using scripts to wrap around tarsnap to mitigate some of the
> above issues, which ones do you recommend?  Currently, we use our own
> scripts.  We used to have our own local hints about archives in tarsnap but
> it leaked some archives and we removed the functionality at one point and
> used —list-archives instead.  But may consider going back to it.

We've collected a list of (not reviewed) scripts on the tarsnap website:
https://www.tarsnap.com/helper-scripts.html

As for recommending one over the others -- I don't use any of them myself,
so I can't really help here.  Maybe other mailing list readers will chime in
with the reasons they prefer one or another.

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid