Tarsnap performance issues

[Victor Boctor <victor@mantishub.net>]

Hi all,

- List archives for a server takes a very long time (order hours) due to a high number of archives.  The numbers are high because we had a bug in archive trimming which caused accumulation of a large number of archives.  But I recall even with a smaller number of archives this was a slow command.  I know that was caused by some earlier design decisions, but is there a have a new approach that can be applied for archives going forward (even if these are the only ones that shows up on new list-archives command), or we just have to script around tarsnap and avoid using such command.

- When deleting archives in batches, if one fails, the whole command fail.  If the goal is to delete an archive and it doesn’t exist, then I would rather get a warning than a fatal error.  Can this be added even if it comes with a new command line switch.

- The daily tarsnap backups takes hours to complete, if we need to run an adhoc backup during such time on a specific one of the web apps being backed up, tarsnap fails because it runs concurrently.  If we have a large N of such apps that have their db and attachments, is there a way to have them no conflict with each other.  So adhoc backup for app C succeeds while daily backup is working on app M.

- If you are using scripts to wrap around tarsnap to mitigate some of the above issues, which ones do you recommend?  Currently, we use our own scripts.  We used to have our own local hints about archives in tarsnap but it leaked some archives and we removed the functionality at one point and used —list-archives instead.  But may consider going back to it.

Thanks,
-Victor