Re: fastest/least bandwidth selective archive removal methods?

[Graham Percival <gperciva@tarsnap.com>]

On Mon, Aug 03, 2020 at 06:14:06PM -0700, james young wrote:
> 
> On Aug 3, 2020, at 6:15 AM, hvjunk <hvjunk@gmail.com> wrote:
> > Question: should I delete them one by one, or  cheaper/beter/more-efficient to bundle them all together in batches?
> 
> The documentation recommends batches: 
> https://www.tarsnap.com/improve-speed.html#faster-delete 

Yes, to quote that page: "Multiple archives can be deleted with
the same command; this is usually faster (and never slower) than
using individual delete commands".

> I've got a fairly simple shell script for daily, weekly, and monthly pruning: 
> https://github.com/pronoiac/tarsnap-cron/blob/master/tarsnap-prune.sh 

Thanks for writing tarsnap-cron!  That's one of the projects listed in
https://www.tarsnap.com/helper-scripts.html
to help with archive management.

(I have no experience with any of those projects, so this is
neither an endorsement nor anti-endorsement of any particular
project.)

> > And if in batches: Does tarsnap recover gracefully from connectivity failures while in the process of deletion? Asking as, being >250ms away from the AWS region/zone, I’ve noticed a “hang” on one of my mass deletions, and not sure whether it was busy, or actually stuck and retrying

Three options that you might want to add to your delete commands:
  -v             (to see which archive is currently being deleted)
  --keep-going   (ignore an error from one archive when deleting
                  multiple archives)
  --archive-names    (to read a list of archive names from a file,
                      instead of using multiple -f options)

The --keep-going is particularly useful for recovering if you
cancel the tarsnap command.  If you have

  tarsnap -d --keep-going -f 1 -f 2 -f 3 -f 4 -f 5 -f 6

and cancel the command after it's deleted archives 1 and 2, you
can just repeat

  tarsnap -d --keep-going -f 1 -f 2 -f 3 -f 4 -f 5 -f 6

without having to remove "-f 1 -f 2" from the command.  (The same
goes if you have the archive names in a file.)

Cheers,
- Graham Percival