Re: Copy/migrate archives/history with a new key?

[creed-january-twig@duck.com]

Thanks for pointing that out, James. 

That was a typo while pasting the command for this mail thread and changing a few things. As seen in the config example in the same mail, I had used '1h' (checked .zsh_history to verify) not '1s'. Actually, in most of those commands with everything in the CLI, I was only keeping '1d 7d' as the '--deltas’ for testing. I pasted the entire thing later in the mail to show my retention plan (which I finally intended to use without '--dry-run' and ended up inserting ‘1s’ by mistake).

Again, thanks.

In fact, I even tried the literal command from the tarsnapper GitHub README:

> tarsnapper -c ~/.config/ts/config --dry-run

and even that failed in:

> "tarsnapper: error: unrecognised arguments: --dry-run"

I tried '-o keyfile' and got the same error. Removed '--dry-run' and (passing the wrong key path - just to be safe), it still failed with error:

> AttributeError: 'Namespace' object has no attribute ‘jobs’

The config did have ‘jobs'.

My problem is that no matter what I tried in the last two days, I was not even able to succeed at running anything ‘tarsnapper' once (and maybe failing at auth or the wrong key; which would have been understandable/intentional). I will try on a bit more maybe.

This finally worked just now (in this specific order) - as in - it reached Tarsnap servers and failed to match any target archive names:

> tarsnapper --target \$date  --deltas 1h 1d 7d 30d 365d -o keyfile path/ts.key - expire --dry-run

which didn’t match anything, neither did '--target "*(\\$date)”'.

This worked *a bit more*:

> tarsnapper --target "Job_<old job name 1>_\$date" \
        --target "Job_<old job name 2>_\$date" \
        --target "Job_<old job name 3>_\$date" \
        --target "Job_<last job name>_\$date" \
        --dateformat "%Y-%m-%d_%H-%M-%S" \
	--deltas 1h 1d 7d 30d 365d \
	-o keyfile /path/ts.key \
	-o v - expire --dry-run

Output received:

> 285 backups are matching
> 274 of those can be deleted
> Deleting <list of 274 archives>

No errors; and the thing is I have ~2000 archives. So I checked (saved '--list-archives' output).

The job 'Job_<last job name>_’ archives count up to exactly 285 i.e. only the last ‘---target’ passed was considered in this command, so no point passing multiple targets. I will try one by one and I guess since I didn’t run them at once (hopefully things should be fine; hopefully!) - I will save all output just to match!

Because I can’t really batch rename archives (even if I try that will be yet another sinkhole).

This is kinda tiring.

/rant

The reason I became frustrated is that people seemed to be using it (and other tools) successfully and easily (even with the very sparse amount of info/activity on the GitHub repos) and I can’t seem to get anywhere with it :(

That’s why I gave up. I don’t think I am the *hacker* kind of person. I need sufficient and clear docs and human-readable examples (preferably layperson-proof). I am someone who wishes for the old CrashPlan to come back (i.e. CrashPlan from before "SMB-and-gone-bonkers”).

In fact, yesterday I prepared a space-separated list of archives I want to delete in a ‘delete.txt' file and another list that I intended to keep in a ‘keep.txt' file. Verified the two lists for being mutually exclusive and combined producing the original same as '--list-archives’ local copy. I then read the time part of each archives in ‘keep.txt' archive name and I was satisfied. I was planning to go:

> tarsnap -d [-f archive]

on it (or a shell loop reading archives from 'delete.txt' file).

But then I saw that something like this is not sustainable for someone like me.

Also, such granular maintenance features are either never coming to Tarsnap or might not be a priority and rightly so. It is literally a backup tool for the “truly paranoid” and why the hell a truly paranoid backup doer would even want to delete an archive ever :) Jokes apart, (IMHO; at the risk of presumption) this tool attracts a certain kind of user - the one that knows their way around the CLI and Unix tools and aren’t deterred by a little or more friction and, if I may say so, might actually welcome a bit of getting their hands dirty parts - in the sense of doing things with the bare hands :)

I think I used Tarsnap since 2018 without thinking about any of these because I never really had to work/deal with the tool before. Graham’s GUI kept me sheltered :) Hopefully someday 'tarsnap-gui' will have such maintenance features.

> On 17 Aug 2025, at 5:53 AM, james young <pronoiac_at_gmail.com_creed-january-twig@duck.com> wrote:
> 
> Glancing at the tarsnapper page:
>> tarsnapper --keyfile <key path> --target "$date" --deltas 1s 1d 7d 30d 365d - expire --dry-run
> 
> I wasn't familiar with tarsnapper; I think:
> * change to "\$date" from "$date"
> * change to "1h" from "1s"
> * keyfile: I'd try something like "-o configfile tarsnap.conf". the
> tarsnapper docs are ... handwavey about this.
> * if you have multiple jobs backing up there, you should have multiple
> tarsnapper runs and a different, specific --target for each of those
> 
> I personally use suffixes for the archives like "-daily", "-weekly",
> "-monthly". Going in, I copied the archives to (effectively) rename
> them, and then shell scripts for pruning them. It feels more
> debuggable and less magic.
> 
> -James
> 
> On Sat, Aug 16, 2025 at 1:17 AM <creed-january-twig@duck.com> wrote:
>> 
>> Hey Chris, thanks a lot for the details.
>> 
>> This is what I ended up having:
>> 
>> tarsnap:
>>  keyfile: "/Users/user/tarsnap.key"
>>  cachedir: "/Users/user/.cache/tarsnap"
>> 
>> jobs:
>>  prune:
>>    target: "$date”     // (comment for this mail) this is the only one that didn’t throw a date-related error
>>    dateformat: "%Y-%m-%d_%H-%M-%S"
>>    deltas: 1h 1d 7d 30d 365d
>> 
>> (^ I basically needed a target that was like <anything - multiple hostname><date as per dateformat>)
>> 
>> I’ve been trying to run Tarsnapper for the past few hours, but I keep getting different errors. I’ve tried skipping the config file and passing everything in the command line, and I’ve tried every combination/order of commands and arguments.
>> 
>> Even tried like this as the GitHub readme says:
>> 
>> tarsnapper --keyfile <key path> --target "$date" --deltas 1s 1d 7d 30d 365d - expire --dry-run
>> 
>> again - removing args, adding, changing orders, etc. (with and without the hyphen before expire and moving expire around).
>> 
>> It hasn’t even reached a stage where it will finally show me the result of the dry-run.
>> 
>> Some of these are here:
>> 
>> - tarsnapper.config.ConfigError: %s: target must make use of the following placeholders: date
>> 
>>> The best part are these two errors, tarsnap complains:
>> - tarsnap: Keys must be provided via --keyfile option
>> 
>>> and when I passed it, tarsnapper complained:
>> - tarsnapper: error: unrecognized arguments: -c // same for --config, --config-file
>> 
>> - tarsnapper: error: argument {make,expire,list}: invalid choice: '/Users/user/.config/tarsnapper/config.yaml' (choose from make, expire, list)
>> - Skipping 'None', does not define deltas
>> 
>> I’m not sure what I’m doing anything wrong (but obviously I am). I thought of reaching out to the GitHub issues, but the last issue was updated in 2022.
>> 
>> I suppose I’ll give up. I should’ve thought before choosing Tarsnap. I lack the expertise, perseverance, and patience to use this tool I realise. Actually, I never considered Tarsnap before Tarsnap-GUI  was launched :) and when I started using the GUI, I didn’t check the maintenance features.
>> 
>>> On 16 Aug 2025, at 3:01 AM, Chris Leyon <cleyon_at_gmail.com_creed-january-twig@duck.com> wrote:
>>> 
>>> I haven't studied tarsnapper's code so I can't tell you exactly what it does or does not delete. But I can show you the resulting behavior from my deltas line (1d 7d 30d 90d 360d) and hopefully explai
>>> DuckDuckGo did not detect any trackers. More Deactivate I haven't studied tarsnapper's code so I can't tell you exactly what it does or does not delete.  But I can show you the resulting behavior from my deltas line (1d 7d 30d 90d 360d) and hopefully explain the results:
>>> $ # current date is 2025-08-15
>>> $ tarsnap --list-archives -v | sort -n -k2,3 | column -t
>>> host/job-20240723-045000  2024-07-23  04:50:00
>>> host/job-20241021-045000  2024-10-21  04:50:00
>>> host/job-20250119-045000  2025-01-19  04:50:00
>>> host/job-20250419-045000  2025-04-19  04:50:00
>>> host/job-20250519-045000  2025-05-19  04:50:00
>>> host/job-20250618-045000  2025-06-18  04:50:00
>>> host/job-20250718-045000  2025-07-18  04:50:00
>>> host/job-20250725-045000  2025-07-25  04:50:00
>>> host/job-20250801-045000  2025-08-01  04:50:00
>>> host/job-20250808-045000  2025-08-08  04:50:00
>>> host/job-20250809-045000  2025-08-09  04:50:00
>>> host/job-20250810-045000  2025-08-10  04:50:00
>>> host/job-20250811-045000  2025-08-11  04:50:00
>>> host/job-20250812-045000  2025-08-12  04:50:00
>>> host/job-20250813-045000  2025-08-13  04:50:00
>>> host/job-20250814-045000  2025-08-14  04:50:00
>>> host/job-20250815-045000  2025-08-15  04:50:00
>>> 
>>> So starting with "1d", I get daily backups ever day until I reach 7 days on the 8th.  Then I get weekly (7d) backups until I reach a month (30 days) on July 18.  Then I get monthly (30 day) backups going back a year.  (My records stop here.)  tarsnap runs every day at 04:50 and tarsnapper's algorithm retains only these archives according to the schedule.  I hope this helps.  I would also add that tarsnapper seems to support a "--dry-run" option that you might explore.
>>> 
>>> On Fri, Aug 15, 2025 at 2:04 PM <creed-january-twig@duck.com> wrote:
>>> Thought of trying that. But I am not sure what it does if it doesn’t find any archive on a certain day where a certain delta will lead it to? What I am trying to say is, since it is not a very widely used or has layperson-proof documentation, I am not really sure how it behaves.
>>> 
>>> Anyway, so if I have to achieve something like "10h 7d 6w 12m 5y” retention and delete the rest, shall I have to set it as
>>> 
>>> ‘ deltas: 1d 2d 3d 4d 5d 6d 7d 8d …. '
>>> 
>>> 1d: the GitHub says - this means 24 retentions, right?
>>> 2d 3d 4d 5d 6d 7d 8d: for 7 days?
>>> and beyond this basically find 6 day-numbers so that it will cover 6 weeks and then 12 day numbers that will cover 12 monthly retentions, right?
>>> 
>>> (With the assumption that it will not only try to keep the exact archive on that exact day/date but one closest such match.)
>>> 
>>> I have kept ‘ target: "{date}” ‘ because I have 4-5 different prefixes but they all backed up the same data set (it’s just because of a change in job name). So my assumption was that should work and it will only look at the {date} part of the archive names which is what I wanted.
>>> 
>>> 
>>>> On 15 Aug 2025, at 11:17 PM, Chris Leyon <cleyon_at_gmail.com_creed-january-twig@duck.com> wrote:
>>>> 
>>>> I use tarsnapper as well, but only run it once per day so I can't speak to a delta of "10h".  I just use the number of days corresponding to the duration.  My deltas line is
>>>>    deltas: 1d 7d 30d 90d 360d
>>>> Tweak as needed.  Also my target includes the `name' variable:
>>>>    target: <HOSTNAME>/$name-$date
>>>> 
>>>> On Fri, Aug 15, 2025 at 1:14 PM <creed-january-twig@duck.com> wrote:
>>>> Thanks Colin, I’ll reach out once it's done.
>>>> 
>>>> I have ~2000 archives. I likely skipped pruning planning when setting up tarsnap-gui (haven’t reinstalled it on my reset Mac, so not sure if it even supported pruning). I don’t want to keep this many archives — and even re-encryption won’t be fast with so many, even if each archive is small.
>>>> 
>>>> I saw '--fsck-prune' but it’s not what I thought.
>>>> 
>>>> Is there a simple way to delete all except something like — hourly:10 daily:7 weekly:6 monthly:12  yearly:5?
>>>> 
>>>> I have 'tarsnap --list-archives | sort' saved (can add '-v' if needed), and since all archive names end in '%Y-%m-%d_%H-%M-%S', I can script something or use an LLM to pick which to keep.
>>>> 
>>>> I found https://mail.tarsnap.com/tarsnap-users/msg01678.html and had seen the helper scripts section already. So Tarsnapper is available on homebrew luckily, because couldn’t make prunef ‘make install’ happen.
>>>> 
>>>> But then Tarsnapper uses some retention scheme (i.e deltas) in such a way that I guess I can only use days. So it doesn't work:
>>>> 
>>>> [tarsnapper’s config.yml]:
>>>> 
>>>> jobs:
>>>>  prune:
>>>>    target: "{date}"
>>>>    dateformat: "%Y-%m-%d_%H-%M-%S"
>>>>    deltas: 10h 7d 6w 12m 5y
>>>> 
>>>> [and then (I will remove dry-run after checking once)]:
>>>> tarsnapper -c ~/.config/tarsnapper/config.yaml expire --dry-run
>>>> 
>>>> results in:
>>>> 
>>>>> tarsnapper.config.ConfigError: Not a valid delta: 12m
>>>> 
>>>> 
>>>> PS. Any easy way to search across https://mail.tarsnap.com/tarsnap-users, other than opening every link one by one?
>>>> 
>>>>> On 15 Aug 2025, at 3:07 AM, Colin Percival <cperciva_at_tarsnap.com_creed-january-twig@duck.com> wrote:
>>>>> 
>>>>> On 8/14/25 11:04, creed-january-twig@duck.com wrote:
>>>>>> I *have to* change my tarsnap key (or rather, stop using the old key).
>>>>>> I see this https://www.tarsnap.com/tips.html#copy-archive as well but I don’t really understand what it is and what it does - but I don’t see a key mentioned in the command so I guess not like “restic copy”.
>>>>> 
>>>>> Right, that's for copying one archive, using the same keys and within the
>>>>> same archival space.
>>>>> 
>>>>>> This https://www.tarsnap.com/man-tarsnap-recrypt.1.html seems to be the only way, right?
>>>>> 
>>>>> Yes.  That creates a new archival space, copies everything across, and then
>>>>> deletes the old copy.
>>>>> 
>>>>>> Also, the original/existing key was not *passworded*, can I generate the new key as ‘--passphrased’ and then proceed with the recrypt? I am asking because I believe to re-encrypt, ‘tarsnap-keyregen’ has to be used and the key is derived from the old key.
>>>>> 
>>>>> Correct.  To be more precise, the chunking parameters are kept from the old
>>>>> key but everything else is generated anew.  (The chunking parameters need to
>>>>> be kept so that new data will deduplicate against the copied data.)
>>>>>> This also raised the question - does it render the old key useless after the re-encryption is done, or both keys have access now?
>>>>> 
>>>>> Both keys will work but they'll access different archival spaces (and the
>>>>> old keys will point to an archival space with no archives after recrypt
>>>>> deletes everything using the old keys).
>>>>> 
>>>>> If this is a "keys were stolen" scenario then let me know and I can disable
>>>>> the old keys.
>>>>> 
>>>>> --
>>>>> Colin Percival
>>>>> FreeBSD Release Engineering Lead & EC2 platform maintainer
>>>>> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
>>>>> 
>>>> 
>>>> 
>>> 
>> 
>>