Re: tarsnap: Decrypting file would require too much memory

[Colin Percival <cperciva@tarsnap.com>]

Hi Daniel (& list),

On 10/19/16 03:02, Daniel Neades wrote:
> 2. The Tarsnap key file has been generated (on a different host) with --passphrase-mem 268435456, so decrypting the key file should not require more than 256 MB of memory.

Yes and no.  The key derivation for decrypting the file will require 256 MB,
but tarsnap will use a maximum of 50% of the memory it thinks is "available"
for this purpose.

> 3. Within the jail, ulimit -a gives: 
> [snip]

This looks fine.

> 4. The host uses ZFS. When the host has been running for a long time, such that there is significant data in the ZFS ARC, top gives the following memory stats:
> 
> Mem: 11M Active, 64M Inact, 1804M Wired, 82M Free
> ARC: 1205M Total, 640M MFU, 229M MRU, 1060K Anon, 19M Header, 317M Other
> Swap: 4096M Total, 17M Used, 4079M Free
> 
> 
> 5. Given that the ZFS ARC is able to release memory, and given also the
> free swap space, a process (e.g. tarsnap) should have no trouble allocating
> 256 MB of memory to decrypt the key file. The ulimit data size (see 3,
> above) is not a limiting factor.

The problem here is that tarsnap can't simply try to allocate space and see
if it works -- on modern VM systems, this will always succeed, but won't
necessarily result in actual RAM being provided.  Given the way that scrypt
works, if any of the allocation ends up paged out, the computation will
effectively never finish; and (as mentioned above) tarsnap tries to avoid
using so much memory as to cause a heavy burden on the system.

> 6. However:
> 
> # tarsnap --keyfile tarsnap.key --fsck
> Please enter passphrase for keyfile tarsnap.key:
> tarsnap: Decrypting file would require too much memory
> tarsnap: Error decrypting key file: tarsnap.key
> tarsnap: Cannot read key file: tarsnap.key
> 
> 
> 7. Interactively reducing the size of the ZFS ARC with, for example,
> ‘sysctl
vfs.zfs.arc_max=512000000‘ doesn’t help, perhaps because FreeBSD seems still
to count the released ARC memory as wired. (top does show the ~700 MB
reduction in ARC size, however.)

I was about to suggest trying this.  The fact that the released memory is not
unwired seems suspicious -- generally speaking, wired memory is not available
for any other purpose, so reducing the size of the ARC doesn't seem to have
the intended effect here.  It's possible that I'm missing something -- I'm not
an expert on either ZFS or the FreeBSD VM system -- but it seems to me like
the wired memory should be released (perhaps over time) once arc_max is cut
down, since otherwise what's the point?

> 8. However, if I reboot the host, and then immediately run the tarsnap --fsck command within the jail, it succeeds:
> [snip]
> 
> 9. Hypothesis: Tarsnap/scrypt is erroneously deciding that there is insufficient memory available on the system in point (6) above. (Either that, or I am doing something very silly.)

Yes, this is exactly what's happening -- the available-memory-detection is
very rough, in large part because the concept of "available memory" is not
even well defined.

Maybe we should add a --passphrase-force option which overrides tarsnap's
attempt to autodetect whether there's enough memory.  (Does anyone have a
preference for what the option is named?)

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid