Re: tarsnap: Decrypting file would require too much memory

[Daniel Neades <djn@araxis.com>]

Hi Colin,

Thank you for the helpful responses.

> <snip> Given the way that scrypt works, if any of the allocation ends up paged out, the computation will effectively never finish; and (as mentioned above) tarsnap tries to avoid using so much memory as to cause a heavy burden on the system.

Ah, that’s a crucial detail that I hadn’t realized.

>> 7. Interactively reducing the size of the ZFS ARC with, for example,
>> ‘sysctl vfs.zfs.arc_max=512000000‘ doesn’t help, perhaps because FreeBSD seems still to count the released ARC memory as wired. (top does show the ~700 MB reduction in ARC size, however.)
> 
> I was about to suggest trying this.  The fact that the released memory is not
> unwired seems suspicious -- generally speaking, wired memory is not available
> for any other purpose, so reducing the size of the ARC doesn't seem to have
> the intended effect here.  It's possible that I'm missing something -- I'm not
> an expert on either ZFS or the FreeBSD VM system -- but it seems to me like
> the wired memory should be released (perhaps over time) once arc_max is cut
> down, since otherwise what's the point?

Yes, it puzzled me too. I believe the option to reduce the ARC dynamically is new in FreeBSD 11, and it’s not something I’ve tried before. I didn’t wait around to see whether the memory would become unwired over time.

> <snip>
>> 8. However, if I reboot the host, and then immediately run the tarsnap --fsck command within the jail, it succeeds:
>> [snip]
>> 
>> 9. Hypothesis: Tarsnap/scrypt is erroneously deciding that there is insufficient memory available on the system in point (6) above. (Either that, or I am doing something very silly.)
> 
> Yes, this is exactly what's happening -- the available-memory-detection is
> very rough, in large part because the concept of "available memory" is not
> even well defined.

I did have a quick peek at the scrypt source yesterday, and saw that it’s trying to do its best with limited information.

> Maybe we should add a --passphrase-force option which overrides tarsnap's
> attempt to autodetect whether there's enough memory.  (Does anyone have a
> preference for what the option is named?)

An option like that would be great. I had anticipated that perhaps one would need to specify the amount of memory for Tarsnap to use. In that case, borrowing the '--passphrase-mem maxmem' option from tarsnap-keygen would have worked nicely. Your suggestion is even easier. I have no preference for what the option is called, merely that it should exist :-)

Thank you again for the assistance.

-- 
Daniel