Re: Backing up large unchanging files

[Tim Bishop <tim@bishnet.net>]

Hi Colin,

On Mon, Jul 14, 2025 at 07:07:10PM +0100, Tim Bishop wrote:
> On Mon, Jul 14, 2025 at 03:53:50PM +0000, Colin Percival wrote:
> > On 7/14/25 08:46, Tim Bishop wrote:
> > > I'm backing up some large unchanging files (web server logs). Aside from
> > > the current log, they mostly are unchanging on a daily basis. As per
> > > recommendations I've not compressed these files which gives Tarsnap the
> > > best chance to deduplicate and compress.
> > > 
> > > But, the problem is that Tarsnap is reading these files every day in
> > > their entirety. I guess it has to so it can identify changed blocks, but
> > > this is making the backup take a long time and creates a fair amount of
> > > I/O. And aside from the monthly log rollover, these files haven't
> > > changed from one day to the next.
> > 
> > Assuming you're not running with --lowmem, tarsnap should recognize files
> > which haven't had their {path, inode number, size, mtime} change since the
> > last backup.  So it should only be re-reading the file which is currently
> > being written, not the rotated logs.
> 
> Thanks - that's what I hoped would happen. But not what I saw (no
> --lowmem option in use). Here's an example rotated log file:

It happened again the following night. I couldn't replicate with a dry
run, or with a dry run using --lowmem. However, a dry run with
--verylowmem did exhibit the same behaviour.

Does Tarsnap automatically enable lowmem/verylowmem in any circumstance?
For example, if system memory is low.

Thanks,

Tim.