[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Backing up large unchanging files



Hi Tim,

On 7/14/25 08:46, Tim Bishop wrote:
I'm backing up some large unchanging files (web server logs). Aside from
the current log, they mostly are unchanging on a daily basis. As per
recommendations I've not compressed these files which gives Tarsnap the
best chance to deduplicate and compress.

But, the problem is that Tarsnap is reading these files every day in
their entirety. I guess it has to so it can identify changed blocks, but
this is making the backup take a long time and creates a fair amount of
I/O. And aside from the monthly log rollover, these files haven't
changed from one day to the next.

Assuming you're not running with --lowmem, tarsnap should recognize files
which haven't had their {path, inode number, size, mtime} change since the
last backup.  So it should only be re-reading the file which is currently
being written, not the rotated logs.

--
Colin Percival
FreeBSD Release Engineering Lead & EC2 platform maintainer
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid