Re: Speed of Backup

[Colin Percival <cperciva@tarsnap.com>]

On 02/25/14 03:15, Mike Kallies wrote:
> We're doing nightly backups using Tarsnap, approximately 100G of data.
>  The backups are taking a very long time, and the time has been
> increasing steadily since early January.

Hmm...

> Backups which would normally start around 9pm and finish around 5am,
> now start at 9pm and finish around 2:30pm.   There is not a huge
> amount of change in the files. (~1-2G or so per day)

Is that 1-2 GB based on tarsnap's statistics (--print-stats), or based
on your internal reckoning of changing files?

How much data is in files which are not being modified between backups?
If the path, inode #, size, and mtime are unchanged then tarsnap avoids
re-reading files in order to save disk I/O and CPU time.

Is this 100 GB of data consisting mostly of large files, or millions and
millions of small files?

How much RAM does this system have?  Are you sure that it's not swapping?

Is this a generic x86 box, or something embedded (ARM/MIPS/etc.) with a
puny CPU?

> I was looking at filtering on modification date/time, but this could
> introduce a retention issue.  We're backing up daily sets and using
> those sets to maintain 30 day retention.  If we skip files based on
> modification date, we would lose those files from the backups if they
> haven't been modified.
> 
> Is there a method to speed this up?  Maybe a retain-file-only
> if-modification-date-unchanged option?

Shouldn't be necessary... there's something weird going on for such a small
amount of data to take so long.

-- 
Colin Percival
Security Officer Emeritus, FreeBSD | The power to serve
Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid